Are you using Veeam & VTL?

The TS4500’s i have both have dual I/O stations where it puts tapes to be ejected/imported. 

When working with tape there are a few things to consider.

1. ENCRYPT THE TAPES.  This is a no brainer
2. PHYSICAL SECURITY. Make sure the location of the library is secure. If you are worred about someone accessing it physically, it’s not in a secure location. The same reason I am not worried about my SAN and Servers, is because our data center is secure.
3. LOGICAL SECURITY. Vlans, firewalls, limited access. Once things are set up, Veeam is really the only thing that should be accessing the library, and it does that via Fiber mostly. (there is some networking but minimal) Lock down the network to only the admins on that systm, create strong passwords etc. 
4. EJECT TAPES. even with the above…. “THINGS” happen. you can’t overwrite something that is not in the machine. sure, you may have to scan all the tapes and import. Sure, it’s going to take a LONG time, but still beats a data loss scenario.
5. Combine the above with object storage, flash, SOBR for faster restores. TAPE is the last line of defense and not meant for speed. It’s meant for Airgap, price, and longevity.
6. Don’t have just 1 copy of data on a tape. They can break, snap, get eaten by a drive. Always have a few copies.. 3-2-1 is best. but if you have object storage, tape can be 1 of those. I tend to keep 2 copies on tape, 

I’ve not come across anything that will auto-eject a tape into a Tub. Mostly done by a robot otherwise, you are risking damage to a tape. 

A tub full of blankets at a DR site haha😀

I don’t see how the “ActiveVault” is any different from the concept of VTL. Based on watching their video on how ActiveVault works, at the end they indicate an Administrator can go into the software and move it back out, that isn’t air-gapped. Their “air-gapped” claim is a marketing gimic. While it may be virtually air-gapped from the application (e.g. Veeam), a malicious actor can still get access to it via the tape library admin console and cause harm.

As for the Ransom Block, this is also a gimic. It is literally just ejection, nothing special, however it seems it ejects multiple drives at once, the whole magazine, so now you have to push it back in every day to handle the next day’s backup. Unless you could leave it in the ejected state, and newly backed up tapes can still be offloaded to that “partially ejected” magazine. I assume since they don’t mention that you can keep adding to the “partially ejected" magazine that you can’t, which makes it a general pointless feature in my opinion. For example, how often will I know I am about to be hit by ransomware and I am also fast enough to beat the malicious actor and log in and enact the “Ransom Block”? 



I guess another way to phrase what I am looking for, which would make me choose Physical Tape Library over Virtual Tape Library, is PTL having 1 magazine for input only and another magazine for output only. 

The “Logical Tape Blocking” reads like what I want, but if it is software based, then they could find a vulnerability. It says you can only undo it at the local console, but it would require more knowledge on how exactly that is enforced, whether it be something mechanical or not. I might just have to call them on that since their documentation is geared to marketing not technicality.


 

At the moment, here would be my reasons for going with VTL, assuming nothing changes my mind in the next couple of days of research (hopefully this helps OP and anyone else…..feel free to pick these apart):

• Costs variance compared to physical tape infrastructure is not large. Could be cheaper and could be slightly more. All matters on what hardware is chosen. So no real clear winner to me. (e.g. VTL could be near $0 for me as I can use re-purposed and QuadStor)
• Excluding physical tape ejection, and depending on what VTL software you choose, you can match all features of a Physical Tape Library and maybe even have more, including syncing between 2 sites at faster speeds.
• No physical interaction needed unless you add any copies to actual physical tape, which QuadStor can do.
• Security wise, I can lock it down the same way I would the Physical Tape Library. Firewall walls, access-list rules in switch/router, no remote access to console, restrict physical access to the local console, host server for console no on the domain, etc.
• Potential flexibility of upgrading system in the future, where you don’t have to worry about aging tape technology. (Assuming you don't put anything on tape for super long term). For example, if the physical server crashes there are more ways to quickly get a new system up (e.g. old re-purposed hardware instead of having to buy a new device, virtualizing potentially).

Thanks @Scott for the list.
@dips I wasn’t exactly thinking a drop of 6 ft or anything. I was thinking more of a an inch drop (I thought tapes were less susceptible to damage because there is nothing mechanical in them?) or even rig a slide to a bucket (blanket lined !!! hah) to reduce or eliminate any possible damage (e.g. like a tray a cartridge at an incline and it slowly filles up). 

In all my years working at IBM as a service tech, I did not see this exist :)

I picture a laundry hamper with cloths in it and a tape failing ever so soft onto it.  Cut away to a fabric softener commercial.

It seems like you have done your research. The thing you need to decide is what's best for you and what risk is ok for you. 

You could run backups and export tapes multiple times a day. Is it reasonable? probably not.

Can you afford to pay someone to be managing tapes all day? 

Some export weekly, monthly, every ¼. It all depend on your personal RTO/RPO, budget, staffing etc.

I used to walk in businesses where someone's job was to sit there, and every so often eject tapes. 

I also mentioned site security. Sure the tapes are in the library, but that is why you have multi sites with security… you are so focused on someone standing at the machine with console access. 

Lets be real here. If I am in your datacenter, standing at your tape library, and have LOGGED IN with admin access. It is too late for you.  The fact someone even getting to the machine should set alarms and have police there immediately. 

On that same example, someone could just light the building on fire. some things you have to accept. This is why we have 3-2-1 rule and 1 copy at another site. Cloud is your friend for that and immutable object storage. 

@Scott Not sure how long it has been since you have worked with IBM, but do you know if they have the Input Only and Output Only functions for cartridges? Is that a thing? Is that what you were essentially alluding to on the TS4500’s when mention dual I/O stations? I looked here and it doesn’t imply that, unless I am reading wrong:
https://www.ibm.com/docs/en/ts4500-tape-library?topic=library-io-stations

I am looking for something that is a physical separation feature, where once written to it is moved, not fully ejected, into a magazine to get taken out a later time. That could be after 1 backup, or after 3 or 4, does not really matter as long as you unloaded the “used/written to” magazine/cartridge before it fills up.

I’m still catching up on this thread so apologies if you have already motioned it. Have you looked into, say Amazon Storage Gateway with Virtual Tapes? 

If you are talking to me and not OP, we have not. Admin will not let us go cloud no matter the benefit. So it is on-prem Tape Library for us as an option only. I am also trying to look at On-Prem S3 Object storage or Veeam SOBR, but running into lack on good info on the former and still researching the latter and how it can help us.


New question relating to all this, since this thread is poppin’ as the ids would say and we see to have a few Veeam experts in here (not me), as VTL can sometimes enable immutability…..if you consider tape that is ejected 100% secure (yes I know), what % secure comparatively would you consider a WORM tape left in the tape library and what % secure would you consider the general idea of data being stored on an immutable store like AWS, S3 compatible, Linux Hardened Repo, SOBR Veeam Archive Tier setup with one of the previous mentioned technologies or something else?

It’s an Input/Output Station, so no, it is no one for input and one for output.

But you have to manually open the I/O, pull the tapes and reinsert them and then close the I/O again to be able to checkin the tapes again. Seems rather air-gapped for me.

And to be absolutely sure you have to put them outside and put them into a safe.

Manual intervention is need in each scenario. Otherwise it is not physically air-gapped….

And a VTL is mostly software, so you can delete either the tapes or the whole VTL.

The I/O works like this.

You fill it full of tapes, they are added to the physical library. from there you would add them to the Veeam VTL and away you go. From Veeam when they are exported they end up in the virtual IO, from there you can move them out.

I don’t believe you can choose one for in and one for out. 

They hold 18 tapes each, so on LTO8 that is a fair chunk of data. Chances are you will not be importing exporting at the same time. You would most likely have Veeam spit the tapes out, then you could pull them out and add the old tapes back in. Those tapes should already be registered with the VTL and you are done. The only time you add to the VTL is when adding new tapes to the TS4500.

The dual IO stations are just a good idea for not having to run to the site if you want to export more than 18 tapes as they can be sitting waiting. 

WORM may be something you are looking for if you guys are that worried about your tapes getting compromised. 

@JMeixner “And a VTL is mostly software, so you can delete either the tapes or the whole VTL.” To counter this, based on the OP and others, me included, looking at VTL, not ejecting tape physically is clearly something OP and other might have considered and are OK with. So with that in mind, I am not sure how VTL is worse than PTL, as a malicious actor could gain access to the tape library and destroy the environment via wiping tapes, configs, flashing the firmware of the library hardware maliciously, etc.

@Stabz Are you all considering VTL and looking at accepting the fact that there won’t be a true air-gap? Or are you fresh in your research and you did not know this yet? Have you found any information here or elsewhere to help you all make a decision yet?

Checkout this doozy of a claim I just came across from Starwind Software, located (“https://www.starwindsoftware.com/features#vtl”):
“Ransomware can't encrypt tape libraries”

Sorry, it seems we have to define the term VTL first.

My understanding of a VTL is a software construct on a normal server which behaves logically as a tape library and stores data in files which are written sequentially like a tape.

These are available as hardware appliances too. For example an IBM TS7700. I have tested them for a big customer some years ago and the main problem at this time was that the bandwidth to the virtual tape drives was not comparable to the bandwidth to physical tape drives (some 10Gb Ethernet connections for all logical tape drives together against a dedicated FibreChannel connection for each physical tape drive).

@Scottseems to understand a VTL as a partitioning of a physical library. IBM calls this “logical library”. This is simply a logical partition of a physical library with assigned tape slots and drives. Your applications see the physical tape library as multiple libraries.

And please excuse me @JustVeeamUser , I am still not sure what you understand under the term VTL.

Mhhh, technically is this correct. They will not encrypt the data on the tapes, they will more likely delete it….

The result is the same 😆 but the marketing claim is different.

@JMeixner Your 2nd paragraph sums it up pretty much what my understanding it:
“My understanding of a VTL is a software construct on a normal server which behaves logically as a tape library and stores data in files which are written sequentially like a tape.”

I am not talking about any sort of feature labeled “VTL” that a product uses on top of physical tape hardware.

As far as Starwinds claim, couldn’t they just encrypt the data on a non-worm tape by forcing rotation and encrypt each tape 1 at a time? Unless you were stating it’s technically correct they can’t encrypt the “tape library” itself, but they can encrypt the tapes/data on tapes.

OK 😎

Yes, tapes in a library can be attacked by malware. Normally they are deleted, not encrypted. It takes a little bit longer to accomplish than with a filesystem, but it is possible. The attacker can access as much tapes as your number of drives at a time. So, you have a chance to be alerted of unusual activity by a monitoring system and rescue the remaining tapes.

This is my main pain point with VTL. If the attacker gains access to the server the VTL runs on, he can simply delete the filesystem or the volume the virtual tapes reside on. In this case immutability does not protect you. It’s dome in seconds.

The tapes are out of reach of an attacker in the case you check-out them out of the library and put them somewhere else. In this case the attacker would have to hack your tape movement process which includes manual intervention of human beings. In this case I would hope that is some “natural intelligence” is part of the process, when suddenly much more tapes are requested to be checked-in into the library.

With a VTL supposedly once a “virtual tape” is written to it can be moved to a partition that the application (Veeam, etc.) cannot see/doesn’t have access to. So if you lock down the host-based firewall on the VTL host server and locked down the network-based firewall to where only the application (Veeam, etc.) can get to it on the required ports…..which means you have no remote access to the management of the VTL, wouldn’t the only way they could theoretically get to it is by a vulnerability in the OS that allows them to traverse from an allowed partition to a non-allowed? Sort of like putting the VTL server in the LAN and everything else in the DMZ?

This is standard setup of websites, with webservers in the DMZ and DBs in the LAN/non-DMZ. Now imagine you go a step further and routinely the data in the DB is archived to another DB in the same SQL server Instance. If you set the user of the web application to only allow it access to that live DB and not the archive DB, wouldn’t the only way a malicious actor could gain access to the archive DB is a vulnerability in SQL? To be more specific, let’s say in this scenario, which would be like the VTL, the web server is the only server in it’s DMZ and the only port allowed open is 1433 to that 1 SQL server.

I’ serviced TS7700’s for years as well. 

Just a bunch of disk acting as tape. Not much different tbh, but you lose the available to have your full airgap. 

I’d say in this day and age, just go with tape if you want tape. connecting Veeam to a VTL isn’t really going to benefit most. The point of the VTL was to connect to things like mainframes and older systems that REQUIRED tape and would have to do many reads and run/boot off tape etc. 

I think object storage would give you what you are looking for if VTL is in your shortlist. Plus immutability, and many other features.  If you want tape, just get tape. 

Exactly 😎

Do you all know of a cheap object storage option for non-profits that pretty much have no budget but have about a 5TB/day of backups (If Full daily backups to tape) and management doesn’t want to use cloud?

MinIO without support?

I would not recommend this without support.

😀 sorry, professional IT will cost. Even without support you will need a machine or more than one to run MinIO...

And why do you want to create a daily full backup? Create daily incremental backups and weekly synthetic fulls. This is fully supported by Veeam with object storage.

Minimizing the amount of tapes needed to restore is the reason for the daily full. I already don’t trust tape based on past experience and what I am reading online about how much you have to double-check it is corrupt-free (routine checks, etc.), so I don’t want to worry about multiple tapes. 

One last question about VTL to dissuade/persuade the use of it, and this is backed by CISSP background knowledge and the fact that I have zero trust about any technology innately, does anyone have a link/docs to prove you can lock down a physical tape library more than a virtual tape library? Otherwise just saying that tape is secure is like saying fax is secure. For example, you can install Endpoint Protection on CentOS that QuadStor sits on, but most physical tape libraries don’t have an OS you can do that with, they are a proprietary OEM solution. Since you do have to consider that a malicious actor can gain access to your physical tape library system and screw up, in various ways, your backups that you think are going smoothly (talking about over something more like a 2 week period, not months, obviously period restore tests would catch the months case but might not catch it for a couple of weeks). 

My comment referred to object storage...

@JMeixner I saw that, and I am looking into minIO, but my question was separate and trying to tie back to the original OP question of VTL, just a more security specific, about controls, question this time.