Are you using Veeam & VTL?


Userlevel 7
Badge +7

Hello guys, 

I'm looking some informations about Virtual Tape Library (VTL) and I wanted to know if some of you use this technology and why ?


 


51 comments

Userlevel 7
Badge +17

No, we are using physical libraries only. 😎

I don’t see much advantages in using a VTL. The data is in files in a filesystem which is not immutable normally. So, The big advantage of tape - the airgap - is gone.
So, for testing purposes… maybe 😁

Ok, you can use some specialized machines like an IBM TS7700, but then you can use a normal library, too….

Userlevel 7
Badge +20

I had exactly one customer using an AWS VTL, and that was because archive tier object storage integration didn’t exist yet, I then migrated them to object storage and all was good! This was a leap from v9 to v11, so again, very specific scenario.

Userlevel 7
Badge +22

I had a VTL gateway setup to AWS back in the day. I also played with Quadstor https://quadstor.com/virtual-tape-library.html 

I believe you could actually export the VTL tape to real tapes as well. 

Tape is Airgap only in the physical understanding, however if going by the strict definition of  Airgap as per the official NIST definition you cannot have automated transfer of data, it must be manually performed by humans, so scheduled Veeam jobs don’t fit the definition.

https://csrc.nist.gov/glossary/term/air_gap#:~:text=Definition(s)%3A,manually%2C%20under%20human%20control).

rfc 4949

 

I have seen some marketing departments using the term air gap a lot even for Veeam’s insider protection but it is not correct.

 

 

Userlevel 7
Badge +20

We currently do not use VTL with Veeam but rather physical tape libraries at the moment.  I know we are investigating VTL but not sure if that will be for Veeam specifically or other services.

Userlevel 7
Badge +17

AWS VTL is something different 😎 but this was not the question if I understand it correct.

Userlevel 7
Badge +8

No, same thought as @JMeixner @Chris.Childerhose @Geoff Burke 

i don’t understand the need for Veeam, maybe a protocol break from tcp/ip to FC but it will make more sense on physical tape library.

Userlevel 7
Badge +22

There are a number of good posts on using Quadstor with Veeam if you want to test. Very quick and easy to setup and it can emulate real drives:

 

https://www.veeam.com/blog/how-to-create-vtl-and-why.html

 

https://jorgedelacruz.uk/2018/10/16/veeam-quadstor-a-perfect-option-for-linux-based-virtual-tape-library-vtl-free-and-very-easy-to-deploy/

 

https://domalab.com/tape-backup-veeam-quadstor/

Userlevel 7
Badge +7

We do use VTL which does work quite well for our use case. It is quite robust and there is the option to have WORM only media so it can not be overwritten, together with expiry options when using GFS. 

The tapes can be taken ‘offline’ and stored into another archive tier. However, recovery times can be quite large when a tape is recalled. 

Userlevel 7
Badge +17

You intend a robot to pick the out of the i/O station and to put them back in?

This would be nice, but unfortunately wi don’t have anything like this. We depend on helping hands in the DC…

 

The tapes with multi year retention are read at least every two years. Or at every tape drive exchange, then they are read and copied to new media.

But most tapes don’t have such a long retention, so they are reused in shorter cycles..

 

Userlevel 7
Badge +20

I just set up CentOS with Quadstor VTL so I have tape server on Linux for v12 and Chapter 4 of my book. 😎

Userlevel 7
Badge +8

@Scott Not sure how long it has been since you have worked with IBM, but do you know if they have the Input Only and Output Only functions for cartridges? Is that a thing? Is that what you were essentially alluding to on the TS4500’s when mention dual I/O stations? I looked here and it doesn’t imply that, unless I am reading wrong:
https://www.ibm.com/docs/en/ts4500-tape-library?topic=library-io-stations

 

I am looking for something that is a physical separation feature, where once written to it is moved, not fully ejected, into a magazine to get taken out a later time. That could be after 1 backup, or after 3 or 4, does not really matter as long as you unloaded the “used/written to” magazine/cartridge before it fills up.

The I/O works like this.

 

You fill it full of tapes, they are added to the physical library. from there you would add them to the Veeam VTL and away you go. From Veeam when they are exported they end up in the virtual IO, from there you can move them out.

 

I don’t believe you can choose one for in and one for out. 

 

They hold 18 tapes each, so on LTO8 that is a fair chunk of data. Chances are you will not be importing exporting at the same time. You would most likely have Veeam spit the tapes out, then you could pull them out and add the old tapes back in. Those tapes should already be registered with the VTL and you are done. The only time you add to the VTL is when adding new tapes to the TS4500.

 

The dual IO stations are just a good idea for not having to run to the site if you want to export more than 18 tapes as they can be sitting waiting. 

 

WORM may be something you are looking for if you guys are that worried about your tapes getting compromised. 

 

 

 

 

Userlevel 7
Badge +6

No VTL here.  Honestly, the only time I’ve seen VTL in use was in my old environment for our IBM I Series (AS/400) and Z-Series (Mainframe) folks so that they could backup to our Avamar/DataDomain environment as if they were tapes.  Otherwise, I’ve never seen a need for it.

Userlevel 7
Badge +17

I am using physical tape libraries in some customer environments. It's the cheapest storage for long time retention.

In one case the tapes are ejected and stored in a safe.

The management consoles are protected by user and passwords, dedicated VLANs and dedicated special protedted systems that can access these VLANs.

 

And yes, object storage - especially with immutability - gets more and more interesting. Prices are decreasing and the handling is easier and more comfortable.

Userlevel 7
Badge +17

I am still not sure if I get your problem completely.

It is the main purpose of a tape library to cange tape for one or more tape drives on demand of a backup application or something similar.

Most tape libraries have a so-called I/O station. Here are tapes deposited that are checked our from the library. From this an operator can collect the tapes and put them for example into a safe.

The checked-out tapes in the I/O station cannot checked-in again without manual intervention of an operator.

He would put tapes that should be checked-in into the library into the I/O station.

 

There are some vendors who have some extra secured mechanisms for checked-out tapes - e.g. Active Vault from Quantum. https://www.quantum.com/en/products/tape-storage/security-framework/

Userlevel 7
Badge +8

Like Joe suggested some vendors have some extra features. ActiveVault is one of them, from my pov is the first step.

Here is the clear documentation: TECH BRIEF: SCALAR RANSOM BLOCK AND ASSOCIATED FEATURES FOR CYBER PROTECTION (allbound.com)

To check data integrity on tape, you can use an edlm drive. You must define a policy check one time per month/year...

Scalar-EDLM-Datasheet-DS00513.pdf (allbound.com)

Userlevel 7
Badge +17

I have tested a VTL some years ago (not with Veeam, but with another backup product).

The functionality is very similar, tthis s no problem.

Our painpoints were:

  • there is no physical separation of the media like with physical tape libs.
  • the throughput was not the same. With a physical library each tape drive has a own fibre channel connection. With a VTL all drives share the network connection(s) - this is much less in most cases, at least when you have a library with several drives.
Userlevel 7
Badge +7

I just set up CentOS with Quadstor VTL so I have tape server on Linux for v12 and Chapter 4 of my book. 😎

Nice One ! For test environnement it’s a good idea 😄

Userlevel 7
Badge +7

No I was just thinking something like it auto ejects in a tub, that way the following day’s isn’t blocked, or if the tape drive was specifically made where the mechanism was strong enough for 1 tape to push another out of the way. Then for putting it back it, someone does put it back in manually, but it would be nice if they had a grace period of a couple of days. That way on the weekend for example, no one has to come into the datacenter. I haven’t ever used a tape library myself, just 1 tape drive, so I don’t know if your standard tape library, like a Dell ml3 would be able to give you a grace period for refill. If anyone could give me a brand and example of that, then that would be nice. This is probably the main reason I am steering away from a physical tape library, it’s got to be easy to use so the effectiveness of the plan is high. 

I’ve not come across anything that will auto-eject a tape into a Tub. Mostly done by a robot otherwise, you are risking damage to a tape. 

 

Userlevel 2

I don’t see how the “ActiveVault” is any different from the concept of VTL. Based on watching their video on how ActiveVault works, at the end they indicate an Administrator can go into the software and move it back out, that isn’t air-gapped. Their “air-gapped” claim is a marketing gimic. While it may be virtually air-gapped from the application (e.g. Veeam), a malicious actor can still get access to it via the tape library admin console and cause harm.

As for the Ransom Block, this is also a gimic. It is literally just ejection, nothing special, however it seems it ejects multiple drives at once, the whole magazine, so now you have to push it back in every day to handle the next day’s backup. Unless you could leave it in the ejected state, and newly backed up tapes can still be offloaded to that “partially ejected” magazine. I assume since they don’t mention that you can keep adding to the “partially ejected" magazine that you can’t, which makes it a general pointless feature in my opinion. For example, how often will I know I am about to be hit by ransomware and I am also fast enough to beat the malicious actor and log in and enact the “Ransom Block”? 



I guess another way to phrase what I am looking for, which would make me choose Physical Tape Library over Virtual Tape Library, is PTL having 1 magazine for input only and another magazine for output only. 

The “Logical Tape Blocking” reads like what I want, but if it is software based, then they could find a vulnerability. It says you can only undo it at the local console, but it would require more knowledge on how exactly that is enforced, whether it be something mechanical or not. I might just have to call them on that since their documentation is geared to marketing not technicality.


 

 

At the moment, here would be my reasons for going with VTL, assuming nothing changes my mind in the next couple of days of research (hopefully this helps OP and anyone else…..feel free to pick these apart):

  • Costs variance compared to physical tape infrastructure is not large. Could be cheaper and could be slightly more. All matters on what hardware is chosen. So no real clear winner to me. (e.g. VTL could be near $0 for me as I can use re-purposed and QuadStor)
  • Excluding physical tape ejection, and depending on what VTL software you choose, you can match all features of a Physical Tape Library and maybe even have more, including syncing between 2 sites at faster speeds.
  • No physical interaction needed unless you add any copies to actual physical tape, which QuadStor can do.
  • Security wise, I can lock it down the same way I would the Physical Tape Library. Firewall walls, access-list rules in switch/router, no remote access to console, restrict physical access to the local console, host server for console no on the domain, etc.
  • Potential flexibility of upgrading system in the future, where you don’t have to worry about aging tape technology. (Assuming you don't put anything on tape for super long term). For example, if the physical server crashes there are more ways to quickly get a new system up (e.g. old re-purposed hardware instead of having to buy a new device, virtualizing potentially).
Userlevel 7
Badge +8

Thanks @Scott for the list.
@dips I wasn’t exactly thinking a drop of 6 ft or anything. I was thinking more of a an inch drop (I thought tapes were less susceptible to damage because there is nothing mechanical in them?) or even rig a slide to a bucket (blanket lined !!! hah) to reduce or eliminate any possible damage (e.g. like a tray a cartridge at an incline and it slowly filles up). 

In all my years working at IBM as a service tech, I did not see this exist :)

I picture a laundry hamper with cloths in it and a tape failing ever so soft onto it.  Cut away to a fabric softener commercial.

 

It seems like you have done your research. The thing you need to decide is what's best for you and what risk is ok for you. 

 

You could run backups and export tapes multiple times a day. Is it reasonable? probably not.

 

Can you afford to pay someone to be managing tapes all day? 

 

Some export weekly, monthly, every ¼. It all depend on your personal RTO/RPO, budget, staffing etc.

 

I used to walk in businesses where someone's job was to sit there, and every so often eject tapes. 

 

I also mentioned site security. Sure the tapes are in the library, but that is why you have multi sites with security… you are so focused on someone standing at the machine with console access. 

 

Lets be real here. If I am in your datacenter, standing at your tape library, and have LOGGED IN with admin access. It is too late for you.  The fact someone even getting to the machine should set alarms and have police there immediately. 

 

On that same example, someone could just light the building on fire. some things you have to accept. This is why we have 3-2-1 rule and 1 copy at another site. Cloud is your friend for that and immutable object storage. 

 

 

 

 

 

 

 

 

Userlevel 2

I’m still catching up on this thread so apologies if you have already motioned it. Have you looked into, say Amazon Storage Gateway with Virtual Tapes? 

If you are talking to me and not OP, we have not. Admin will not let us go cloud no matter the benefit. So it is on-prem Tape Library for us as an option only. I am also trying to look at On-Prem S3 Object storage or Veeam SOBR, but running into lack on good info on the former and still researching the latter and how it can help us.


New question relating to all this, since this thread is poppin’ as the ids would say and we see to have a few Veeam experts in here (not me), as VTL can sometimes enable immutability…..if you consider tape that is ejected 100% secure (yes I know), what % secure comparatively would you consider a WORM tape left in the tape library and what % secure would you consider the general idea of data being stored on an immutable store like AWS, S3 compatible, Linux Hardened Repo, SOBR Veeam Archive Tier setup with one of the previous mentioned technologies or something else?

Userlevel 7
Badge +17

Checkout this doozy of a claim I just came across from Starwind Software, located (“https://www.starwindsoftware.com/features#vtl”):
“Ransomware can't encrypt tape libraries”

Mhhh, technically is this correct. They will not encrypt the data on the tapes, they will more likely delete it….

The result is the same 😆 but the marketing claim is different.

Userlevel 7
Badge +17

OK 😎

Yes, tapes in a library can be attacked by malware. Normally they are deleted, not encrypted. It takes a little bit longer to accomplish than with a filesystem, but it is possible. The attacker can access as much tapes as your number of drives at a time. So, you have a chance to be alerted of unusual activity by a monitoring system and rescue the remaining tapes.

This is my main pain point with VTL. If the attacker gains access to the server the VTL runs on, he can simply delete the filesystem or the volume the virtual tapes reside on. In this case immutability does not protect you. It’s dome in seconds.

 

The tapes are out of reach of an attacker in the case you check-out them out of the library and put them somewhere else. In this case the attacker would have to hack your tape movement process which includes manual intervention of human beings. In this case I would hope that is some “natural intelligence” is part of the process, when suddenly much more tapes are requested to be checked-in into the library.

Userlevel 7
Badge +8

Last time I touched a VTL was doing hardware support on an IBM Mainframe haha

Userlevel 7
Badge +20

Yes, all of these are really great resources.  I actually started looking at them this week. 😋

Comment