• EN

Historical archive of deleted VM's.

Scott
Userlevel 7
Badge +8

I was wondering if anyone else does this.

When we delete a VM, I often have someone come ask me where it was 6 months to a year later and sometimes there is something unnoticed that is broken. 

Due to the amount of storage we use here (PB’s) I tend to not want to keep all these old VM’s powered off, or even on the Veeam SANs as they take up a ton of space for something that most likely isn’t needed.

 

To resolve this I created a media pool on both tape libraries that has a retention of 2 years. When I power the VM off for the final time I do a Veeam zip, then add it to a file to tape job at both sites.

In the file to tape job I just grab the VBK from the repo and then can remove the Veeam ZIP when it’s done.  Even though it’s not 3-2-1 technically the VM’s are requested to be deleted anyways. If I lost them it wouldn’t be critical. Having them on tape is cheap, and people are very happy when they can get their data.

 

 

 

 

 

 

30 comments

Scott
Userlevel 7
Badge +8

The growth is exponential too. Just think how much video is created from peoples NVR systems, Wyze cameras etc. When crimes happen, those all have to get added to the case and kept for a long long time. Now we have 4k, 8k video.   what's next, 16k? 32k? holograms? haha It will happen.

 

Storage requirements get larger and more expensive. Cloud companies are laughing these days and are all for it to.

 

At the end of the day, I am a hands on specialist and can only give so much advice/direction which I have many times. I will do as I am told so my employer will continue to employee me. 

 

It’s a good thing I have a lot of tape storage. but every few years switching LTO generations means migrating it, Restores can get tedious. but at this rate, cloud and object storage for something that is going to be that old are not worth it. 

 

 

BertrandFR
Userlevel 7
Badge +8

It is an interesting debate that we have, we can see differents opinions/use case and regulations of each country. At the European level, retention is more regulated since the emergence of GDPR. There may be subtleties for the justice and police part for example, the majority of use case could be done in DB by business apps. I insist on it but it was a nice battle with the projects to convince them that the backup should not be in adhesion for this archiving, specially when they don’t pay for the backup service.

Example for logs collecting and sequestration, i prefer to handle this by a distinct part. Example you can push you data directly to an object storage with tiering and ensure legal hold by the bucket configuration...

If necessary it seems to me to have a wise discussion with the DPO, without forgetting that one moment it will be necessary to evoke the technical aspect and especially financial constraints.

JMeixner
Userlevel 7
Badge +17

Unfortunately you are right that not all regulations make sense.

And even more unfortunately it is expected in Germany that these data is readable in 30 years. I am very sceptical about this….

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
Scott
Userlevel 7
Badge +8

If you charge for the service, why not good for you, but it is abnormal if the business application uses the backup as an archive. There is a small problem of application architecture.

So the IT department and team that does the backups, file servers, storage, servers shouldn’t be in charge of the data archives and retention? Even though we use the backup software of Veeam, TSM and cloud storage, and no one else even has credentials to do so?

We are not an MSP, this is our data. The law is that we have to keep it. we are paying for this, not charging. 

 

It’s quite normal in our circumstances to have crazy retention. 

 

I just want it off prod as easy as possible and into Wasabi and tape as the chances of needed to restore it get smaller every year. It still doesn’t change it’s mandated by law. 

 

 

 

OK, my experience is, if you charge internally for the storage consumption, then are suddenly many hard requirement not so hard anymore… 😎

And yes, I agree, some legal requirements make no really sense. perhaps you have the date after all these years, but do you have machines and software to read them and work with them?

Funny you asked this. I said the same thing today. In 40 years will Veeam version 30 be able to read Windows server 2016? who knows. Will that file type of that video still work? 

 

Heck it’s not been too long since VHS and it’s hard to find a VHS player these days, or a Betamax. 

 

However, Those are things I ask to prove a point, but at the end of the day I am told to keep and archive the data for legal reasons. It doesn’t state I have to be able to access the files. Most likely the IT department will be tasked with that too, but in reality, I'll be retired in 20 years and this data will be stale. there is a large chance most of it is garbage, but not my place to say. 

 

No one said all laws made sense, or were even productive.  I have been pushing for retention policies though and getting a BIT of traction. The price tags on PB’s of disk, cloud, archive, backup, etc have helped.

 

 

Scott
Userlevel 7
Badge +8

With our 50-100 year retention policies for some stuff it gets pretty insane to keep this stuff on our SANS.

 

Catching up on some old posts/tabs I forgot about, but I have to ask...what kind of data requires a 50-100 year retention policy?  The only thing I can think of is government records, like property records and such….

 

I work for government, and lots to do with things such as policing, homicides etc. 

dloseke
Userlevel 7
Badge +6

With our 50-100 year retention policies for some stuff it gets pretty insane to keep this stuff on our SANS.

 

Catching up on some old posts/tabs I forgot about, but I have to ask...what kind of data requires a 50-100 year retention policy?  The only thing I can think of is government records, like property records and such….

Derek M. Loseke, Senior Systems Engineer | Veeam Legend 2022-2023 | VMSP/VMTSP | VCP6-DCV | VSP/VTSP | CCNA | https://technotesanddadjokes.com | @dloseke
JMeixner
Userlevel 7
Badge +17

If you charge for the service, why not good for you, but it is abnormal if the business application uses the backup as an archive. There is a small problem of application architecture.

So the IT department and team that does the backups, file servers, storage, servers shouldn’t be in charge of the data archives and retention? Even though we use the backup software of Veeam, TSM and cloud storage, and no one else even has credentials to do so?

We are not an MSP, this is our data. The law is that we have to keep it. we are paying for this, not charging. 

 

It’s quite normal in our circumstances to have crazy retention. 

 

I just want it off prod as easy as possible and into Wasabi and tape as the chances of needed to restore it get smaller every year. It still doesn’t change it’s mandated by law. 

 

 

 

OK, my experience is, if you charge internally for the storage consumption, then are suddenly many hard requirement not so hard anymore… 😎

And yes, I agree, some legal requirements make no really sense. perhaps you have the date after all these years, but do you have machines and software to read them and work with them?

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
BertrandFR
Userlevel 7
Badge +8

It was just my POV, it depends on many factors (opinion, internal organization, use cases). I'm not a huge fan of using backup to archive when depending on the use case it can be managed in a database.

I didn't say that it was your case but I have often come across these cases. I did not want to create any misunderstanding :)

Scott
Userlevel 7
Badge +8

If you charge for the service, why not good for you, but it is abnormal if the business application uses the backup as an archive. There is a small problem of application architecture.

So the IT department and team that does the backups, file servers, storage, servers shouldn’t be in charge of the data archives and retention? Even though we use the backup software of Veeam, TSM and cloud storage, and no one else even has credentials to do so?

We are not an MSP, this is our data. The law is that we have to keep it. we are paying for this, not charging. 

 

It’s quite normal in our circumstances to have crazy retention. 

 

I just want it off prod as easy as possible and into Wasabi and tape as the chances of needed to restore it get smaller every year. It still doesn’t change it’s mandated by law. 

 

 

 

BertrandFR
Userlevel 7
Badge +8

If you charge for the service, why not good for you, but it is abnormal if the business application uses the backup as an archive. There is a small problem of application architecture.

Scott
Userlevel 7
Badge +8

Wow, it doesn’t make sense… Sometimes reason does not prevail over intriguing customer demand, i don’t want to be in a brain of a DPO who asks for this.

Some of it has to do with criminal activity so the legal requirements are set by government. It does make sense once you are on this side, but it’s difficult because every year data gets bigger and bigger

 

BertrandFR
Userlevel 7
Badge +8

Wow, it doesn’t make sense… Sometimes reason does not prevail over intriguing customer demand, i don’t want to be in a brain of a DPO who asks for this.

Scott
Userlevel 7
Badge +8

Well, it depends of the need if necessary i will quickly discuss about it with the requestor. My BaaS has 6 months retention on tape for Production so normally that’s enough. If duly justified we can make an exception for longer retention with a Veeam Zip like suggested before :)

Lucky. the 60 year and 100 year legal requirements on some of the large stuff I have is crazy. And because of that everyone figures they have to save EVERYTHING. 

BertrandFR
Userlevel 7
Badge +8

Well, it depends of the need if necessary i will quickly discuss about it with the requestor. My BaaS has 6 months retention on tape for Production so normally that’s enough. If duly justified we can make an exception for longer retention with a Veeam Zip like suggested before :)

marco_s
Userlevel 7
Badge +7

I was joking Hannes! :)

In any case, the request is not for me or for a specific person, but it is to improve the product, so it is for everyone..

We are all here because we love veeam, the goal is common!

Marco Sorrentino - Italy | System Engineer | VMCE & Veeam Lover
H
Userlevel 5
Badge +2

Hannes is not taking us seriously in the forum!  :-)

 

hmm, why do you think so? The first request can make sense, but it adds complexity with hard to estimate benefit and currently only one customer asking for it. So let’s see how many others ask for features like that

the second thing is a different topic / technology and it’s confusing for every reader to have two topics / discussions mixed in one thread :-)

Scott
Userlevel 7
Badge +8

 

 

 

Hannes is not taking us seriously in the forum! 😀

HAHA

 

Quite the opposite actually. I have had amazing help from those guys. 

marco_s
Userlevel 7
Badge +7

I have already requested both the VM backup, and the ability to do a similar thing for files in the forums a while back, but more requests can actually help make things happen.

 

A big one for us is our retention. We have to keep 100’s of TB’s of data for 60-100 years.    I’d love to “archive” files to 2 locations, (tape and Wasabi) then delete it off of the production file share. 

 

Obviously Veeam don’t want to be responsible for the deletion so if that is manual, or a script on our end I’d understand that. but Having some sort of file tree of my “archived” files under another menu would be so good. 

 

This is something we would invest in VUL licenses for even. I don’t need full file shares backed up with NAS backup as we can’t afford that vs regular Veeam backups, but this non prod stuff is costing a ton in storage for data that is not used often.

 

I checked on a few of our 30TB servers and the change rate is a few gig a day lol. another one was in the MB range.

 

 

Hannes is not taking us seriously in the forum! 😀

Marco Sorrentino - Italy | System Engineer | VMCE & Veeam Lover
Scott
Userlevel 7
Badge +8

I have already requested both the VM backup, and the ability to do a similar thing for files in the forums a while back, but more requests can actually help make things happen.

 

A big one for us is our retention. We have to keep 100’s of TB’s of data for 60-100 years.    I’d love to “archive” files to 2 locations, (tape and Wasabi) then delete it off of the production file share. 

 

Obviously Veeam don’t want to be responsible for the deletion so if that is manual, or a script on our end I’d understand that. but Having some sort of file tree of my “archived” files under another menu would be so good. 

 

This is something we would invest in VUL licenses for even. I don’t need full file shares backed up with NAS backup as we can’t afford that vs regular Veeam backups, but this non prod stuff is costing a ton in storage for data that is not used often.

 

I checked on a few of our 30TB servers and the change rate is a few gig a day lol. another one was in the MB range.

 

 

Chris.Childerhose
Userlevel 7
Badge +20

@marco_s - that is a good suggestion and would recommend adding it to the Veeam Forums - https://forums.veeam.com/

Also, Tape is another great solution for this as well as the direct to object.  To me the direct to OBS is better since tapes would need to be tested more frequently to ensure it does not get corrupted.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
regnor
Userlevel 7
Badge +14

Storing such VMs/archives on a tape sounds like a perfect solution. As an alterantive with v12 and it’s direct to object storage capability, I would create a one-time job and upload such VMs to an object storage of choice; most likely Wasabi.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
marco_s
Userlevel 7
Badge +7

The best could be a feature mixing “remove deleted data after” and SOBR: if we use capacity/archive tier, Veeam can let use to choose if we would automatically offload a deleted item e retain until the specified days set in the maintenance tab..what do you think?!

Marco Sorrentino - Italy | System Engineer | VMCE & Veeam Lover
Chris.Childerhose
Userlevel 7
Badge +20

VeeamZIP is your friend for sure.  We do this with every cleanup of any VMs is VeeamZIP them first and store them off on to different storage.  Keeps us sane so we can do that restore if needed. 😋

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Iams3le
Userlevel 7
Badge +9

Great point @Scott! You are doing it correctly with Veeam ZIP. Here is a fantastic guide from @Nico Losschaert on this topic. There are other ways to achieve this without Veeam as well. 

Thx @Iams3le 😍

You are welcome😊

[Microsoft MVP | Veeam Legend 5* | VMware vExpert 4* | Cisco Champion 3* | Object First Ace]
Scott
Userlevel 7
Badge +8

Great point @Scott! You are doing it correctly with Veeam ZIP. Here is a fantastic guide from @Nico Losschaert on this topic. There are other ways to achieve this without Veeam as well. 

Very similar. I use the EXPORT too 

Comment


Terms & Conditions