Question

Immutability flag for copy to S3-compatible (Wasabi) is much longer than what is set in the Repository


Userlevel 7
Badge +6

I’ve come across this twice now, and I’m curious if any of you have seen it.  

I have a relatively new client that we setup backups for with a copy job to run to a Wasabi bucket.  I have set the repo for 30 days of immutability.  I’m having issues with one of the VM’s being backed up and decided to just blow it away.  Backups are fine, but the copy job has failed on this VM for a couple of months (the newest restore point is from the end of November) due to an odd error when Application Aware Processing is enabled.  Since AAP isn’t really needed for this VM, my plan was to  work around this error and move this VM to a separate backup and copy job where AAP is not enabled but moving to the new job failed.  So I tried to delete the data, and that also failed.  I get the error that the data is immutable until October.  What am I missing here?

 

 

 


24 comments

Userlevel 7
Badge +17

Immutability retention ends 30 days after the last restore point in the active backup chain.

Update: Corrected my statement above 

Userlevel 7
Badge +17

See here Derek:

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository_immutability.html?zoom_highlight=Immutability&ver=120

Userlevel 7
Badge +22

Also Wasabi has some specifications as well:

 

https://knowledgebase.wasabi.com/hc/en-us/articles/360058734492-How-does-Wasabi-s-minimum-storage-duration-policy-work-

Userlevel 7
Badge +6

Immutability retention ends 30 days after the last restore point in the active backup chain.

Update: Corrected my statement above 

Thanks, I had assumed you meant 30 and cited 10 based on the article, not based off of what I have immutability set for.  With that said, 30 days after the last restore point would have been in December, not October 10 months from now, right?

Userlevel 7
Badge +17

Not necessarily. Immutability is for the amount of days configured after the last RP in the ACTIVE chain. An active chain is defined as a Full and its corresponding Increments. This doesn't mean the same as full restore points configured. 

Userlevel 7
Badge +6

Reading the retention scenarios, it looks like Veeam could be setting the immutable period for full backup’s to comply with the retention policy, but it looks like both my full and incremental backup files have the immutability flag set for October 2nd.  It’s a possibility that I had the rentention period set for 1 year initially, but that still doesn’t make sense since the last restore point is 11/30/23 and Veeam is reporting the files immutable until 10/2/24.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository_immutability.html?zoom_highlight=Immutability&ver=120#retention

Also, is it safe to assume, since this article addresses VHR, that it applies also to object?

Userlevel 7
Badge +6

Not necessarily. Immutability is for the amount of days configured after the last RP in the ACTIVE chain. An active chain is defined as a Full and its corresponding Increments. This doesn't mean the same as full restore points configured. 

Ah, yes, thanks for pointing that out.  That makes more sense.

Userlevel 7
Badge +17

So, for ex, if you have 30 RPs configured and do weekly synthetic fulls, then the active chain would be the current week's full & it's Increments. Make sense? 

Userlevel 7
Badge +17

I'm went through storage issues on a brand new array right now because I also miscalculated this same thing 🤷

Userlevel 7
Badge +6

 

Thanks Geoff, this is a good point, but I don’t think it applies here.  We were initially using pay-go but I had support convert my account from 90 days to 30 days at the beginning, and we have since converted to RCS...although, I need to go back and look but the date looks suspect to approximately when we converted to RCS.  Still, even 90 days immutability wouldn’t push me out to October.

Userlevel 7
Badge +6

So, for ex, if you have 30 RPs configured and do weekly synthetic fulls, then the active chain would be the current week's full & it's Increments. Make sense? 

Yep, that part totally makes sense.

Userlevel 7
Badge +22

 

Thanks Geoff, this is a good point, but I don’t think it applies here.  We were initially using pay-go but I had support convert my account from 90 days to 30 days at the beginning, and we have since converted to RCS...although, I need to go back and look but the date looks suspect to approximately when we converted to RCS.  Still, even 90 days immutability wouldn’t push me out to October.

good point. I know some people here had some issues and the support folks at Wasabi were very responsive… and yeah October :) I keep thinking it is 2023 fall still 

Userlevel 7
Badge +6

Reading the retention scenarios, it looks like Veeam could be setting the immutable period for full backup’s to comply with the retention policy, but it looks like both my full and incremental backup files have the immutability flag set for October 2nd.  It’s a possibility that I had the rentention period set for 1 year initially, but that still doesn’t make sense since the last restore point is 11/30/23 and Veeam is reporting the files immutable until 10/2/24.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository_immutability.html?zoom_highlight=Immutability&ver=120#retention

Also, is it safe to assume, since this article addresses VHR, that it applies also to object?

 

To clarify a bit, I’m looking at this line regarding retention as I am using GFS retention.

 

  • Veeam Backup & Replication compares the immutability period of the backup repository and the GFS backup file lifetime, and sets an immutability period for full backup files with GFS retention policy as equal to the longest of these periods. For example: the backup repository immutability period is 10 days; the GFS backup file lifetime is 3 years; the backup file will be immutable for 3 years; the increments from this full backup file will be immutable for 10 days from the moment of the last increment creation.
Userlevel 7
Badge +6

 

Thanks Geoff, this is a good point, but I don’t think it applies here.  We were initially using pay-go but I had support convert my account from 90 days to 30 days at the beginning, and we have since converted to RCS...although, I need to go back and look but the date looks suspect to approximately when we converted to RCS.  Still, even 90 days immutability wouldn’t push me out to October.

Right….so….I checked my emails and my confirmation email from Wasabi about converting to RCS was received on 10/2/2023, so I wonder if that could have caused some sort of issue where it set those files to be immutable to 10/2/2024 at that point?  Seems like a strange coincidence….

Userlevel 7
Badge +17

Interesting indeed. Worth verifying I think. 

Userlevel 7
Badge +6

I have sent an email to Wasabi support and will report back what we find.  This might be troublesome if data can’t be deleted across all of our buckets for a year for those buckets that have less retention.

Userlevel 6
Badge +3

@dloseke there’s a great post by @rhys.hammond/@rhyshammond about this Understanding Immutability Periods for GFS Backups | rhyshammond.com

Userlevel 7
Badge +20

To confirm, do you have a GFS backup in Wasabi that is set for a year immutable? Veeam won’t report every date of every object, so as you’ve said a group of backups you want to delete it’s just giving you the furthest out I believe. Use an object browser to see if there’s any granularity in the acceptable deletion dates of the objects whilst you’re waiting for Wasabi to get back to you 🙂

Userlevel 7
Badge +6

To confirm, do you have a GFS backup in Wasabi that is set for a year immutable? Veeam won’t report every date of every object, so as you’ve said a group of backups you want to delete it’s just giving you the furthest out I believe. Use an object browser to see if there’s any granularity in the acceptable deletion dates of the objects whilst you’re waiting for Wasabi to get back to you 🙂

GFS is actually set to keep yearly for 3 years.  So again, not making sense to me.

Userlevel 7
Badge +20

😆 oh boy, some object investigation is needed then!

Userlevel 7
Badge +17

I have sent an email to Wasabi support and will report back what we find.  This might be troublesome if data can’t be deleted across all of our buckets for a year for those buckets that have less retention.

Yes...keep us posted!

Userlevel 7
Badge +17

Hey Derek...did you hear back from Wasabi? Curious what they said.

Userlevel 7
Badge +6

Still investigating.  I just pulled away to other issues and am just getting a chance to circle back around on this.

Userlevel 7
Badge +17

Ah ok. Was wondering about this the other day. Thanks for the update Derek.

Comment