Veeam v11 - Hardened Repository aka Immutable backups



Show first post

88 comments

Userlevel 7
Badge +13

@JMeixnercheck links below why disable remote access etc…

https://bp.veeam.com/vbr/VBP/Security/infrastructure_hardening.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_linux.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_windows.html

For windows hosts even hardened who are very important, i break all suspicious processus after some learning with elastic

Thanks for the links! Great ressources!

Userlevel 7
Badge +8

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Userlevel 7
Badge +13

I've looked at the hardened repositories in the last days and the configuration itself is really easy. The Linux side is more complicated if you're not used to setup such systems 😅

@vNote42In your blog you've posted a screenshot of a dialog box requesting the change of the directory owner. Is this checkbox no longer existing in the GA release?

Yes, for what you get it is really simple!

You probably mean this dialog-box?

Good question! I think I have not seen the dialog in installations since GA. But this could also be because I ran the command before. Did you run it before?

Userlevel 7
Badge +1

Thanks @vNote42

Userlevel 7
Badge +14

Yes I meant that dialog. I forgot to change the owner and was wondering why Veeam couldn't create the job folders. So the dialog did probably only exist in the beta.

Userlevel 7
Badge +13

Yes I meant that dialog. I forgot to change the owner and was wondering why Veeam couldn't create the job folders. So the dialog did probably only exist in the beta.

Probably! Maybe @Rick Vanover  can answer this question?

Userlevel 1

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

Userlevel 1

In my case I'd say 90% of my customers are currently using Windows ReFS to store backup files as their main repository, so I have some questions:

  1. Do you know if Microsoft is working to provide something equivalent to the  XFS “i” flag feature in ReFS?. For 100% Microsoft shops every time you mention Linux is still something they tried to avoid as much as they can. I can envision a lot of resistance if we are talking about the main repository in this specific case
  2. In terms of performance / space saving have Veeam done any lab test to compare XFS vs ReFS?
  3. For customers that will be willing to migrate their main repository from ReFS to XFS to take advantage of this feature: any tips, best practices? It would be great if Veeam provides a Whitepaper regarding this

I think immutable backups is the top driver to adopt V11 in the short term for a lot of customers

Userlevel 7
Badge +13

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?

Userlevel 7
Badge +13

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

Thanks for your feedback!

This could be done by PowerShell: Running such a script daily and send a mail, when disabled.

Basically: make sure your access to the Linux-repo server is as secure as possible! Good idea to also set MFA for VBR server!

As I described, when i-flag is removed on file-level, VBR does not delete these files before immutable-interval ended. So this is not a problem. The problem here is: when a user gets that far, he is able to delete the files manually.

 

@vNote42 have you been able to post anything on How to setup Linux as repository server?

 

Userlevel 7
Badge +13

[Update]

Veeam Hardened Repository passes independent compliance assessment

When properly configured, the Hardened Repository meets the requirements for non-rewritable, non-erasable storage as specified by SEC 17a-4(f), FINRA 4511(c) and CFTC 1.31(c)-(d) regulations.

https://www.veeam.com/blog/hardened-repository-passes-compliance.html

 

Userlevel 7
Badge +13

@vNote42 have you been able to post anything on How to setup Linux as repository server?

 

Hi @MAC_Daddy_1974 ! Just wrote an internal installation guide. I can recommend this post:

https://nolabnoparty.com/en/veeam-v11-hardened-repository-immutability-pt-1/ by @PValsecchi 

Userlevel 7
Badge +13

Thanks for posting this, interesting. Will Linux be the repo of choice after v11 is released, i think so?

I talk more and more with customers about Linux as repo-server. Yes, most of them are very interested in XFS with immutable backups!

Userlevel 7
Badge +22

Great Information folks this really helps!

Userlevel 7
Badge +20

Nice to see all of this information in one post.  Not sure it is possible but you should see about editing the main post with the updates versus them being within the pages.  Just would make things easier to find all in the first post.  Maybe we don’t have the editing ability either.  LOL

Userlevel 7
Badge +13

Thanks for the great article on immutability! So chattr should be restricted with selinux on linux host, good to know! I will put a ML task specific with elastic auditbeat on it

“ML task specific with elastic auditbeat” sounds interesting! And what I just read it is for sure!

Userlevel 2
Badge +3

You may remove the file /etc/veeamhubrepo to reset the tool. 

May I ask at what stage you run into trouble? I am personally using the tool multiple times per week to provision new repositories for our customers, but any feedback that can improve reliability would be much appreciated.

Userlevel 7
Badge +13

Thank you.! Is there a step-by-step guide somewhere? I planning to upgrade to V11, and want to set up this hardening as soon as possible

I would recommend to start here: https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

There you find the section: Deployment of Hardened Repository.

I am sure more detailed information and best practices for Linux repositories will come soon.

Userlevel 7
Badge +14

I've looked at the hardened repositories in the last days and the configuration itself is really easy. The Linux side is more complicated if you're not used to setup such systems 😅

@vNote42In your blog you've posted a screenshot of a dialog box requesting the change of the directory owner. Is this checkbox no longer existing in the GA release?

Userlevel 7
Badge +14

@vNote42 @regnor Yes anything that has "TBD" Is likely a beta, preview or otherwise non-Generally Available build.

 

If this is in GA, let me know and I will get it into the bug fix cycle.

 

I do not believe I have seen this in GA myself.


Well no it's not in GA but it would be a useful hint especially if you don't look in the documentation.

Userlevel 7
Badge +3

@vNote42 thanks for share and your update 

Userlevel 7
Badge +20

Nice to see all of this information in one post.  Not sure it is possible but you should see about editing the main post with the updates versus them being within the pages.  Just would make things easier to find all in the first post.  Maybe we don’t have the editing ability either.  LOL

Good point chris!

Legends have the permission to edit their own posts only since a few weeks now. I future I will edit the original post to add updates. Thanks!

 

No problem.  Just figured Rick does it for the v11 post he has so it keeps it tidy. :grinning:

Userlevel 2

I’m having problems with the tool, in the begining I was not aware that the xfs progs needed to be installed on my ubuntu version, and the installer failed, well, I just downloaded the xfsprogs and then the veeamhubrepo won’t start again, I deleted the file  ‘/etc/veeamhubtinyrepoman’ and started all over again, but in the end it crashes again with this error:

 

Can any one tell me how to reset the tool to start over clean? Thanx.

 

Userlevel 7
Badge +13

Thanks @Adolfo_Veloz! I think I understand the problem you had. But I did not get the solution. Can you please tell the solution again.

Comment