Veeam v11 - Hardened Repository aka Immutable backups



Show first post

88 comments

Userlevel 7
Badge +3

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

Userlevel 1

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 

Userlevel 7
Badge +3

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?


Maybe i can have the use case due to  the CSO demands :grin:

 

 

Userlevel 7
Badge +7

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

Thanks for your feedback!

This could be done by PowerShell: Running such a script daily and send a mail, when disabled.

Basically: make sure your access to the Linux-repo server is as secure as possible! Good idea to also set MFA for VBR server!

As I described, when i-flag is removed on file-level, VBR does not delete these files before immutable-interval ended. So this is not a problem. The problem here is: when a user gets that far, he is able to delete the files manually.

 

Userlevel 7
Badge +7

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?

Userlevel 1

In my case I'd say 90% of my customers are currently using Windows ReFS to store backup files as their main repository, so I have some questions:

  1. Do you know if Microsoft is working to provide something equivalent to the  XFS “i” flag feature in ReFS?. For 100% Microsoft shops every time you mention Linux is still something they tried to avoid as much as they can. I can envision a lot of resistance if we are talking about the main repository in this specific case
  2. In terms of performance / space saving have Veeam done any lab test to compare XFS vs ReFS?
  3. For customers that will be willing to migrate their main repository from ReFS to XFS to take advantage of this feature: any tips, best practices? It would be great if Veeam provides a Whitepaper regarding this

I think immutable backups is the top driver to adopt V11 in the short term for a lot of customers

Userlevel 1

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

Userlevel 7
Badge +3

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Userlevel 7
Badge +7

[Update]

In the course of hardened repository, Veeam changed the way how to handle retention in backup copy jobs. In short: with v11, backup copy jobs retentions works as forward incremental with synthetic fulls! So keep in mind to add weekly backups to keep your backup chains short.

See this post for more information:

 

Userlevel 7
Badge +7

I would recommend to use Restore Point Simulator (https://rps.dewin.me/). Put in data for your backup jobs and adjust “Retention Points”  like interval for immutable backups.

Userlevel 1

Very good to have this feature, just wondering how best to spread retention and calculate space requirements

Userlevel 7
Badge +7

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

Never trust a guy who wants to enter your datacenter with this tool

 

Userlevel 7
Badge +8

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

Userlevel 7
Badge +7

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

Userlevel 2

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

Userlevel 7
Badge +7

Yeah….  can we just start respecting the native immutability already present on certain Dedupe appliances?

they also have some “advantage” in providing a proprietary access. As long as ransomware is not able to use this, it increases security.

Userlevel 7
Badge +7

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

Userlevel 7
Badge +6

Having an insider in your environment who has enough time to prepare his attack is always bad. The immutable repository will be only a part of the complete solution. I would still utilize offline or secure offsite backups. Also monitoring your environment for changes will be a good idea in order to catch such events.

 

Userlevel 1

Yeah….  can we just start respecting the native immutability already present on certain Dedupe appliances?

Userlevel 2

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

Userlevel 7
Badge +7

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

A, pretty nice! t's simpler than I imagined...

Yes, it just works :joy:

Userlevel 7
Badge +6

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

A, pretty nice! t's simpler than I imagined...

Userlevel 7
Badge +7

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

Userlevel 7
Badge +6

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Userlevel 7
Badge +7

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Comment