• EN

Veeam v11 - Hardened Repository aka Immutable backups

Show first post

88 comments

BertrandFR
Userlevel 7
Badge +8

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

S
Userlevel 1

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 

BertrandFR
Userlevel 7
Badge +8

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?


Maybe i can have the use case due to  the CSO demands :grin:

 

 

vNote42
Userlevel 7
Badge +13

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

Thanks for your feedback!

This could be done by PowerShell: Running such a script daily and send a mail, when disabled.

Basically: make sure your access to the Linux-repo server is as secure as possible! Good idea to also set MFA for VBR server!

As I described, when i-flag is removed on file-level, VBR does not delete these files before immutable-interval ended. So this is not a problem. The problem here is: when a user gets that far, he is able to delete the files manually.

 

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?

Wolfgang | vnote42.net | @vNote42
A
Userlevel 1

In my case I'd say 90% of my customers are currently using Windows ReFS to store backup files as their main repository, so I have some questions:

  1. Do you know if Microsoft is working to provide something equivalent to the  XFS “i” flag feature in ReFS?. For 100% Microsoft shops every time you mention Linux is still something they tried to avoid as much as they can. I can envision a lot of resistance if we are talking about the main repository in this specific case
  2. In terms of performance / space saving have Veeam done any lab test to compare XFS vs ReFS?
  3. For customers that will be willing to migrate their main repository from ReFS to XFS to take advantage of this feature: any tips, best practices? It would be great if Veeam provides a Whitepaper regarding this

I think immutable backups is the top driver to adopt V11 in the short term for a lot of customers

A
Userlevel 1

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

BertrandFR
Userlevel 7
Badge +8

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

vNote42
Userlevel 7
Badge +13

[Update]

In the course of hardened repository, Veeam changed the way how to handle retention in backup copy jobs. In short: with v11, backup copy jobs retentions works as forward incremental with synthetic fulls! So keep in mind to add weekly backups to keep your backup chains short.

See this post for more information:

 

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

I would recommend to use Restore Point Simulator (https://rps.dewin.me/). Put in data for your backup jobs and adjust “Retention Points”  like interval for immutable backups.

Wolfgang | vnote42.net | @vNote42
I
Userlevel 1

Very good to have this feature, just wondering how best to spread retention and calculate space requirements

vNote42
Userlevel 7
Badge +13

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

Never trust a guy who wants to enter your datacenter with this tool

 

Wolfgang | vnote42.net | @vNote42
MicoolPaul
Userlevel 7
Badge +20

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
vNote42
Userlevel 7
Badge +13

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

Wolfgang | vnote42.net | @vNote42
R
Userlevel 2

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

vNote42
Userlevel 7
Badge +13

Yeah….  can we just start respecting the native immutability already present on certain Dedupe appliances?

they also have some “advantage” in providing a proprietary access. As long as ransomware is not able to use this, it increases security.

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

Wolfgang | vnote42.net | @vNote42
regnor
Userlevel 7
Badge +14

Having an insider in your environment who has enough time to prepare his attack is always bad. The immutable repository will be only a part of the complete solution. I would still utilize offline or secure offsite backups. Also monitoring your environment for changes will be a good idea in order to catch such events.

 

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
M
Userlevel 1

Yeah….  can we just start respecting the native immutability already present on certain Dedupe appliances?

R
Userlevel 2

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

vNote42
Userlevel 7
Badge +13

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

A, pretty nice! t's simpler than I imagined...

Yes, it just works :joy:

Wolfgang | vnote42.net | @vNote42
wolff.mateus
Userlevel 7
Badge +11

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

A, pretty nice! t's simpler than I imagined...

IT Lover | Veeam Vanguard | VUG Leader | Blogger
vNote42
Userlevel 7
Badge +13

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

Wolfgang | vnote42.net | @vNote42
wolff.mateus
Userlevel 7
Badge +11

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

IT Lover | Veeam Vanguard | VUG Leader | Blogger
vNote42
Userlevel 7
Badge +13

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Wolfgang | vnote42.net | @vNote42

Comment


Terms & Conditions