Skip to main content
  • EN

I spent some time in testing a new great feature in v11: Hardened Repository. Read here about:

  • What is immutability is about
  • Requirements
  • Setup
  • How it works

 

What is immutability is about

Immutability in this context means, a backup file cannot be changed or deleted without having root access within hosing Linux OS. So, even the backup administrator is not able to delete backups on such a repository.

Why is this important? Think about ransomware. This software is that smart these days, it is able to recognize backup systems. It can trigger tasks like deletion of backup files. But when they are immutable, this cannot be done!

 

Requirements

What is needed to get immutable backups? First of all: Veeam Backup&Replication v11. This is the first version supporting hardened repositories. Secondly a Linux server hosting repository volumes.

What about the filesystem? Veeam is using immutable flag. So every filesystem supporting this flag can be used. These are pretty much all. Veeam supports reflink/Fast Clone on XFS. Because of this XFS is the recommended filesystem.

What about the distribution? At the moment of writing I had no information about this. I think this feature will not constrain the selection of supported Linux distributions. When using XFS, we get a fist choice: Ubuntu 20.04 LTS (long-term support). Because: (1) Ubuntu is supported by Veeam. (2) 20.04 uses kernel version of 5.4. This version seems to provide highest quality of reflink, tested by Veeam.

Thirdly: Backup chains must be compatible with immutable files. What does this mean? Because files cannot be changed, the backup chain only can create new files without changing any of the existing. Only forward incremental with periodic synthetic or active fulls fulfill this requirement. For backup copy jobs, GFS settings are required.

 

Setup

Immutable backups are enabled on repository level. Either at creating the repository. Or for a existing repository. How to setup Linux as repository server I will covered in another post.

Settings are easy to understand:

Immutable backup settings

 

How it (just) works

The beauty of this feature is the use of native filesystem features. In Linux each file can have an attribute i. When this is set, file cannot be changed or removed. When Veeam creates backup files, this flag is set. After the entered period of immutability, flag is removed and file can be deleted.

To see file attributes, including immutable flag, run: lsattr filename in Linux shell. Sample output see here:

Immutable flag set on files

Note, flags are removed from a whole backup chain, not just a single file.

Flag removed after protection period

The question may arise how the flag is set in Linux. Because, when the specified Linux user gets privileged access to add or remove the flag, this could be used by a hacker to get access to these files as well. Right, BUT: flag is not set by this user. Instead it is set by root. This can be done by running a service with root access: veeamimmureposvc:

service to set immutable flag

Notice: this service has no connection to the network, so it cannot be compromised remotely!

Apropos network: What ports are being used? Also new in v11 is that just one port is used to communicate with repository host: TCP/6162. During a backup other ports can be opened on demand.

Open ports with no running job
Open port with running job

The whole blog post, with some more details, you can find here: 

https://vnote42.net/2020/11/23/new-in-veeam-v11-hardened-repository-immutable-backups-part-1/

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

Never trust a guy who wants to enter your datacenter with this tool

 

Very good to have this feature, just wondering how best to spread retention and calculate space requirements

I would recommend to use Restore Point Simulator (https://rps.dewin.me/). Put in data for your backup jobs and adjust “Retention Points”  like interval for immutable backups.

[Update]

In the course of hardened repository, Veeam changed the way how to handle retention in backup copy jobs. In short: with v11, backup copy jobs retentions works as forward incremental with synthetic fulls! So keep in mind to add weekly backups to keep your backup chains short.

See this post for more information:

 

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

In my case I'd say 90% of my customers are currently using Windows ReFS to store backup files as their main repository, so I have some questions:

  1. Do you know if Microsoft is working to provide something equivalent to the  XFS “i” flag feature in ReFS?. For 100% Microsoft shops every time you mention Linux is still something they tried to avoid as much as they can. I can envision a lot of resistance if we are talking about the main repository in this specific case
  2. In terms of performance / space saving have Veeam done any lab test to compare XFS vs ReFS?
  3. For customers that will be willing to migrate their main repository from ReFS to XFS to take advantage of this feature: any tips, best practices? It would be great if Veeam provides a Whitepaper regarding this

I think immutable backups is the top driver to adopt V11 in the short term for a lot of customers

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

Thanks for your feedback!

This could be done by PowerShell: Running such a script daily and send a mail, when disabled.

Basically: make sure your access to the Linux-repo server is as secure as possible! Good idea to also set MFA for VBR server!

As I described, when i-flag is removed on file-level, VBR does not delete these files before immutable-interval ended. So this is not a problem. The problem here is: when a user gets that far, he is able to delete the files manually.

 

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?


Maybe i can have the use case due to  the CSO demands :grin:

 

 

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 

Right! I would also warn against using raw device mappings (RDM) in vSphere! These are physical disk devices mapped directly into VMs. But with this a hacker having access to the hypervisor can also destroy your backup data - no matter if they are immutable or not. 

So when using hardened repository, use a (hardened) physical Linux host! 

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?


Hi,

ok I understand the intention to disable all remote access...

But I don’t have physical access to my backup servers because they are located in datacenters all over Germany and some other european countries.

How do you handle such servers in your environments without remote access?

Ok, I could hire a helping hand every time any action has to be done at a server… But this would be expensive and I don’t think this would  be more secure… :confounded:

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?


Hi,

ok I understand the intention to disable all remote access...

But I don’t have physical access to my backup servers because they are located in datacenters all over Germany and some other european countries.

How do you handle such servers in your environments without remote access?

Ok, I could hire a helping hand every time any action has to be done at a server… But this would be expensive and I don’t think this would  be more secure… :confounded:

To keep remote access more secure, you can implement MFA for Linux. If this is possible for IPMI/iLO/iDRAC (is it?) this would sense too! If not, you could disable network port on the switch and enable it if access is necessary. 

Thanks @vNote42

In my case I'd say 90% of my customers are currently using Windows ReFS to store backup files as their main repository, so I have some questions:

  1. Do you know if Microsoft is working to provide something equivalent to the  XFS “i” flag feature in ReFS?. For 100% Microsoft shops every time you mention Linux is still something they tried to avoid as much as they can. I can envision a lot of resistance if we are talking about the main repository in this specific case
  2. In terms of performance / space saving have Veeam done any lab test to compare XFS vs ReFS?
  3. For customers that will be willing to migrate their main repository from ReFS to XFS to take advantage of this feature: any tips, best practices? It would be great if Veeam provides a Whitepaper regarding this

I think immutable backups is the top driver to adopt V11 in the short term for a lot of customers

@andy40241 sorry for the late response.

  1. I am not aware of Microsoft is planning such feature. In my experience customer minds are open for Linux repositories when they hear about immutable backup files. A lot of them see Linux repositories as storage of a different media in the 3-2-1 rule. If there is no know-how at customer site, I would recommend to buy support for distribution of choice.
  2. I do not know official Veeam publications about performance and/or space savings comparison between ReFS and XFS. I would recommend to check this post for experiences with XFS of the community. Furthermore, Veeam does not make any differences between XFS and ReFS is their official sizer tools. Also for the (unofficial) Restore Point Simulator space savings are the same.
  3. There is also a discussion about this topic in the community. In my opinion the simples way is to fade out restore points on ReFS after starting new backups on XFS.

@JMeixner check links below why disable remote access etc…

https://bp.veeam.com/vbr/VBP/Security/infrastructure_hardening.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_linux.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_windows.html

For windows hosts even hardened who are very important, i break all suspicious processus after some learning with elastic

@JMeixnercheck links below why disable remote access etc…

https://bp.veeam.com/vbr/VBP/Security/infrastructure_hardening.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_linux.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_windows.html

For windows hosts even hardened who are very important, i break all suspicious processus after some learning with elastic

Thanks for the links! Great ressources!

@JMeixner No physical access and no easy hands-on could be challenging. MFA, as @vNote42 says, could be the solution in that case.

Thanks  @regnor and @vNote42  ,

MFA and the activation of network ports when access is neccessary are good mechanisms for my environments.

Thank you.! Is there a step-by-step guide somewhere? I planning to upgrade to V11, and want to set up this hardening as soon as possible

Comment


Terms & Conditions