Veeam v11 - Hardened Repository aka Immutable backups


Userlevel 7
Badge +13

I spent some time in testing a new great feature in v11: Hardened Repository. Read here about:

  • What is immutability is about
  • Requirements
  • Setup
  • How it works

 

What is immutability is about

Immutability in this context means, a backup file cannot be changed or deleted without having root access within hosing Linux OS. So, even the backup administrator is not able to delete backups on such a repository.

Why is this important? Think about ransomware. This software is that smart these days, it is able to recognize backup systems. It can trigger tasks like deletion of backup files. But when they are immutable, this cannot be done!

 

Requirements

What is needed to get immutable backups? First of all: Veeam Backup&Replication v11. This is the first version supporting hardened repositories. Secondly a Linux server hosting repository volumes.

What about the filesystem? Veeam is using immutable flag. So every filesystem supporting this flag can be used. These are pretty much all. Veeam supports reflink/Fast Clone on XFS. Because of this XFS is the recommended filesystem.

What about the distribution? At the moment of writing I had no information about this. I think this feature will not constrain the selection of supported Linux distributions. When using XFS, we get a fist choice: Ubuntu 20.04 LTS (long-term support). Because: (1) Ubuntu is supported by Veeam. (2) 20.04 uses kernel version of 5.4. This version seems to provide highest quality of reflink, tested by Veeam.

Thirdly: Backup chains must be compatible with immutable files. What does this mean? Because files cannot be changed, the backup chain only can create new files without changing any of the existing. Only forward incremental with periodic synthetic or active fulls fulfill this requirement. For backup copy jobs, GFS settings are required.

 

Setup

Immutable backups are enabled on repository level. Either at creating the repository. Or for a existing repository. How to setup Linux as repository server I will covered in another post.

Settings are easy to understand:

Immutable backup settings

 

How it (just) works

The beauty of this feature is the use of native filesystem features. In Linux each file can have an attribute i. When this is set, file cannot be changed or removed. When Veeam creates backup files, this flag is set. After the entered period of immutability, flag is removed and file can be deleted.

To see file attributes, including immutable flag, run: lsattr filename in Linux shell. Sample output see here:

Immutable flag set on files

Note, flags are removed from a whole backup chain, not just a single file.

Flag removed after protection period

The question may arise how the flag is set in Linux. Because, when the specified Linux user gets privileged access to add or remove the flag, this could be used by a hacker to get access to these files as well. Right, BUT: flag is not set by this user. Instead it is set by root. This can be done by running a service with root access: veeamimmureposvc:

service to set immutable flag

Notice: this service has no connection to the network, so it cannot be compromised remotely!

Apropos network: What ports are being used? Also new in v11 is that just one port is used to communicate with repository host: TCP/6162. During a backup other ports can be opened on demand.

Open ports with no running job
Open port with running job

The whole blog post, with some more details, you can find here: 

https://vnote42.net/2020/11/23/new-in-veeam-v11-hardened-repository-immutable-backups-part-1/


88 comments

Userlevel 7
Badge +13

[Important update]

You CAN NOT run immutable repository AND proxy role on the same Linux server in v11!

PS: There is a workaround: you theoretically can run one of these roles in a container.

Userlevel 7
Badge +13

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

Never trust a guy who wants to enter your datacenter with this tool

 

Userlevel 7
Badge +13

[Update information]

Veeam is planning to support fast cloning with Ubuntu LTS 18.04, RHEL/CentOS 8.2 or later, SLES 15 SP2 and Debian 10 as well! For other distributions: support will be experimental. Kernel version 5.4 or later recommended.

So there will be more options when is comes to distribution selection.

Userlevel 7
Badge +13

Update information:

How does it work, when I start now with v10 and XFS repo and want to use immutable backups when v11 is available? 

Short answer by @Gostev :

“After you upgrade to v11, you will only need to run a shell command to change the owner for the existing backup files. There's will be a message with instructions in the user interface if you enable immutable backups option on the existing repository. The rest is optional (additional hardening like disabling SSH server and remote consoles like iLO completely).”

https://forums.veeam.com/veeam-backup-replication-f2/v11-linux-immutability-t70932.html

Userlevel 7
Badge +12

Oh, that’s a limitation... On the other hand it makes sense as you would need root-access for the proxy.

Wouldn’t running one or both roles inside a container open new attack possibilites? 

Userlevel 7
Badge +13

Oh, that’s a limitation... On the other hand it makes sense as you would need root-access for the proxy.

Wouldn’t running one or both roles inside a container open new attack possibilites? 

Yes it is. And it is - from my perspective - no technical limitation, it is for security reasons. 

I would say, you will always take a higher security risk when running a second role on the repository server. When - for example because of a kernel-bug - a user can break out/in a container, your backup data is in danger.

Userlevel 7
Badge +11

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

Userlevel 7
Badge +11

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Userlevel 7
Badge +13

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

Userlevel 7
Badge +8

Thanks for the great article on immutability! So chattr should be restricted with selinux on linux host, good to know! I will put a ML task specific with elastic auditbeat on it

Userlevel 2

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

Userlevel 2

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

Userlevel 7
Badge +13

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

Userlevel 7
Badge +13

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

A, pretty nice! t's simpler than I imagined...

Yes, it just works :joy:

Userlevel 7
Badge +13

[Update]

In the course of hardened repository, Veeam changed the way how to handle retention in backup copy jobs. In short: with v11, backup copy jobs retentions works as forward incremental with synthetic fulls! So keep in mind to add weekly backups to keep your backup chains short.

See this post for more information:

 

Userlevel 1

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 

Userlevel 7
Badge +13

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?


Hi,

ok I understand the intention to disable all remote access...

But I don’t have physical access to my backup servers because they are located in datacenters all over Germany and some other european countries.

How do you handle such servers in your environments without remote access?

Ok, I could hire a helping hand every time any action has to be done at a server… But this would be expensive and I don’t think this would  be more secure… :confounded:

To keep remote access more secure, you can implement MFA for Linux. If this is possible for IPMI/iLO/iDRAC (is it?) this would sense too! If not, you could disable network port on the switch and enable it if access is necessary. 

Userlevel 1

Yeah….  can we just start respecting the native immutability already present on certain Dedupe appliances?

Userlevel 7
Badge +12

Having an insider in your environment who has enough time to prepare his attack is always bad. The immutable repository will be only a part of the complete solution. I would still utilize offline or secure offsite backups. Also monitoring your environment for changes will be a good idea in order to catch such events.

 

Userlevel 7
Badge +10

@vNote42 @regnor Yes anything that has "TBD" Is likely a beta, preview or otherwise non-Generally Available build.

 

If this is in GA, let me know and I will get it into the bug fix cycle.

 

I do not believe I have seen this in GA myself.

Userlevel 6
Badge +1

Excellent work !!! Thanks for sharing.

Userlevel 2

Thanks for posting this, interesting. Will Linux be the repo of choice after v11 is released, i think so?

Userlevel 7
Badge +13

I would recommend to use Restore Point Simulator (https://rps.dewin.me/). Put in data for your backup jobs and adjust “Retention Points”  like interval for immutable backups.

Userlevel 7
Badge +13

Hi 

I am looking to implement hardened repository im already using Veeam v11 however I have zero experience created Linux VM and required config on it.

Would anyone be so kind as to give me some pointers?

Thanks.

 

Check out excellent blog series by @PValsecchi :

https://nolabnoparty.com/en/veeam-v11-hardened-repository-immutability-pt-1/

https://nolabnoparty.com/en/veeam-v11-hardened-repository-immutability-configuration-pt-2/

Userlevel 7
Badge +13

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Comment