Skip to main content
  • EN

I spent some time in testing a new great feature in v11: Hardened Repository. Read here about:

  • What is immutability is about
  • Requirements
  • Setup
  • How it works

 

What is immutability is about

Immutability in this context means, a backup file cannot be changed or deleted without having root access within hosing Linux OS. So, even the backup administrator is not able to delete backups on such a repository.

Why is this important? Think about ransomware. This software is that smart these days, it is able to recognize backup systems. It can trigger tasks like deletion of backup files. But when they are immutable, this cannot be done!

 

Requirements

What is needed to get immutable backups? First of all: Veeam Backup&Replication v11. This is the first version supporting hardened repositories. Secondly a Linux server hosting repository volumes.

What about the filesystem? Veeam is using immutable flag. So every filesystem supporting this flag can be used. These are pretty much all. Veeam supports reflink/Fast Clone on XFS. Because of this XFS is the recommended filesystem.

What about the distribution? At the moment of writing I had no information about this. I think this feature will not constrain the selection of supported Linux distributions. When using XFS, we get a fist choice: Ubuntu 20.04 LTS (long-term support). Because: (1) Ubuntu is supported by Veeam. (2) 20.04 uses kernel version of 5.4. This version seems to provide highest quality of reflink, tested by Veeam.

Thirdly: Backup chains must be compatible with immutable files. What does this mean? Because files cannot be changed, the backup chain only can create new files without changing any of the existing. Only forward incremental with periodic synthetic or active fulls fulfill this requirement. For backup copy jobs, GFS settings are required.

 

Setup

Immutable backups are enabled on repository level. Either at creating the repository. Or for a existing repository. How to setup Linux as repository server I will covered in another post.

Settings are easy to understand:

Immutable backup settings

 

How it (just) works

The beauty of this feature is the use of native filesystem features. In Linux each file can have an attribute i. When this is set, file cannot be changed or removed. When Veeam creates backup files, this flag is set. After the entered period of immutability, flag is removed and file can be deleted.

To see file attributes, including immutable flag, run: lsattr filename in Linux shell. Sample output see here:

Immutable flag set on files

Note, flags are removed from a whole backup chain, not just a single file.

Flag removed after protection period

The question may arise how the flag is set in Linux. Because, when the specified Linux user gets privileged access to add or remove the flag, this could be used by a hacker to get access to these files as well. Right, BUT: flag is not set by this user. Instead it is set by root. This can be done by running a service with root access: veeamimmureposvc:

service to set immutable flag

Notice: this service has no connection to the network, so it cannot be compromised remotely!

Apropos network: What ports are being used? Also new in v11 is that just one port is used to communicate with repository host: TCP/6162. During a backup other ports can be opened on demand.

Open ports with no running job
Open port with running job

The whole blog post, with some more details, you can find here: 

https://vnote42.net/2020/11/23/new-in-veeam-v11-hardened-repository-immutable-backups-part-1/

Thanks for the great article on immutability! So chattr should be restricted with selinux on linux host, good to know! I will put a ML task specific with elastic auditbeat on it

Thanks for the great article on immutability! So chattr should be restricted with selinux on linux host, good to know! I will put a ML task specific with elastic auditbeat on it

“ML task specific with elastic auditbeat” sounds interesting! And what I just read it is for sure!

Great post, thank you for taking the time to go in-depth and provide the screenshots with useful commands for checking even the network ports listening.

Update information:

How does it work, when I start now with v10 and XFS repo and want to use immutable backups when v11 is available? 

Short answer by @Gostev :

“After you upgrade to v11, you will only need to run a shell command to change the owner for the existing backup files. There's will be a message with instructions in the user interface if you enable immutable backups option on the existing repository. The rest is optional (additional hardening like disabling SSH server and remote consoles like iLO completely).”

https://forums.veeam.com/veeam-backup-replication-f2/v11-linux-immutability-t70932.html

Thanks for posting this, interesting. Will Linux be the repo of choice after v11 is released, i think so?

Thanks for posting this, interesting. Will Linux be the repo of choice after v11 is released, i think so?

I talk more and more with customers about Linux as repo-server. Yes, most of them are very interested in XFS with immutable backups!

[Important update]

You CAN NOT run immutable repository AND proxy role on the same Linux server in v11!

PS: There is a workaround: you theoretically can run one of these roles in a container.

Oh, that’s a limitation... On the other hand it makes sense as you would need root-access for the proxy.

Wouldn’t running one or both roles inside a container open new attack possibilites? 

Oh, that’s a limitation... On the other hand it makes sense as you would need root-access for the proxy.

Wouldn’t running one or both roles inside a container open new attack possibilites? 

Yes it is. And it is - from my perspective - no technical limitation, it is for security reasons. 

I would say, you will always take a higher security risk when running a second role on the repository server. When - for example because of a kernel-bug - a user can break out/in a container, your backup data is in danger.

[Update information]

Veeam is planning to support fast cloning with Ubuntu LTS 18.04, RHEL/CentOS 8.2 or later, SLES 15 SP2 and Debian 10 as well! For other distributions: support will be experimental. Kernel version 5.4 or later recommended.

So there will be more options when is comes to distribution selection.

Excellent work !!! Thanks for sharing.

[additional information about immutable file handling in linux]

Check out this great post for details about how immutable files are stored in linux. There is also a xml-file and file-attributes for storing date until file is immutable.

https://blog.workinghardinit.work/2021/01/18/immutability-of-linux-files-on-the-veeam-hardened-linux-repository/

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

A, pretty nice! t's simpler than I imagined...

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

A, pretty nice! t's simpler than I imagined...

Yes, it just works :joy:

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

Yeah….  can we just start respecting the native immutability already present on certain Dedupe appliances?

Having an insider in your environment who has enough time to prepare his attack is always bad. The immutable repository will be only a part of the complete solution. I would still utilize offline or secure offsite backups. Also monitoring your environment for changes will be a good idea in order to catch such events.

 

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

Yeah….  can we just start respecting the native immutability already present on certain Dedupe appliances?

they also have some “advantage” in providing a proprietary access. As long as ransomware is not able to use this, it increases security.

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

Comment


Terms & Conditions