Skip to main content
  • EN

I spent some time in testing a new great feature in v11: Hardened Repository. Read here about:

  • What is immutability is about
  • Requirements
  • Setup
  • How it works

 

What is immutability is about

Immutability in this context means, a backup file cannot be changed or deleted without having root access within hosing Linux OS. So, even the backup administrator is not able to delete backups on such a repository.

Why is this important? Think about ransomware. This software is that smart these days, it is able to recognize backup systems. It can trigger tasks like deletion of backup files. But when they are immutable, this cannot be done!

 

Requirements

What is needed to get immutable backups? First of all: Veeam Backup&Replication v11. This is the first version supporting hardened repositories. Secondly a Linux server hosting repository volumes.

What about the filesystem? Veeam is using immutable flag. So every filesystem supporting this flag can be used. These are pretty much all. Veeam supports reflink/Fast Clone on XFS. Because of this XFS is the recommended filesystem.

What about the distribution? At the moment of writing I had no information about this. I think this feature will not constrain the selection of supported Linux distributions. When using XFS, we get a fist choice: Ubuntu 20.04 LTS (long-term support). Because: (1) Ubuntu is supported by Veeam. (2) 20.04 uses kernel version of 5.4. This version seems to provide highest quality of reflink, tested by Veeam.

Thirdly: Backup chains must be compatible with immutable files. What does this mean? Because files cannot be changed, the backup chain only can create new files without changing any of the existing. Only forward incremental with periodic synthetic or active fulls fulfill this requirement. For backup copy jobs, GFS settings are required.

 

Setup

Immutable backups are enabled on repository level. Either at creating the repository. Or for a existing repository. How to setup Linux as repository server I will covered in another post.

Settings are easy to understand:

Immutable backup settings

 

How it (just) works

The beauty of this feature is the use of native filesystem features. In Linux each file can have an attribute i. When this is set, file cannot be changed or removed. When Veeam creates backup files, this flag is set. After the entered period of immutability, flag is removed and file can be deleted.

To see file attributes, including immutable flag, run: lsattr filename in Linux shell. Sample output see here:

Immutable flag set on files

Note, flags are removed from a whole backup chain, not just a single file.

Flag removed after protection period

The question may arise how the flag is set in Linux. Because, when the specified Linux user gets privileged access to add or remove the flag, this could be used by a hacker to get access to these files as well. Right, BUT: flag is not set by this user. Instead it is set by root. This can be done by running a service with root access: veeamimmureposvc:

service to set immutable flag

Notice: this service has no connection to the network, so it cannot be compromised remotely!

Apropos network: What ports are being used? Also new in v11 is that just one port is used to communicate with repository host: TCP/6162. During a backup other ports can be opened on demand.

Open ports with no running job
Open port with running job

The whole blog post, with some more details, you can find here: 

https://vnote42.net/2020/11/23/new-in-veeam-v11-hardened-repository-immutable-backups-part-1/

Thanks @Adolfo_Veloz! I think I understand the problem you had. But I did not get the solution. Can you please tell the solution again.

aamm… let me explain a little more.. I did several installations, and in all of them I did something wrong, and I can not reset the veeamhubrepo in a way so I can start all over again and succeed, so.. a used a virtual machine with a snapshot and returned to the initial state before using the veeamhubrepo and begin configuring again.. well… I have a physical server with Ubuntu Server 20 LTS and I did something wrong, and the veeamhubrepo won’t star again, so I need to re-install  the server in order to do everything correctly, but with no opportunity to fail AT ALL! or else.. I need to re-install the server again, that’s why I need to know how to reset the veamhubrepo tool in order to avoid any re-installation. Where are the configuration files? So I can delete them and start a fresh install.

 

The image you see on one of my comments is when I succeeded on a virtual machine, but not with my physical server.

I'm seeing this tool for the first time; is it really intended to be used interactively? Normally you just have to create a user, setup the directory permissions and then you add the system via the Veeam console; and all config is done by Veeam itself. In my case, the Ubuntu server image had all dependencies already installed out of the box; it didn't have to install any packages at all.

Have you tried this way or is there a reason why you use the veeamhubrepo executable?

I'm seeing this tool for the first time; is it really intended to be used interactively? Normally you just have to create a user, setup the directory permissions and then you add the system via the Veeam console; and all config is done by Veeam itself. In my case, the Ubuntu server image had all dependencies already installed out of the box; it didn't have to install any packages at all.

Have you tried this way or is there a reason why you use the veeamhubrepo executable?

This seems to be the way it is done as I have not seen the other method shown.  Interesting.

I don’t see a method thru veeam server.. I was given a link in which to follow al the instructions to configure it.. and btw, Ubuntu does not come with everything Veeam needs, the xfsprogs needs to be installed before, as well as the configuration of the veeamhubrepo.

This is the link I followed:

 

https://24xsiempre.com/veeam-hardened-immutable-repository/

Ok, now I get it; thanks for providing the link to the blog post. I didn't now that script but according to the github page it's experimental and so I wouldn’t use it in production. I would suggest that you can contact the authors via github for some help. Or perhaps @poulpreben can give you an advice here in the community? 

Regarding the setup in general. You can setup everything manually and don't necessarily need a script. I went with the information from @vNote42 and after having some problems with Ubuntu I checked out this blog series:

https://www.starwindsoftware.com/blog/veeam-hardened-linux-repository-part-1

You may remove the file /etc/veeamhubrepo to reset the tool. 

May I ask at what stage you run into trouble? I am personally using the tool multiple times per week to provision new repositories for our customers, but any feedback that can improve reliability would be much appreciated.

That was quick, thanks @poulpreben 🙂

aamm… let me explain a little more.. I did several installations, and in all of them I did something wrong, and I can not reset the veeamhubrepo in a way so I can start all over again and succeed, so.. a used a virtual machine with a snapshot and returned to the initial state before using the veeamhubrepo and begin configuring again.. well… I have a physical server with Ubuntu Server 20 LTS and I did something wrong, and the veeamhubrepo won’t star again, so I need to re-install  the server in order to do everything correctly, but with no opportunity to fail AT ALL! or else.. I need to re-install the server again, that’s why I need to know how to reset the veamhubrepo tool in order to avoid any re-installation. Where are the configuration files? So I can delete them and start a fresh install.

 

The image you see on one of my comments is when I succeeded on a virtual machine, but not with my physical server.

Thanks for your description! You may ask in R&D forum for help?

I don’t see a method thru veeam server.. I was given a link in which to follow al the instructions to configure it.. and btw, Ubuntu does not come with everything Veeam needs, the xfsprogs needs to be installed before, as well as the configuration of the veeamhubrepo.

This is the link I followed:

 

https://24xsiempre.com/veeam-hardened-immutable-repository/

Thanks for the link! Never used a script to do the installation. There are just a few steps to do at the linux server, I do not see that much advantages to use a script for these.

I agree with @regnor , I never installed any tools after the ubuntu installation. Everything I needed was there. Even tools for XFS.

Hi Wolfgang,

 

If the Linux machine is a VM, how do you prevent root from just connecting through vSphere and changing the file attributes and just removing everything?

 

Thanks,

 

I dont know if it's possible to restrict root access via the local console. You could disable root and only work with sudo, but that wouldn't change anything. Perhaps you can setup something like multifactor to increase the security.

But the bigger problem I see in you case, is running the repository virtualized. If an attacker can login to you vSphere management, then he could just wipe or delete your repository VM; so no need to use the root user itself. If you want to virtualize, then I would take care to lockdown or even disable any management possibility.

Hi Wolfgang,

 

If the Linux machine is a VM, how do you prevent root from just connecting through vSphere and changing the file attributes and just removing everything?

 

Thanks,

 

Hi Adam!

As @regnor already mentioned, it is definitely not recommended to use a virtualized Linux Server as Hardened Repository Server!

If you still want to do this, I would recommend not to install VMware Tools there, so the VIX-Interface shouldn’t be available. Furthermore, you should implement at least certificate-based authentication.

I want to stretch this: Do not use a VM for Backup Repository in production!

Comment


Terms & Conditions