Veeam v11 - Hardened Repository aka Immutable backups


Userlevel 7
Badge +13

I spent some time in testing a new great feature in v11: Hardened Repository. Read here about:

  • What is immutability is about
  • Requirements
  • Setup
  • How it works

 

What is immutability is about

Immutability in this context means, a backup file cannot be changed or deleted without having root access within hosing Linux OS. So, even the backup administrator is not able to delete backups on such a repository.

Why is this important? Think about ransomware. This software is that smart these days, it is able to recognize backup systems. It can trigger tasks like deletion of backup files. But when they are immutable, this cannot be done!

 

Requirements

What is needed to get immutable backups? First of all: Veeam Backup&Replication v11. This is the first version supporting hardened repositories. Secondly a Linux server hosting repository volumes.

What about the filesystem? Veeam is using immutable flag. So every filesystem supporting this flag can be used. These are pretty much all. Veeam supports reflink/Fast Clone on XFS. Because of this XFS is the recommended filesystem.

What about the distribution? At the moment of writing I had no information about this. I think this feature will not constrain the selection of supported Linux distributions. When using XFS, we get a fist choice: Ubuntu 20.04 LTS (long-term support). Because: (1) Ubuntu is supported by Veeam. (2) 20.04 uses kernel version of 5.4. This version seems to provide highest quality of reflink, tested by Veeam.

Thirdly: Backup chains must be compatible with immutable files. What does this mean? Because files cannot be changed, the backup chain only can create new files without changing any of the existing. Only forward incremental with periodic synthetic or active fulls fulfill this requirement. For backup copy jobs, GFS settings are required.

 

Setup

Immutable backups are enabled on repository level. Either at creating the repository. Or for a existing repository. How to setup Linux as repository server I will covered in another post.

Settings are easy to understand:

Immutable backup settings

 

How it (just) works

The beauty of this feature is the use of native filesystem features. In Linux each file can have an attribute i. When this is set, file cannot be changed or removed. When Veeam creates backup files, this flag is set. After the entered period of immutability, flag is removed and file can be deleted.

To see file attributes, including immutable flag, run: lsattr filename in Linux shell. Sample output see here:

Immutable flag set on files

Note, flags are removed from a whole backup chain, not just a single file.

Flag removed after protection period

The question may arise how the flag is set in Linux. Because, when the specified Linux user gets privileged access to add or remove the flag, this could be used by a hacker to get access to these files as well. Right, BUT: flag is not set by this user. Instead it is set by root. This can be done by running a service with root access: veeamimmureposvc:

service to set immutable flag

Notice: this service has no connection to the network, so it cannot be compromised remotely!

Apropos network: What ports are being used? Also new in v11 is that just one port is used to communicate with repository host: TCP/6162. During a backup other ports can be opened on demand.

Open ports with no running job
Open port with running job

The whole blog post, with some more details, you can find here: 

https://vnote42.net/2020/11/23/new-in-veeam-v11-hardened-repository-immutable-backups-part-1/


88 comments

Userlevel 7
Badge +13

Thank you.! Is there a step-by-step guide somewhere? I planning to upgrade to V11, and want to set up this hardening as soon as possible

I would recommend to start here: https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

There you find the section: Deployment of Hardened Repository.

I am sure more detailed information and best practices for Linux repositories will come soon.

Userlevel 7
Badge +13

[Update]

Veeam Hardened Repository passes independent compliance assessment

When properly configured, the Hardened Repository meets the requirements for non-rewritable, non-erasable storage as specified by SEC 17a-4(f), FINRA 4511(c) and CFTC 1.31(c)-(d) regulations.

https://www.veeam.com/blog/hardened-repository-passes-compliance.html

 

Userlevel 7
Badge +13

Check out the new Whitepaper from Veeam ( @HannesK  ): 

Protect against Ransomware with Immutable Backups: a Veeam Guide

@vNote42 have you been able to post anything on How to setup Linux as repository server?

 

Userlevel 7
Badge +13

@vNote42 have you been able to post anything on How to setup Linux as repository server?

 

Hi @MAC_Daddy_1974 ! Just wrote an internal installation guide. I can recommend this post:

https://nolabnoparty.com/en/veeam-v11-hardened-repository-immutability-pt-1/ by @PValsecchi 

Userlevel 7
Badge +3

@vNote42 thanks for share and your update 

Userlevel 7
Badge +17

There is new blog post from @PValsecchi about setting up MFA for SSH loginsto Linux Hosts.

https://nolabnoparty.com/en/veeam-v11-hardened-repository-immutability-add-mfa-pt-3/

 

Nice and detailled tutorial...

Userlevel 7
Badge +22

Great Information folks this really helps!

Hi 

I am looking to implement hardened repository im already using Veeam v11 however I have zero experience created Linux VM and required config on it.

Would anyone be so kind as to give me some pointers?

Thanks.

 

Userlevel 7
Badge +13

Hi 

I am looking to implement hardened repository im already using Veeam v11 however I have zero experience created Linux VM and required config on it.

Would anyone be so kind as to give me some pointers?

Thanks.

 

Check out excellent blog series by @PValsecchi :

https://nolabnoparty.com/en/veeam-v11-hardened-repository-immutability-pt-1/

https://nolabnoparty.com/en/veeam-v11-hardened-repository-immutability-configuration-pt-2/

Userlevel 7
Badge +1

Thanks @vNote42

Userlevel 7
Badge +4

Thanks @vNote42 

Userlevel 7
Badge +20

Nice to see all of this information in one post.  Not sure it is possible but you should see about editing the main post with the updates versus them being within the pages.  Just would make things easier to find all in the first post.  Maybe we don’t have the editing ability either.  LOL

Userlevel 7
Badge +13

Nice to see all of this information in one post.  Not sure it is possible but you should see about editing the main post with the updates versus them being within the pages.  Just would make things easier to find all in the first post.  Maybe we don’t have the editing ability either.  LOL

Good point chris!

Legends have the permission to edit their own posts only since a few weeks now. I future I will edit the original post to add updates. Thanks!

 

Userlevel 7
Badge +20

Nice to see all of this information in one post.  Not sure it is possible but you should see about editing the main post with the updates versus them being within the pages.  Just would make things easier to find all in the first post.  Maybe we don’t have the editing ability either.  LOL

Good point chris!

Legends have the permission to edit their own posts only since a few weeks now. I future I will edit the original post to add updates. Thanks!

 

No problem.  Just figured Rick does it for the v11 post he has so it keeps it tidy. :grinning:

Userlevel 7
Badge +12

I've looked at the hardened repositories in the last days and the configuration itself is really easy. The Linux side is more complicated if you're not used to setup such systems 😅

@vNote42In your blog you've posted a screenshot of a dialog box requesting the change of the directory owner. Is this checkbox no longer existing in the GA release?

Userlevel 7
Badge +13

I've looked at the hardened repositories in the last days and the configuration itself is really easy. The Linux side is more complicated if you're not used to setup such systems 😅

@vNote42In your blog you've posted a screenshot of a dialog box requesting the change of the directory owner. Is this checkbox no longer existing in the GA release?

Yes, for what you get it is really simple!

You probably mean this dialog-box?

Good question! I think I have not seen the dialog in installations since GA. But this could also be because I ran the command before. Did you run it before?

Userlevel 7
Badge +12

Yes I meant that dialog. I forgot to change the owner and was wondering why Veeam couldn't create the job folders. So the dialog did probably only exist in the beta.

Userlevel 7
Badge +13

Yes I meant that dialog. I forgot to change the owner and was wondering why Veeam couldn't create the job folders. So the dialog did probably only exist in the beta.

Probably! Maybe @Rick Vanover  can answer this question?

Userlevel 7
Badge +10

@vNote42 @regnor Yes anything that has "TBD" Is likely a beta, preview or otherwise non-Generally Available build.

 

If this is in GA, let me know and I will get it into the bug fix cycle.

 

I do not believe I have seen this in GA myself.

Userlevel 7
Badge +12

@vNote42 @regnor Yes anything that has "TBD" Is likely a beta, preview or otherwise non-Generally Available build.

 

If this is in GA, let me know and I will get it into the bug fix cycle.

 

I do not believe I have seen this in GA myself.


Well no it's not in GA but it would be a useful hint especially if you don't look in the documentation.

Userlevel 2

I’m having problems with the tool, in the begining I was not aware that the xfs progs needed to be installed on my ubuntu version, and the installer failed, well, I just downloaded the xfsprogs and then the veeamhubrepo won’t start again, I deleted the file  ‘/etc/veeamhubtinyrepoman’ and started all over again, but in the end it crashes again with this error:

 

Can any one tell me how to reset the tool to start over clean? Thanx.

 

Userlevel 7
Badge +12

I'm not sure if it works, but if you remove the Linux server in Veeam, does it uninstall all components and cleanup the system?

Userlevel 2

I’m not getting to that step yet… this is before adding it to a Veeam Server.

Userlevel 2

It is supposed that, once enabling ssh access the next screen will be:

 

 

Once there.. the number 3 option must be selected, and then add it to the Veeam Server 11.

Comment