Veeam v11 - Hardened Repository aka Immutable backups



Show first post

88 comments

Userlevel 7
Badge +6

I think this it’ll be the silver bullet of V11. It sounds like the best feature of this new version of Veeam, isn't? 

Just a question. The Linux repository (/veeam/Rep/xfs_1) on your example... Is it a sharing on operating system?

 

What do you mean by “sharing on operating system”? /veeamRepo is a mountpoint for local disk-volume. xfs_1 is a directory on this volume.

Ah, ok. Is a local mount point… So how do you add this as a repository on Veeam B&R?

Just like any other repositories: start the Add new Repository wizard (shown in first screenshot in the post), add the Linux server and add the directory as repository.  

A, pretty nice! t's simpler than I imagined...

Userlevel 7
Badge +3

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

Userlevel 7
Badge +8

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

Userlevel 7
Badge +6

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?

Userlevel 7
Badge +7

Thanks @vNote42

Userlevel 7
Badge +7

In my case I'd say 90% of my customers are currently using Windows ReFS to store backup files as their main repository, so I have some questions:

  1. Do you know if Microsoft is working to provide something equivalent to the  XFS “i” flag feature in ReFS?. For 100% Microsoft shops every time you mention Linux is still something they tried to avoid as much as they can. I can envision a lot of resistance if we are talking about the main repository in this specific case
  2. In terms of performance / space saving have Veeam done any lab test to compare XFS vs ReFS?
  3. For customers that will be willing to migrate their main repository from ReFS to XFS to take advantage of this feature: any tips, best practices? It would be great if Veeam provides a Whitepaper regarding this

I think immutable backups is the top driver to adopt V11 in the short term for a lot of customers

@andy40241 sorry for the late response.

  1. I am not aware of Microsoft is planning such feature. In my experience customer minds are open for Linux repositories when they hear about immutable backup files. A lot of them see Linux repositories as storage of a different media in the 3-2-1 rule. If there is no know-how at customer site, I would recommend to buy support for distribution of choice.
  2. I do not know official Veeam publications about performance and/or space savings comparison between ReFS and XFS. I would recommend to check this post for experiences with XFS of the community. Furthermore, Veeam does not make any differences between XFS and ReFS is their official sizer tools. Also for the (unofficial) Restore Point Simulator space savings are the same.
  3. There is also a discussion about this topic in the community. In my opinion the simples way is to fade out restore points on ReFS after starting new backups on XFS.
Userlevel 7
Badge +3

@JMeixner check links below why disable remote access etc…

https://bp.veeam.com/vbr/VBP/Security/infrastructure_hardening.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_linux.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_windows.html

For windows hosts even hardened who are very important, i break all suspicious processus after some learning with elastic

Userlevel 7
Badge +6

@JMeixner No physical access and no easy hands-on could be challenging. MFA, as @vNote42 says, could be the solution in that case.

Userlevel 7
Badge +7

Thanks  @regnor and @vNote42  ,

MFA and the activation of network ports when access is neccessary are good mechanisms for my environments.

Userlevel 7
Badge +7

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

Userlevel 7
Badge +7

Yeah….  can we just start respecting the native immutability already present on certain Dedupe appliances?

they also have some “advantage” in providing a proprietary access. As long as ransomware is not able to use this, it increases security.

Userlevel 7
Badge +8

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

Userlevel 1

Thank you.! Is there a step-by-step guide somewhere? I planning to upgrade to V11, and want to set up this hardening as soon as possible

Userlevel 7
Badge +3

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?


Maybe i can have the use case due to  the CSO demands :grin:

 

 

Userlevel 7
Badge +7

Check out the new Whitepaper from Veeam ( @HannesK  ): 

Protect against Ransomware with Immutable Backups: a Veeam Guide

Userlevel 1

Very good to have this feature, just wondering how best to spread retention and calculate space requirements

Userlevel 7
Badge +3

Thanks @vNote42 

Userlevel 7
Badge +7

There is new blog post from @PValsecchi about setting up MFA for SSH loginsto Linux Hosts.

https://nolabnoparty.com/en/veeam-v11-hardened-repository-immutability-add-mfa-pt-3/

 

Nice and detailled tutorial...

Userlevel 7
Badge +7

Nice to see all of this information in one post.  Not sure it is possible but you should see about editing the main post with the updates versus them being within the pages.  Just would make things easier to find all in the first post.  Maybe we don’t have the editing ability either.  LOL

Good point chris!

Legends have the permission to edit their own posts only since a few weeks now. I future I will edit the original post to add updates. Thanks!

 

Userlevel 7
Badge +7

[additional information about immutable file handling in linux]

Check out this great post for details about how immutable files are stored in linux. There is also a xml-file and file-attributes for storing date until file is immutable.

https://blog.workinghardinit.work/2021/01/18/immutability-of-linux-files-on-the-veeam-hardened-linux-repository/

Hi 

I am looking to implement hardened repository im already using Veeam v11 however I have zero experience created Linux VM and required config on it.

Would anyone be so kind as to give me some pointers?

Thanks.

 

Userlevel 7
Badge +6

Great post, thank you for taking the time to go in-depth and provide the screenshots with useful commands for checking even the network ports listening.

Userlevel 2

aamm… let me explain a little more.. I did several installations, and in all of them I did something wrong, and I can not reset the veeamhubrepo in a way so I can start all over again and succeed, so.. a used a virtual machine with a snapshot and returned to the initial state before using the veeamhubrepo and begin configuring again.. well… I have a physical server with Ubuntu Server 20 LTS and I did something wrong, and the veeamhubrepo won’t star again, so I need to re-install  the server in order to do everything correctly, but with no opportunity to fail AT ALL! or else.. I need to re-install the server again, that’s why I need to know how to reset the veamhubrepo tool in order to avoid any re-installation. Where are the configuration files? So I can delete them and start a fresh install.

 

The image you see on one of my comments is when I succeeded on a virtual machine, but not with my physical server.

Userlevel 7
Badge +7

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 

Right! I would also warn against using raw device mappings (RDM) in vSphere! These are physical disk devices mapped directly into VMs. But with this a hacker having access to the hypervisor can also destroy your backup data - no matter if they are immutable or not. 

So when using hardened repository, use a (hardened) physical Linux host! 

Userlevel 7
Badge +7

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?


Hi,

ok I understand the intention to disable all remote access...

But I don’t have physical access to my backup servers because they are located in datacenters all over Germany and some other european countries.

How do you handle such servers in your environments without remote access?

Ok, I could hire a helping hand every time any action has to be done at a server… But this would be expensive and I don’t think this would  be more secure… :confounded:

Comment