Skip to main content

Veeam v11 - Hardened Repository aka Immutable backups


Show first post

88 comments

MicoolPaul
Forum|alt.badge.img+22
  • 2343 comments
  • January 28, 2021
vNote42 wrote:
rmaynard wrote:
vNote42 wrote:
rmaynard wrote:

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • January 28, 2021
MicoolPaul wrote:
vNote42 wrote:
rmaynard wrote:
vNote42 wrote:
rmaynard wrote:

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

You are right, would be more secure if set on repo-server itself.

But: A VBR-admin is just able to disable immutability when SSH is running on repo-server. As a best practice, SSH should be disabled after installation. So it is still rather secure because the admin has to have access to the Linux server.

If you do not trust your admin, do not give him access to linux-repo :wink:

This is the exact message I’ve been pushing, the main risks are insider threats and physical access at this point!

Never trust a guy who wants to enter your datacenter with this tool

 


  • New Here
  • 1 comment
  • February 5, 2021

Very good to have this feature, just wondering how best to spread retention and calculate space requirements


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • February 5, 2021

I would recommend to use Restore Point Simulator (https://rps.dewin.me/). Put in data for your backup jobs and adjust “Retention Points”  like interval for immutable backups.


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • February 15, 2021

[Update]

In the course of hardened repository, Veeam changed the way how to handle retention in backup copy jobs. In short: with v11, backup copy jobs retentions works as forward incremental with synthetic fulls! So keep in mind to add weekly backups to keep your backup chains short.

See this post for more information:

 


BertrandFR
Forum|alt.badge.img+8
  • Influencer
  • 525 comments
  • February 18, 2021

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?


  • New Here
  • 2 comments
  • February 19, 2021
rmaynard wrote:
vNote42 wrote:
rmaynard wrote:

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw


  • New Here
  • 2 comments
  • February 19, 2021

In my case I'd say 90% of my customers are currently using Windows ReFS to store backup files as their main repository, so I have some questions:

  1. Do you know if Microsoft is working to provide something equivalent to the  XFS “i” flag feature in ReFS?. For 100% Microsoft shops every time you mention Linux is still something they tried to avoid as much as they can. I can envision a lot of resistance if we are talking about the main repository in this specific case
  2. In terms of performance / space saving have Veeam done any lab test to compare XFS vs ReFS?
  3. For customers that will be willing to migrate their main repository from ReFS to XFS to take advantage of this feature: any tips, best practices? It would be great if Veeam provides a Whitepaper regarding this

I think immutable backups is the top driver to adopt V11 in the short term for a lot of customers


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • February 19, 2021
BertrandFR wrote:

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • February 19, 2021
andy40241 wrote:
rmaynard wrote:
vNote42 wrote:
rmaynard wrote:

So what happens if an insider gets into your VBR infrastructure and disables the “Make recent backups immutable for” checkbox, then waits for N days for the immutability flags to cycle out from the hardened repository… then deletes all your backups?

I agree with you! I would recommend to monitor this setting by running a scheduled script.

That sounds a) tricky and b) failure-prone - for example a tech changes the job name, script fails… we missed it… whoops.

My concern is that people will look at a Hardened Linux Repository (as I did) as way to provide an effectively air-gapped backup when it really can’t.  If immutability could be set and enforced by the repository itself and not changed from VBR then it would be a lot closer to that goal but obviously it will never be truly air-gapped.

 

Is the script to verify that “Make recent backups immutable for” is still checked already posted somewhere?


Some ideas to improve this specific scenario which I'm 100% sure it's going to happen in the real life:

  • Put in place a password verification method if somebody tries to uncheck this feature, or ever something more sophisticated like and OTP with Microsoft/Google authenticator
  • Send an email alert if somebody disable this feature
  • Make the backup engine to verify if the i attribute is still set in the backup chain and provide a warning if it has been disabled before run the job

great info btw

Thanks for your feedback!

This could be done by PowerShell: Running such a script daily and send a mail, when disabled.

Basically: make sure your access to the Linux-repo server is as secure as possible! Good idea to also set MFA for VBR server!

As I described, when i-flag is removed on file-level, VBR does not delete these files before immutable-interval ended. So this is not a problem. The problem here is: when a user gets that far, he is able to delete the files manually.

 


BertrandFR
Forum|alt.badge.img+8
  • Influencer
  • 525 comments
  • February 19, 2021
vNote42 wrote:
BertrandFR wrote:

Hey! Have you tried the reflink behavior when you write encrypted backup? Is it work with encrypted backup or it’s like dedupe?

Good question! I did not try this in my lab. I have to say, it seems to be not very common to encrypt backup files. Do you so?


Maybe i can have the use case due to  the CSO demands :grin:

 

 


  • New Here
  • 1 comment
  • February 25, 2021

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


BertrandFR
Forum|alt.badge.img+8
  • Influencer
  • 525 comments
  • February 25, 2021
Salad360 wrote:

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.


MicoolPaul
Forum|alt.badge.img+22
  • 2343 comments
  • February 25, 2021
BertrandFR wrote:
Salad360 wrote:

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!


regnor
Forum|alt.badge.img+14
  • Veeam MVP
  • 1333 comments
  • February 25, 2021
MicoolPaul wrote:
BertrandFR wrote:
Salad360 wrote:

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • February 25, 2021
Salad360 wrote:

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 

Right! I would also warn against using raw device mappings (RDM) in vSphere! These are physical disk devices mapped directly into VMs. But with this a hacker having access to the hypervisor can also destroy your backup data - no matter if they are immutable or not. 

So when using hardened repository, use a (hardened) physical Linux host! 


JMeixner
Forum|alt.badge.img+17
  • Veeam Legend, Veeam Vanguard
  • 2641 comments
  • February 25, 2021
regnor wrote:
MicoolPaul wrote:
BertrandFR wrote:
Salad360 wrote:

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?


Hi,

ok I understand the intention to disable all remote access...

But I don’t have physical access to my backup servers because they are located in datacenters all over Germany and some other european countries.

How do you handle such servers in your environments without remote access?

Ok, I could hire a helping hand every time any action has to be done at a server… But this would be expensive and I don’t think this would  be more secure… :confounded:


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • February 26, 2021
JMeixner wrote:
regnor wrote:
MicoolPaul wrote:
BertrandFR wrote:
Salad360 wrote:

I assume your Linux repo would need to be a bare-metal machine to gain any security benefit from this? If I were to virtualize it, some hacker could just blow up my VHD file on the Hyper-V host? 


Cryptolocker will be happy to encrypt all your vhd and filesystem, in the worst case you VM will not longer be available so your backup repo too.

It’s definitely worth using physical for this, I had an emergency response to a business that had their Hyper-V domain joined, the ransomware they got hit with had encrypted the contents of the VHDX’s then the VHDX’s themselves on top for good measure as it just hit everything it could in the domain!

But management domains and whether or not to domain in general is a conversation for another day…

 

If you are going physical though do consider other forms of access such as IPMI/iLO/iDRAC that could be used as backdoors to the system and lock them down!

I would suggest to disable any remote access after the initial setup. If there's really a need to access the system why not do I physically?


Hi,

ok I understand the intention to disable all remote access...

But I don’t have physical access to my backup servers because they are located in datacenters all over Germany and some other european countries.

How do you handle such servers in your environments without remote access?

Ok, I could hire a helping hand every time any action has to be done at a server… But this would be expensive and I don’t think this would  be more secure… :confounded:

To keep remote access more secure, you can implement MFA for Linux. If this is possible for IPMI/iLO/iDRAC (is it?) this would sense too! If not, you could disable network port on the switch and enable it if access is necessary. 


JMeixner
Forum|alt.badge.img+17
  • Veeam Legend, Veeam Vanguard
  • 2641 comments
  • February 26, 2021

Thanks @vNote42


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • February 26, 2021
andy40241 wrote:

In my case I'd say 90% of my customers are currently using Windows ReFS to store backup files as their main repository, so I have some questions:

  1. Do you know if Microsoft is working to provide something equivalent to the  XFS “i” flag feature in ReFS?. For 100% Microsoft shops every time you mention Linux is still something they tried to avoid as much as they can. I can envision a lot of resistance if we are talking about the main repository in this specific case
  2. In terms of performance / space saving have Veeam done any lab test to compare XFS vs ReFS?
  3. For customers that will be willing to migrate their main repository from ReFS to XFS to take advantage of this feature: any tips, best practices? It would be great if Veeam provides a Whitepaper regarding this

I think immutable backups is the top driver to adopt V11 in the short term for a lot of customers

@andy40241 sorry for the late response.

  1. I am not aware of Microsoft is planning such feature. In my experience customer minds are open for Linux repositories when they hear about immutable backup files. A lot of them see Linux repositories as storage of a different media in the 3-2-1 rule. If there is no know-how at customer site, I would recommend to buy support for distribution of choice.
  2. I do not know official Veeam publications about performance and/or space savings comparison between ReFS and XFS. I would recommend to check this post for experiences with XFS of the community. Furthermore, Veeam does not make any differences between XFS and ReFS is their official sizer tools. Also for the (unofficial) Restore Point Simulator space savings are the same.
  3. There is also a discussion about this topic in the community. In my opinion the simples way is to fade out restore points on ReFS after starting new backups on XFS.

BertrandFR
Forum|alt.badge.img+8
  • Influencer
  • 525 comments
  • February 26, 2021

@JMeixner check links below why disable remote access etc…

https://bp.veeam.com/vbr/VBP/Security/infrastructure_hardening.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_linux.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_windows.html

For windows hosts even hardened who are very important, i break all suspicious processus after some learning with elastic


vNote42
Forum|alt.badge.img+13
  • Author
  • On the path to Greatness
  • 1246 comments
  • February 26, 2021
BertrandFR wrote:

@JMeixnercheck links below why disable remote access etc…

https://bp.veeam.com/vbr/VBP/Security/infrastructure_hardening.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_linux.html

https://bp.veeam.com/vbr/VBP/Security/hardening_backup_repository_windows.html

For windows hosts even hardened who are very important, i break all suspicious processus after some learning with elastic

Thanks for the links! Great ressources!


regnor
Forum|alt.badge.img+14
  • Veeam MVP
  • 1333 comments
  • February 26, 2021

@JMeixner No physical access and no easy hands-on could be challenging. MFA, as @vNote42 says, could be the solution in that case.


JMeixner
Forum|alt.badge.img+17
  • Veeam Legend, Veeam Vanguard
  • 2641 comments
  • February 26, 2021

Thanks  @regnor and @vNote42  ,

MFA and the activation of network ports when access is neccessary are good mechanisms for my environments.


  • New Here
  • 1 comment
  • February 27, 2021

Thank you.! Is there a step-by-step guide somewhere? I planning to upgrade to V11, and want to set up this hardening as soon as possible


Comment