• EN

Veeam v11 - Hardened Repository - how it behaves

Show first post

40 comments

vNote42
Userlevel 7
Badge +13

@Zucchetti Spain , just add my two cents:

As already said, it is easily possible to disable immutability if you get backup-admin access to VBR server. This is because immutability is just a property of the repository, saved in the VBR-database. Therefore it is essential to monitor these settings. With Veeam ONE, you have a good tool to do so:

Monitor Hardened Repository with Veeam ONE v11a

 

Maybe also important to note: GFS restore points stay immutable as long as they are kept by retention-policy.

 

 

 

Wolfgang | vnote42.net | @vNote42
BertrandFR
Userlevel 7
Badge +8

Ntp Antenna on DC roof ftw :cowboy:

MicoolPaul
Userlevel 7
Badge +20

The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?

Agree! 

I think the last physical server without time-sync is more than 20 years ago :older_man_tone1:

You should be able to enable daylight saving in BIOS/EFI of the host, so even this should be not problem.

As @Gostev stated: you have to take care when troubleshooting! Time on repo-server will be slightly different from other hosts.

I think one key thing is once again it drives the argument of physical instead of virtual repository to prevent time being overwritten by host.

 

Also in the event the CMOS battery is depleted and the server reboots the time will revert back to 1970 most likely anyway so no risk of violating retention policies there! As for other management methods for time, I don’t have any alternative solutions that are more tamperproof. You could try and layer a separate management network that has no out of band management or control that delivers time, but it runs the risk of adding unnecessary complexity for diminishing security concerns. It’s not KISS and still not as good as disabling NTP entirely and just using the CMOS.

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
MicoolPaul
Userlevel 7
Badge +20

[update] 

What happens, when choosing immutable repository for backup types that are not supported for immutability? For example: DB logs, NAS.

Backup runs anyway! File are just not immutable after job is finished. See in screenshot, SQL log backups (*.vlm, *.vlb, *.vsm) are not flagged.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

Was so happy to see this as it means we don’t need to split out repositories based on whether the job is supported for immutability!

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
JMeixner
Userlevel 7
Badge +17

Just talked with a customer during installation about this topic. He had a similar idea, that fits at least for central Europe based installations: DCF77 based dongles. 

With GPS based dongles, I think you have a problem, when your dongle is not able to receive GPS signals. For example hardware is located in basement.


Good idea :grin:

With this you would have to hack the broadcasting system, very unlikely…
And this signal reaches literally every place you can image in central and western europe. No walls are too thick :grin:

Similar broadcasts are available in many parts of the world - e.g RWM in Russion, WWV in the USA or MSF in Great Britain…

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
MicoolPaul
Userlevel 7
Badge +20

Also to add, if you’re using Veeam ONE (which, you really should be!), there’s built in alarms for Immutability state and Immutability change tracking. So this would drive your warnings.

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
MicoolPaul
Userlevel 7
Badge +20

@Zucchetti Spain, correct on waiting the immutability time, I can’t find the document but I thought I saw something about not being able to change the immutability state of a hardened repository after initial deployment, I may be thinking of Object Storage, @vNote42 / @Mildur, does this ring any bells to yourselves?

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
MicoolPaul
Userlevel 7
Badge +20

@Zucchetti Spain, correct on waiting the immutability time, I can’t find the document but I thought I saw something about not being able to change the immutability state of a hardened repository after initial deployment, I may be thinking of Object Storage, @vNote42 / @Mildur, does this ring any bells to yourselves?

I have tried It and I can remove the chek in Veeam… with no warning. now i’m waiting 7 days to pass but I’m quite sure I will be able to delete my (test) backups…

 

In my opinion Veeam should warn about “this backup has changed form inmutable to non inmutable” for at least some days...

Thanks @Mildur for confirming, couldn’t remember if it was either or both, so your comments there are appreciated!

@Zucchetti Spain, just to also say, the immutability will be 7 days from the last point in the chain by the way, Veeam increments this on the dependent points in the active chain to ensure the chain is immutable to the same time so there’s no isolated incremental backups that aren’t usable.

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
vNote42
Userlevel 7
Badge +13

Yes, @MicoolPaul you are right: NTP is a attack vector! I talk with Veeam a few times about this topic. They actually recommend using a GPS-dongle for time synchronization. So I see currently these options: 

  • no time sync at all
  • GPS time dongle

Do any of you see other options?

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

Just talked with a customer during installation about this topic. He had a similar idea, that fits at least for central Europe based installations: DCF77 based dongles. 

With GPS based dongles, I think you have a problem, when your dongle is not able to receive GPS signals. For example hardware is located in basement.

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

[update] 

What happens, when choosing immutable repository for backup types that are not supported for immutability? For example: DB logs, NAS.

Backup runs anyway! File are just not immutable after job is finished. See in screenshot, SQL log backups (*.vlm, *.vlb, *.vsm) are not flagged.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

Was so happy to see this as it means we don’t need to split out repositories based on whether the job is supported for immutability!

Absolutely! This would be a huge impact in each design!

Wolfgang | vnote42.net | @vNote42
Z
Userlevel 1

So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

I
Userlevel 7
Badge +4

Awesome post! 

Inder | www.thenetworkdna.com | Twitter @inder8588
vNote42
Userlevel 7
Badge +13

The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?

Agree! 

I think the last physical server without time-sync is more than 20 years ago :older_man_tone1:

You should be able to enable daylight saving in BIOS/EFI of the host, so even this should be not problem.

As @Gostev stated: you have to take care when troubleshooting! Time on repo-server will be slightly different from other hosts.

Wolfgang | vnote42.net | @vNote42
BertrandFR
Userlevel 7
Badge +8

i’m wondering if someone deployed an hardened repo with openscap for hardening the OS? What is your experiences about it? did you encounter any inconvenience about OS hardening enable ?

BTW, i discovered this article on Redhat Website about the deployment of an hardened repository.

https://www.redhat.com/en/blog/veeam-ransomware-protection-rhel-immutable-repository

@vNote42 not sure it’s the right topic, if it’s not tell me i will move my message to another :)

Comment


Terms & Conditions