• EN

Veeam v11 - Hardened Repository - how it behaves

vNote42
Userlevel 7
Badge +13

It is always interesting, when thinking of special situations, to ask: What happens, when …

Here are the answers to at least a few of these questions :grin: - for immutable backups in VBR v11:

What happens, when…

… removing immutable restore points?
Task starts and ends up with errors. No restore point is removed.

Warning when trying to remove immutable files

… hacker changes date/time on Veeam server to remove files with next backup run?
Excellent question! But it will not work! Configured period of time is checked at Linux server, not at Veeam B&R Server! So a job running one year in the future will create new files, it cannot delete old ones.

… enabling feature at existing repository, not compatible with hardened repository?
Veeam checks if there are backup jobs configured with an incompatible backup chain. If found, error is shown:

Error with not compatible backups

… changing a backup job writing to immutable repository?
When backup chain is changed to an incompatible one, error is shown:

… days for immutability is higher than the number of restore points?
Restore points are kept to the end of the period, defined for immutability. Notice that there is an information message in job log:

Info that files are still immutable

… immutability turned on on repository containing restore points?
In this case immutable flag is set on all files, according immutability period.

… immutability turned off on repository containing immutable backups?
In this case, immutable flags are kept. That makes perfect sense. Otherwise a hacker with backup administrator permissions could disable hardened repository to delete backup files. Notice: new backups are not immutable after disabling the feature.

… trying to remove restore points in GUI on files with immutable flag manually removed?
Interestingly files will not be deleted! Probably because Veeam knows, files should be protected. There will be more details on that after the release of v11.

Cannot remove files even immutable flag is manually removed

… use immutable repository as target for copy job without GFS?
Backup copy jobs requires GFS when copied to immutable repository. If no GFS is set up, error appears.

 

Wolfgang | vnote42.net | @vNote42

40 comments

BertrandFR
Userlevel 7
Badge +8

i’m wondering if someone deployed an hardened repo with openscap for hardening the OS? What is your experiences about it? did you encounter any inconvenience about OS hardening enable ?

BTW, i discovered this article on Redhat Website about the deployment of an hardened repository.

https://www.redhat.com/en/blog/veeam-ransomware-protection-rhel-immutable-repository

@vNote42 not sure it’s the right topic, if it’s not tell me i will move my message to another :)

vNote42
Userlevel 7
Badge +13

@Zucchetti Spain , just add my two cents:

As already said, it is easily possible to disable immutability if you get backup-admin access to VBR server. This is because immutability is just a property of the repository, saved in the VBR-database. Therefore it is essential to monitor these settings. With Veeam ONE, you have a good tool to do so:

Monitor Hardened Repository with Veeam ONE v11a

 

Maybe also important to note: GFS restore points stay immutable as long as they are kept by retention-policy.

 

 

 

Wolfgang | vnote42.net | @vNote42
MicoolPaul
Userlevel 7
Badge +20

@Zucchetti Spain, correct on waiting the immutability time, I can’t find the document but I thought I saw something about not being able to change the immutability state of a hardened repository after initial deployment, I may be thinking of Object Storage, @vNote42 / @Mildur, does this ring any bells to yourselves?

I have tried It and I can remove the chek in Veeam… with no warning. now i’m waiting 7 days to pass but I’m quite sure I will be able to delete my (test) backups…

 

In my opinion Veeam should warn about “this backup has changed form inmutable to non inmutable” for at least some days...

Thanks @Mildur for confirming, couldn’t remember if it was either or both, so your comments there are appreciated!

@Zucchetti Spain, just to also say, the immutability will be 7 days from the last point in the chain by the way, Veeam increments this on the dependent points in the active chain to ensure the chain is immutable to the same time so there’s no isolated incremental backups that aren’t usable.

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
JMeixner
Userlevel 7
Badge +17

You can it disable on a hardened repository. But the backups which are already in the repository stay immutable. Only the new ones are not immutable.

Yes, the VeeamOne alarm is very useful. But I agree, it would be useful when Veeam would warn itself, too….

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
Z
Userlevel 1

@MicoolPaul

Are you talking about disabling Immutability? Disabling immutability on an object Storage is not possible. You get an error.

 

 

Immutability on Linux Hardened Repositories can be disabled in my testlab, but Veeam One will alarm you about it.

 

yes… I mean disabling it on Linux hardened repository…  Veeam One will warn me… but in my opinion it shoud be a good idea put a warning in the result email of the copy itself

 

I
Userlevel 7
Badge +4

Awesome post! 

Inder | www.thenetworkdna.com | Twitter @inder8588
Mildur
Userlevel 7
Badge +12

@MicoolPaul

Are you talking about disabling Immutability? Disabling immutability on an object Storage is not possible. You get an error.

 

 

Immutability on Linux Hardened Repositories can be disabled in my testlab, but Veeam One will alarm you about it.

 

Product Management Analyst @ Veeam Software
Z
Userlevel 1

@Zucchetti Spain, correct on waiting the immutability time, I can’t find the document but I thought I saw something about not being able to change the immutability state of a hardened repository after initial deployment, I may be thinking of Object Storage, @vNote42 / @Mildur, does this ring any bells to yourselves?

I have tried It and I can remove the chek in Veeam… with no warning. now i’m waiting 7 days to pass but I’m quite sure I will be able to delete my (test) backups…

 

In my opinion Veeam should warn about “this backup has changed form inmutable to non inmutable” for at least some days...

MicoolPaul
Userlevel 7
Badge +20

@Zucchetti Spain, correct on waiting the immutability time, I can’t find the document but I thought I saw something about not being able to change the immutability state of a hardened repository after initial deployment, I may be thinking of Object Storage, @vNote42 / @Mildur, does this ring any bells to yourselves?

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
Z
Userlevel 1

Also to add, if you’re using Veeam ONE (which, you really should be!), there’s built in alarms for Immutability state and Immutability change tracking. So this would drive your warnings.

that’s a good point!! thanks!

Z
Userlevel 1

So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

you would still have to wait for the immutability time to expire, so if I put 160 days…. The lock is on the object store so Veeam and windows can’t change it.

Yes, I said “wait the inmutability time”… if it’s very long you will be safer… but you will need a really big disk

MicoolPaul
Userlevel 7
Badge +20

Also to add, if you’re using Veeam ONE (which, you really should be!), there’s built in alarms for Immutability state and Immutability change tracking. So this would drive your warnings.

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
Geoff Burke
Userlevel 7
Badge +22

So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

you would still have to wait for the immutability time to expire, so if I put 160 days…. The lock is on the object store so Veeam and windows can’t change it.

Now if you got console on the linux server then that is not good. If you are running your object storgage in containers in orchestrated environment then we add yet another level of difficult into the equation for the bad guys ;)

VMCA2022, VMCE2023, CKA, CKAD, KCNA, LFCS, PCA,TARS
Geoff Burke
Userlevel 7
Badge +22

So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

you would still have to wait for the immutability time to expire, so if I put 160 days…. The lock is on the object store so Veeam and windows can’t change it.

VMCA2022, VMCE2023, CKA, CKAD, KCNA, LFCS, PCA,TARS
regnor
Userlevel 7
Badge +14

I wouldn't say it's the easy way, but yes this would work. While I would say 7 days is a rather short time frame for immutability, at least you would lose your current backups.

That's why you still need to secure your environment and should monitor all configuration changes. Immutability is only a part of the solution.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
Z
Userlevel 1

So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

JMeixner
Userlevel 7
Badge +17

Just talked with a customer during installation about this topic. He had a similar idea, that fits at least for central Europe based installations: DCF77 based dongles. 

With GPS based dongles, I think you have a problem, when your dongle is not able to receive GPS signals. For example hardware is located in basement.


Good idea :grin:

With this you would have to hack the broadcasting system, very unlikely…
And this signal reaches literally every place you can image in central and western europe. No walls are too thick :grin:

Similar broadcasts are available in many parts of the world - e.g RWM in Russion, WWV in the USA or MSF in Great Britain…

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
vNote42
Userlevel 7
Badge +13

Just talked with a customer during installation about this topic. He had a similar idea, that fits at least for central Europe based installations: DCF77 based dongles. 

With GPS based dongles, I think you have a problem, when your dongle is not able to receive GPS signals. For example hardware is located in basement.

Wolfgang | vnote42.net | @vNote42
MicoolPaul
Userlevel 7
Badge +20

The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?

Agree! 

I think the last physical server without time-sync is more than 20 years ago :older_man_tone1:

You should be able to enable daylight saving in BIOS/EFI of the host, so even this should be not problem.

As @Gostev stated: you have to take care when troubleshooting! Time on repo-server will be slightly different from other hosts.

I think one key thing is once again it drives the argument of physical instead of virtual repository to prevent time being overwritten by host.

 

Also in the event the CMOS battery is depleted and the server reboots the time will revert back to 1970 most likely anyway so no risk of violating retention policies there! As for other management methods for time, I don’t have any alternative solutions that are more tamperproof. You could try and layer a separate management network that has no out of band management or control that delivers time, but it runs the risk of adding unnecessary complexity for diminishing security concerns. It’s not KISS and still not as good as disabling NTP entirely and just using the CMOS.

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
vNote42
Userlevel 7
Badge +13

The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?

Agree! 

I think the last physical server without time-sync is more than 20 years ago :older_man_tone1:

You should be able to enable daylight saving in BIOS/EFI of the host, so even this should be not problem.

As @Gostev stated: you have to take care when troubleshooting! Time on repo-server will be slightly different from other hosts.

Wolfgang | vnote42.net | @vNote42
JMeixner
Userlevel 7
Badge +17

The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
BertrandFR
Userlevel 7
Badge +8

Ntp Antenna on DC roof ftw :cowboy:

vNote42
Userlevel 7
Badge +13

Yes, @MicoolPaul you are right: NTP is a attack vector! I talk with Veeam a few times about this topic. They actually recommend using a GPS-dongle for time synchronization. So I see currently these options: 

  • no time sync at all
  • GPS time dongle

Do any of you see other options?

Wolfgang | vnote42.net | @vNote42
MicoolPaul
Userlevel 7
Badge +20

Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

I have the feeling there is a misunderstanding. There is no pre-hardened image with immutable repositories. Linux host must be installed and configured (like a windows repository server). So you can configure Linux as you wish.

Good to know! Have been just gleaming different bits of information since we can’t play with it yet.

 

Thanks for sharing :)

Just to update this, Gostev addressed these concerns in his latest “The word from Gostev”

 

Another interesting topic is a time sync. There are not too many attack vectors against a hardened repository with SSH Server disabled and all incoming traffic blocked, so this one is definitely something to worry about considering that the immutability flags are removed according to the local time on the repository. While many Linux OS set the time automatically using NTP during boot or when they detect the network is up. So, naturally it's best to disable time sync completely: the CMOS clock are not terrible in precision, and you can simply adjust the time manually each time you install a Veeam update. The only drawback of this approach is it complicates forensic analysis due to the security audit logs not showing the precise time, so using a GPS time dongle could be another option. Here are some thoughts about this attack vector from our Linux guru Tom Sightler.

By default, Ubuntu runs timesyncd which implements only a super simple SNTP client which queries one server on boot or network up events. So an attacker with network level access could, in theory, force an NTP time sync even if you otherwise didn't have a full fledged NTP client running, and you might not even realize you were vulnerable. As far as I know, all the known time-shift attacks require MitM, which can be gained for example by getting access to the network switch itself. If that's the case, the attacker might be able to trigger timesyncd to sync by toggling the network port up/down.

Of course if a hacker has gained so much control over your network that they can manipulate your time servers, inject packets into the network, control network ports, etc. then you're going to eventually lose anyway (arguably, you've already lost, you just don't know the final score yet). But a GPS time source would make even this type of attack quite difficult. Although, believe it or not, there are people that even worry about this kind of attack. A fake GPS signal could change the time and such a targeted attack is totally possible. But if you've got somebody that is this serious about getting your data, there's definitely no excuse to not have off-facility, offline backups.

Interesting to see what suggestions are being made from Veeam!

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
vNote42
Userlevel 7
Badge +13

[update] 

What happens, when choosing immutable repository for backup types that are not supported for immutability? For example: DB logs, NAS.

Backup runs anyway! File are just not immutable after job is finished. See in screenshot, SQL log backups (*.vlm, *.vlb, *.vsm) are not flagged.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

Was so happy to see this as it means we don’t need to split out repositories based on whether the job is supported for immutability!

Absolutely! This would be a huge impact in each design!

Wolfgang | vnote42.net | @vNote42

Comment


Terms & Conditions