Skip to main content

It is always interesting, when thinking of special situations, to ask: What happens, when …

Here are the answers to at least a few of these questions :grin: - for immutable backups in VBR v11:

What happens, when…

… removing immutable restore points?
Task starts and ends up with errors. No restore point is removed.

Warning when trying to remove immutable files

… hacker changes date/time on Veeam server to remove files with next backup run?
Excellent question! But it will not work! Configured period of time is checked at Linux server, not at Veeam B&R Server! So a job running one year in the future will create new files, it cannot delete old ones.

… enabling feature at existing repository, not compatible with hardened repository?
Veeam checks if there are backup jobs configured with an incompatible backup chain. If found, error is shown:

Error with not compatible backups

… changing a backup job writing to immutable repository?
When backup chain is changed to an incompatible one, error is shown:

… days for immutability is higher than the number of restore points?
Restore points are kept to the end of the period, defined for immutability. Notice that there is an information message in job log:

Info that files are still immutable

… immutability turned on on repository containing restore points?
In this case immutable flag is set on all files, according immutability period.

… immutability turned off on repository containing immutable backups?
In this case, immutable flags are kept. That makes perfect sense. Otherwise a hacker with backup administrator permissions could disable hardened repository to delete backup files. Notice: new backups are not immutable after disabling the feature.

… trying to remove restore points in GUI on files with immutable flag manually removed?
Interestingly files will not be deleted! Probably because Veeam knows, files should be protected. There will be more details on that after the release of v11.

Cannot remove files even immutable flag is manually removed

… use immutable repository as target for copy job without GFS?
Backup copy jobs requires GFS when copied to immutable repository. If no GFS is set up, error appears.

 

PS: If you think about other situations, let me know. If possible I will test it ...


Awesome post! Didn’t take the time to test this, it’s on my list! How is the answer from the system if you try dd or rm -f on it? Or Encryption?


“Permission denied”, even root gets this error. Your really have remote immutable-flag before removing/changing.


Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

Not sure you need to worry about this, but you could send out which NTP servers to use via DHCP option 42. On Linux, I think you can add ntp-servers to the dhclient.conf. More info in https://unix.stackexchange.com/questions/327954/how-do-you-set-up-a-linux-client-to-use-ntp-information-provided-through-dhcp


Another great post @vNote42! I really love how you took the time to think through different scenarios and test them out in the lab. Thank you for sharing.

Thank you for the compliment! It is always great to get such a feedback!


[Update]

… trying to add a non-immutable repository on a Linux server hosting a immutable repository?

Currently mixing immutable with non-immutable repositories on the same Linux server is not supported. So you get an error:

 


Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :


Another great post @vNote42! I really love how you took the time to think through different scenarios and test them out in the lab. Thank you for sharing.


Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?


Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.


Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

I have the feeling there is a misunderstanding. There is no pre-hardened image with immutable repositories. Linux host must be installed and configured (like a windows repository server). So you can configure Linux as you wish.


Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

I have the feeling there is a misunderstanding. There is no pre-hardened image with immutable repositories. Linux host must be installed and configured (like a windows repository server). So you can configure Linux as you wish.

Good to know! Have been just gleaming different bits of information since we can’t play with it yet.

 

Thanks for sharing :)

Just to update this, Gostev addressed these concerns in his latest “The word from Gostev”

 

Another interesting topic is a time sync. There are not too many attack vectors against a hardened repository with SSH Server disabled and all incoming traffic blocked, so this one is definitely something to worry about considering that the immutability flags are removed according to the local time on the repository. While many Linux OS set the time automatically using NTP during boot or when they detect the network is up. So, naturally it's best to disable time sync completely: the CMOS clock are not terrible in precision, and you can simply adjust the time manually each time you install a Veeam update. The only drawback of this approach is it complicates forensic analysis due to the security audit logs not showing the precise time, so using a GPS time dongle could be another option. Here are some thoughts about this attack vector from our Linux guru Tom Sightler.

By default, Ubuntu runs timesyncd which implements only a super simple SNTP client which queries one server on boot or network up events. So an attacker with network level access could, in theory, force an NTP time sync even if you otherwise didn't have a full fledged NTP client running, and you might not even realize you were vulnerable. As far as I know, all the known time-shift attacks require MitM, which can be gained for example by getting access to the network switch itself. If that's the case, the attacker might be able to trigger timesyncd to sync by toggling the network port up/down.

Of course if a hacker has gained so much control over your network that they can manipulate your time servers, inject packets into the network, control network ports, etc. then you're going to eventually lose anyway (arguably, you've already lost, you just don't know the final score yet). But a GPS time source would make even this type of attack quite difficult. Although, believe it or not, there are people that even worry about this kind of attack. A fake GPS signal could change the time and such a targeted attack is totally possible. But if you've got somebody that is this serious about getting your data, there's definitely no excuse to not have off-facility, offline backups.


Interesting to see what suggestions are being made from Veeam!


The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?


[update] 

What happens, when choosing immutable repository for backup types that are not supported for immutability? For example: DB logs, NAS.

Backup runs anyway! File are just not immutable after job is finished. See in screenshot, SQL log backups (*.vlm, *.vlb, *.vsm) are not flagged.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110


Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

I have the feeling there is a misunderstanding. There is no pre-hardened image with immutable repositories. Linux host must be installed and configured (like a windows repository server). So you can configure Linux as you wish.

Good to know! Have been just gleaming different bits of information since we can’t play with it yet.

 

Thanks for sharing :)


Excellent Post !!! :ok_hand:


I wouldn't say it's the easy way, but yes this would work. While I would say 7 days is a rather short time frame for immutability, at least you would lose your current backups.

That's why you still need to secure your environment and should monitor all configuration changes. Immutability is only a part of the solution.


So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

you would still have to wait for the immutability time to expire, so if I put 160 days…. The lock is on the object store so Veeam and windows can’t change it.


So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

you would still have to wait for the immutability time to expire, so if I put 160 days…. The lock is on the object store so Veeam and windows can’t change it.

Now if you got console on the linux server then that is not good. If you are running your object storgage in containers in orchestrated environment then we add yet another level of difficult into the equation for the bad guys ;)


So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

you would still have to wait for the immutability time to expire, so if I put 160 days…. The lock is on the object store so Veeam and windows can’t change it.

Yes, I said “wait the inmutability time”… if it’s very long you will be safer… but you will need a really big disk


Also to add, if you’re using Veeam ONE (which, you really should be!), there’s built in alarms for Immutability state and Immutability change tracking. So this would drive your warnings.

that’s a good point!! thanks!


@Zucchetti Spain, correct on waiting the immutability time, I can’t find the document but I thought I saw something about not being able to change the immutability state of a hardened repository after initial deployment, I may be thinking of Object Storage, @vNote42@Mildur, does this ring any bells to yourselves?

I have tried It and I can remove the chek in Veeam… with no warning. now i’m waiting 7 days to pass but I’m quite sure I will be able to delete my (test) backups…

 

In my opinion Veeam should warn about “this backup has changed form inmutable to non inmutable” for at least some days...


@MicoolPaul

Are you talking about disabling Immutability? Disabling immutability on an object Storage is not possible. You get an error.

 

 

Immutability on Linux Hardened Repositories can be disabled in my testlab, but Veeam One will alarm you about it.

 


@MicoolPaul

Are you talking about disabling Immutability? Disabling immutability on an object Storage is not possible. You get an error.

 

 

Immutability on Linux Hardened Repositories can be disabled in my testlab, but Veeam One will alarm you about it.

 

yes… I mean disabling it on Linux hardened repository…  Veeam One will warn me… but in my opinion it shoud be a good idea put a warning in the result email of the copy itself

 


You can it disable on a hardened repository. But the backups which are already in the repository stay immutable. Only the new ones are not immutable.

Yes, the VeeamOne alarm is very useful. But I agree, it would be useful when Veeam would warn itself, too….


Comment