Skip to main content
  • EN

It is always interesting, when thinking of special situations, to ask: What happens, when …

Here are the answers to at least a few of these questions :grin: - for immutable backups in VBR v11:

What happens, when…

… removing immutable restore points?
Task starts and ends up with errors. No restore point is removed.

Warning when trying to remove immutable files

… hacker changes date/time on Veeam server to remove files with next backup run?
Excellent question! But it will not work! Configured period of time is checked at Linux server, not at Veeam B&R Server! So a job running one year in the future will create new files, it cannot delete old ones.

… enabling feature at existing repository, not compatible with hardened repository?
Veeam checks if there are backup jobs configured with an incompatible backup chain. If found, error is shown:

Error with not compatible backups

… changing a backup job writing to immutable repository?
When backup chain is changed to an incompatible one, error is shown:

… days for immutability is higher than the number of restore points?
Restore points are kept to the end of the period, defined for immutability. Notice that there is an information message in job log:

Info that files are still immutable

… immutability turned on on repository containing restore points?
In this case immutable flag is set on all files, according immutability period.

… immutability turned off on repository containing immutable backups?
In this case, immutable flags are kept. That makes perfect sense. Otherwise a hacker with backup administrator permissions could disable hardened repository to delete backup files. Notice: new backups are not immutable after disabling the feature.

… trying to remove restore points in GUI on files with immutable flag manually removed?
Interestingly files will not be deleted! Probably because Veeam knows, files should be protected. There will be more details on that after the release of v11.

Cannot remove files even immutable flag is manually removed

… use immutable repository as target for copy job without GFS?
Backup copy jobs requires GFS when copied to immutable repository. If no GFS is set up, error appears.

 

PS: If you think about other situations, let me know. If possible I will test it ...

Awesome post! Didn’t take the time to test this, it’s on my list! How is the answer from the system if you try dd or rm -f on it? Or Encryption?

“Permission denied”, even root gets this error. Your really have remote immutable-flag before removing/changing.

Excellent Post !!! :ok_hand:

Another great post @vNote42! I really love how you took the time to think through different scenarios and test them out in the lab. Thank you for sharing.

Another great post @vNote42! I really love how you took the time to think through different scenarios and test them out in the lab. Thank you for sharing.

Thank you for the compliment! It is always great to get such a feedback!

[Update]

… trying to add a non-immutable repository on a Linux server hosting a immutable repository?

Currently mixing immutable with non-immutable repositories on the same Linux server is not supported. So you get an error:

 

Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

I have the feeling there is a misunderstanding. There is no pre-hardened image with immutable repositories. Linux host must be installed and configured (like a windows repository server). So you can configure Linux as you wish.

Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

Not sure you need to worry about this, but you could send out which NTP servers to use via DHCP option 42. On Linux, I think you can add ntp-servers to the dhclient.conf. More info in https://unix.stackexchange.com/questions/327954/how-do-you-set-up-a-linux-client-to-use-ntp-information-provided-through-dhcp

Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

I have the feeling there is a misunderstanding. There is no pre-hardened image with immutable repositories. Linux host must be installed and configured (like a windows repository server). So you can configure Linux as you wish.

Good to know! Have been just gleaming different bits of information since we can’t play with it yet.

 

Thanks for sharing :)

[update] 

What happens, when choosing immutable repository for backup types that are not supported for immutability? For example: DB logs, NAS.

Backup runs anyway! File are just not immutable after job is finished. See in screenshot, SQL log backups (*.vlm, *.vlb, *.vsm) are not flagged.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

[update] 

What happens, when choosing immutable repository for backup types that are not supported for immutability? For example: DB logs, NAS.

Backup runs anyway! File are just not immutable after job is finished. See in screenshot, SQL log backups (*.vlm, *.vlb, *.vsm) are not flagged.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

Was so happy to see this as it means we don’t need to split out repositories based on whether the job is supported for immutability!

[update] 

What happens, when choosing immutable repository for backup types that are not supported for immutability? For example: DB logs, NAS.

Backup runs anyway! File are just not immutable after job is finished. See in screenshot, SQL log backups (*.vlm, *.vlb, *.vsm) are not flagged.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

Was so happy to see this as it means we don’t need to split out repositories based on whether the job is supported for immutability!

Absolutely! This would be a huge impact in each design!

Do we know what time servers are being used? It was an interesting point you raised about changing the time on Veeam not causing any issues but I’d assume NTP is via DNS and to resolve internal resources it would require internal DNS servers (or even be supplied them via DHCP), so hijacking the FQDN of the NTP server could be an attack to wipe immutability?

Good point, BUT :grin: :

As it’s a pre-hardened image, does it give you the ability to configure this NTP as IP Address via any deployment or do we need to manually consider this?

 

It’d be good to confirm if NTP is going to ignore these panic thresholds.

I have the feeling there is a misunderstanding. There is no pre-hardened image with immutable repositories. Linux host must be installed and configured (like a windows repository server). So you can configure Linux as you wish.

Good to know! Have been just gleaming different bits of information since we can’t play with it yet.

 

Thanks for sharing :)

Just to update this, Gostev addressed these concerns in his latest “The word from Gostev”

 

Another interesting topic is a time sync. There are not too many attack vectors against a hardened repository with SSH Server disabled and all incoming traffic blocked, so this one is definitely something to worry about considering that the immutability flags are removed according to the local time on the repository. While many Linux OS set the time automatically using NTP during boot or when they detect the network is up. So, naturally it's best to disable time sync completely: the CMOS clock are not terrible in precision, and you can simply adjust the time manually each time you install a Veeam update. The only drawback of this approach is it complicates forensic analysis due to the security audit logs not showing the precise time, so using a GPS time dongle could be another option. Here are some thoughts about this attack vector from our Linux guru Tom Sightler.

By default, Ubuntu runs timesyncd which implements only a super simple SNTP client which queries one server on boot or network up events. So an attacker with network level access could, in theory, force an NTP time sync even if you otherwise didn't have a full fledged NTP client running, and you might not even realize you were vulnerable. As far as I know, all the known time-shift attacks require MitM, which can be gained for example by getting access to the network switch itself. If that's the case, the attacker might be able to trigger timesyncd to sync by toggling the network port up/down.

Of course if a hacker has gained so much control over your network that they can manipulate your time servers, inject packets into the network, control network ports, etc. then you're going to eventually lose anyway (arguably, you've already lost, you just don't know the final score yet). But a GPS time source would make even this type of attack quite difficult. Although, believe it or not, there are people that even worry about this kind of attack. A fake GPS signal could change the time and such a targeted attack is totally possible. But if you've got somebody that is this serious about getting your data, there's definitely no excuse to not have off-facility, offline backups.

Interesting to see what suggestions are being made from Veeam!

Yes, @MicoolPaul you are right: NTP is a attack vector! I talk with Veeam a few times about this topic. They actually recommend using a GPS-dongle for time synchronization. So I see currently these options: 

  • no time sync at all
  • GPS time dongle

Do any of you see other options?

Ntp Antenna on DC roof ftw :cowboy:

The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?

The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?

Agree! 

I think the last physical server without time-sync is more than 20 years ago :older_man_tone1:

You should be able to enable daylight saving in BIOS/EFI of the host, so even this should be not problem.

As @Gostev stated: you have to take care when troubleshooting! Time on repo-server will be slightly different from other hosts.

The CMOS clocks of servers are rather precise. The last server with a really unprecise clock I have seen... 20 years ago.

So I think, I could live with a repository server without time synchronization… Even the daylight saving should not be a big problem. OK, some immutable backups will be eligible to deletion one hour earlier or later. But is this a real problem?

Agree! 

I think the last physical server without time-sync is more than 20 years ago :older_man_tone1:

You should be able to enable daylight saving in BIOS/EFI of the host, so even this should be not problem.

As @Gostev stated: you have to take care when troubleshooting! Time on repo-server will be slightly different from other hosts.

I think one key thing is once again it drives the argument of physical instead of virtual repository to prevent time being overwritten by host.

 

Also in the event the CMOS battery is depleted and the server reboots the time will revert back to 1970 most likely anyway so no risk of violating retention policies there! As for other management methods for time, I don’t have any alternative solutions that are more tamperproof. You could try and layer a separate management network that has no out of band management or control that delivers time, but it runs the risk of adding unnecessary complexity for diminishing security concerns. It’s not KISS and still not as good as disabling NTP entirely and just using the CMOS.

Just talked with a customer during installation about this topic. He had a similar idea, that fits at least for central Europe based installations: DCF77 based dongles. 

With GPS based dongles, I think you have a problem, when your dongle is not able to receive GPS signals. For example hardware is located in basement.

Just talked with a customer during installation about this topic. He had a similar idea, that fits at least for central Europe based installations: DCF77 based dongles. 

With GPS based dongles, I think you have a problem, when your dongle is not able to receive GPS signals. For example hardware is located in basement.


Good idea :grin:

With this you would have to hack the broadcasting system, very unlikely…
And this signal reaches literally every place you can image in central and western europe. No walls are too thick :grin:

Similar broadcasts are available in many parts of the world - e.g RWM in Russion, WWV in the USA or MSF in Great Britain…

So the easy way to attack a inmutable copy is:

  • Take control of Veeam server.
  • change the repository config. (remove inmutability)
  • wait the inmutability time (let’s say 7 days)
  • You will be able to delete restore points from Veeam console… and final user has had no warning.

 

Am I correct?

Comment


Terms & Conditions