Skip to main content
  • EN

 

In my last post I covered the requirements and necessary steps to configure immutability for HPE StoreOnce with Veeam V12. In this post I want to focus on how immutability behaves in different scenarios.

Enable immutability for existing Catalyst stores

If you have existing Catalyst stores, the good news is that it’s possible to enable immutability for them. Let’s take a look at the Veeam helpcenter:

Note that if you enable immutability and Veeam Backup & Replication does not start a new backup chain and still continues the chain, the whole backup chain is marked as immutable.

https://helpcenter.veeam.com/docs/backup/vsphere/storeonce_supported_features.html?ver=120#immutability

So, after you configure immutability, the current or new backup chain will be made immutable. All other existing backup chains won’t be touched and therefore stay mutable.

This is also exactly what I could reproduce in my lab environment. After enabling immutability, the current chain is made immutable, while all other backup files aren’t.

Afterwards I tried to remove all restore points via the Veeam Console, and not surprisingly all restore points except the 2 immutable ones were deleted.

 

So keep in mind, that your historical or GFS restore points won’t be protected, if you enable immutability.

Increasing immutability

Next I wanted to find out what happens if I increase the immutability. As expected only the current chain gets the update, all other chains will keep their previous configuration.

Decreasing immutability

Decreasing the immutability only affects newly created backup files. For existing files/chains, the previous value is remained.

Disabling immutability

Also disabling immutability only affects newly created backup files. All other files keep their last immutable timestamp.

 

Changing time/date on VBR side

I would expect that changing the time/date on the VBR servers doesn’t affect the immutability of the backup files.

So I changed the date of my VBR server and afterwards the backup file was marked as expired.

Trying to delete the backup files ended with a warning.

Despite the warning, the job/restore point was no longer displayed in the Veeam console; neither under Disk, nor as imported or orphaned. Also a rescan didn’t re-import the backups.

I was glad to still see the backup file via the StoreOnce Web GUI. This means, Veeam isn’t able to delete the file because of it’s immutability. On the other hand it cleaned up the VBM and the entry in the configuration database. I’ve posted this behaviour in the R&D forums and will update my post on how to make such files available again.

Changing time/date on VBR & StoreOnce

Now I was interested in finding out what would happen if we change the date on both VBR and StoreOnce. Again I created a new full backup.

Then I changed the date on the VBR server and the Storeonce, and deleted the backup from the VBR console. And as expected, it was possible to delete the backup file.

Changing the date on the StoreOnce requires an approval of the Security Officer. So it’s essential to keep the credentials of this user secret.

Deleting the StoreOnce Catalyst store

The backup files itself are protected by immutability and cannot be deleted from inside Veeam. The danger is, that an attacker deletes the whole StoreOnce Catalyst store from within the StoreOnce management GUI.

As Dual Authorization is enabled, the Security Officer needs to approve the deletion of the Catalyst store.

Afterwards, the Store and all of it’s content is gone.

 

Just like in the last scenario, if an attacker gets access to both the administrator and security officer credentials, there’s nothing which prevents him from wiping all backups.

In addition, be aware that wiping the store is also possible via Remote Management (iLO) of your physical StoreOnce, or via the hypervisor of the virtual appliance.

Summary

Immutability on a HPE StoreOnce system is a great addition for the security of your backup files and I recommend that you enable it by default. But it doesn’t offer 100% security (nothing does) and is only a part of the complete backup solution.

Therefore I will just quote my recommendations from my first post:

  • keep the Security Officer’s credentials secret
    • an attacker will be able to circumvent the immutability if he gets access to those credentials
    • I would even go as far and say that you print them out and stick them physically on the StoreOnce
  • keep the iLO Interface secure or even disconnected: all StoreOnce security measurements won’t help if someone physically wipes the device remotely via iLO
  • monitor the immutability setting in Veeam: If an attacker can’t delete your backups, he might just alter or completely disable immutability in Veeam

Great second part to this series.  Thanks for sharing.

Hi @regnor, thnx for sharing your test results.


I couldn’t find the forum post that you mentioned in your topic about .VBM file deletions,
did you manage to get more info and can you share the link?
 

Sharing our findings

Using Veeam V12.0.0.1420 P20230223: We tested the HPe StoreOnce and its behavior as well and found out that the .VBM file is indeed not immutable. We are also able to delete this file in the catalyst store directly or through (files → storeonce), whereas the .VBK files can’t as they are ‘read only’.
You can’t delete .VBM using (home → backups) if your datetime is correct on both devices.
But if the datetime of your StoreOnce is somehow in the future it seems to delete it.

In the Veeam documentation - immutability section it states that Metadata files of a backup chain (.VBM) cannot be immutable because they are updated on every job pass. This explains its behavior of being able to delete them.
https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository_limitations.html?ver=120

Although this behavior seems correct it is arguable to have these files not to be deleted from GUI when the Storeonce time is in the future (Home → Backups) and your backupserver is not as the immutable .VBK files are still there. (feature request #1) and/or (Files → Storeonce) without changing the Storeonce time (feature request #1b). 

VBK import for HPE StoreOnce devices

If there are no more .VBM files but there are .VBK files still available on the StoreOnce Catalyst due to immutability; There is no GUI option to import standalone .VBK from GUI from HPe Storeonce devices (Feature request #2). A rescan doesn’t import them either. A workaround using Powershell worked for us:

$repository = Get-VBRBackupRepository -Name "StoreOnce"

Import-VBRbackup -FileName "StoreOnce://10.0.0.1:Backup_Store/Backup_Job/Backup_Job.vbk" -Repository $repository

Source: https://helpcenter.veeam.com/docs/backup/powershell/import-vbrbackup.html?zoom_highlight=import-vbrbackup&ver=120

A ‘funny’ thing to do is try to delete the .VBK again after manual Powershell import. Deletion fails due to immutability but it is removed from GUI and you need to import it manually again …. :-)

A few remaining notes:

8Test situation]
1. After changing the time to the future on the Storeonce only (no rescan), we found that deleting files through “Home → Backups” gave us a message of it being immutable (and .VBM files are removed).
2. However when using “Files → HPE Storeonce”, it deleted the file without warning.
*. If you however rescan after trying to delete in step 1, you loose your backups in the GUI because the delete action earlier removed the .VBM files as they were not immutable, the .VBK files are still on storeonce.

wTest situation - extra note]
Changing the time to the future on the Storeonce only (no rescan), also gave us another strange message when deleting even though the options are obviously enabled the error message says they aren’t: “immutability is not supported by the repository. Dual Authorization is disabled on the specified HPE StoreOnce Catalyst Store. Please enable Dual Authorization.”
→ Changing the time to present gave no more errors. (Feature request #3 - different error message?)

Hope this helps anyone else 🙂.

@SDV Thanks for sharing your insights. Here’s the forum post: https://forums.veeam.com/post480569.html#p480569

Alexander Fogelson also mentioned the Powershell cmdlet. I was able to reimport the backups but FLR was failing afterwards. Could you please check if you see the same behaviour in your environment?

Maybe you want to share your feature requests in the post above. That way we have everything in the same topic.

 

 

 

Hi @regnor, I have a question. In case we have a copy with a 7-day retention and it activates the 1-month immutability, is this copy that is expired according to its retention deleted by Veeam in compliance with the assigned retention? Or due to immutability is it kept until one month old?

Hi @regnor, I have a question. In case we have a copy with a 7-day retention and it activates the 1-month immutability, is this copy that is expired according to its retention deleted by Veeam in compliance with the assigned retention? Or due to immutability is it kept until one month old?

This would be kept for the 1-month immutability period versus the retention period.

@jeremias.hernandez The configured immutability wins over the retention in your copy job. I'm just not sure if you'll see an error or only an informational event in Veeam.

Comment


Terms & Conditions