How to avoid an Immutability Migraine
With direct to object storage backups coming in Veeam Backup & Replication (VBR) v12, we expect to see many more folks using object storage for their backup repositories. This means there will be folks using object storage for the 1st time. If they are like me they will probably use the graphical user interface (GUI) native to the object storage platform to manage their object storage.
The native GUI will definately make the management easier, but it can also cause some headaches that you don’t expect. The reason I am writing this is to help you avoid one such migraine when you are using your object storage’s native GUI to configure your object storage bucket to create an immutable object storage repository in VBR.
The Scenario
The 1st thing you need to do to create your immutable object storage repository is to create the bucket which will store your VBR backup data. For this scenario we want to create the bucket with immutability (object lock) enabled and have the objects immutable for 30 days.
So we head over to our object storage’s native GUI and configure the bucket with the values we want. We know we need to use Compliance mode because we checked Veeam's online documentation to check what the considerations and limitations are for object storage repositories:
To verify that the object lock settings were applied correctly when the bucket was created we can use the same GUI to see the bucket’s lock details:
After we verify the bucket settings, we can head to the VBR console to create the object storage repository. But when we try to create the object storage repository we get this error:
Based on the error message the initial response would be to think either the bucket was created with the wrong settings or the object storage platform doesn’t support Compliance mode and isn’t compatible with Veeam Backup & Recovery.
Since we checked the settings on the bucket earlier when we created it that certainly can’t be the cause. Additionally, we checked the Veeam Ready Object with Immutability site to verify compatibility with VBR prior to purchasing the object storage platform. So we know this isn’t the cause of the error.
So what is the cause?
The Cause
Well, it turns out that our initial assumption was incorrect. The cause of the issue is in fact the bucket we created. But how can that be? We checked and double checked the settings on the bucket.
To explain the issue, we need to use the AWS Command Line Interface to get to the bottom of it.
Let’s start by looking at the bucket we created:
aws s3api get-object-lock-configuration --bucket test-bucket --endpoint-url=<my object storage>
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 30
}
}
}
}
The object lock configuration has a rule that specified that the mode is set to “COMPLIANCE” and the immutability is set to 30 days. Looks good. Right?
Nope!!! That is actually the problem.
Veeam doesn’t support buckets with default retention rules and it should look like this:
aws s3api get-object-lock-configuration --bucket test-bucket --endpoint-url=<my object storage>
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
}
}
Why? VBR doesn’t use bucket level locking. When an object is PUT (written) to the bucket, VBR applies the object lock settings on the individual object.
The Solution
We have two options to solve this issue and both use the AWS CLI.
- We can create a new bucket
- We can “remove” the default object lock rule from the existing bucket
To create the bucket from the AWS CLI, you can use this command:
aws s3api create-bucket --bucket test-bucket --object-lock-enabled-for-bucket --endpoint-url=<my object storage>
You can verify this using the AWS CLI get-object-lock-configuration command shown earlier and below. Or you can use the object storage platform’s GUI:
This looks great.
The other option we have is to remove the default object lock rule:
aws s3api put-object-lock-configuration --bucket test-bucket --object-lock-configuration "ObjectLockEnabled=Enabled" --endpoint-url=<my object storage>
Now when you look at the bucket’s lock configuration, we don’t see a default object lock rule:
aws s3api get-object-lock-configuration --bucket test-bucket --endpoint-url=<my object storage>
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
Now the bucket can be used when we create the object storage repository within Veeam Backup & Replication.
The moral of this narrative is to be careful when using an object storage platform’s native console/GUI when creating bucket(s) that you intend to use with Veeam’s software. If the interface requires you to assign an object lock mode and/or object lock duration, you now have the knowledge required to verify that the bucket was created in a way that will work with Veeam. If it wasn’t, you also have the steps required to create a new bucket or “repair” the existing one.
Hopefully this will prevent and/or solve this particular immutability headache.
