Veeam Amazing Object Storage Tips & Techniques Part 1

  • 14 November 2022
  • 8 comments
  • 1804 views

Userlevel 6
Badge +2

 

How to avoid an Immutability Migraine

With direct to object storage backups coming in Veeam Backup & Replication (VBR) v12, we expect to see many more folks using object storage for their backup repositories.  This means there will be folks using object storage for the 1st time.  If they are like me they will probably use the graphical user interface (GUI) native to the object storage platform to manage their object storage.

The native GUI will definately make the management easier, but it can also cause some headaches that you don’t expect.  The reason I am writing this is to help you avoid one such migraine when you are using your object storage’s native GUI to configure your object storage bucket to create an immutable object storage repository in VBR.

The Scenario

The 1st thing you need to do to create your immutable object storage repository is to create the bucket which will store your VBR backup data.  For this scenario we want to create the bucket with immutability (object lock) enabled and have the objects immutable for 30 days.

So we head over to our object storage’s native GUI and configure the bucket with the values we want.  We know we need to use Compliance mode because we checked Veeam's online documentation to check what the considerations and limitations are for object storage repositories:

To verify that the object lock settings were applied correctly when the bucket was created we can use the same GUI to see the bucket’s lock details:

After we verify the bucket settings, we can head to the VBR console to create the object storage repository.  But when we try to create the object storage repository we get this error:

Based on the error message the initial response would be to think either the bucket was created with the wrong settings or the object storage platform doesn’t support Compliance mode and isn’t compatible with Veeam Backup & Recovery. 

Since we checked the settings on the bucket earlier when we created it that certainly can’t be the cause.  Additionally, we checked the Veeam Ready Object with Immutability site to verify compatibility with VBR prior to purchasing the object storage platform.  So we know this isn’t the cause of the error. 

So what is the cause?

The Cause

Well, it turns out that our initial assumption was incorrect.  The cause of the issue is in fact the bucket we created.  But how can that be?  We checked and double checked the settings on the bucket.

To explain the issue, we need to use the AWS Command Line Interface to get to the bottom of it.

Let’s start by looking at the bucket we created:

aws s3api get-object-lock-configuration --bucket test-bucket --endpoint-url=<my object storage>

{

    "ObjectLockConfiguration": {

        "ObjectLockEnabled": "Enabled",

        "Rule": {

            "DefaultRetention": {

                "Mode": "COMPLIANCE",

                "Days": 30

            }

        }

    }

}

 

The object lock configuration has a rule that specified that the mode is set to “COMPLIANCE” and the immutability is set to 30 days.  Looks good.  Right?

Nope!!!  That is actually the problem.

Veeam doesn’t support buckets with default retention rules and it should look like this:

aws s3api get-object-lock-configuration --bucket test-bucket --endpoint-url=<my object storage>

{

    "ObjectLockConfiguration": {

        "ObjectLockEnabled": "Enabled",

    }

}

Why? VBR doesn’t use bucket level locking.  When an object is PUT (written) to the bucket, VBR applies the object lock settings on the individual object. 

The Solution

 We have two options to solve this issue and both use the AWS CLI.

  1. We can create a new bucket
  2. We can “remove” the default object lock rule from the existing bucket

To create the bucket from the AWS CLI, you can use this command:

aws s3api create-bucket --bucket test-bucket --object-lock-enabled-for-bucket --endpoint-url=<my object storage>

You can verify this using the AWS CLI get-object-lock-configuration command shown earlier and below. Or you can use the object storage platform’s GUI:

This looks great.

The other option we have is to remove the default object lock rule:

aws s3api put-object-lock-configuration --bucket test-bucket --object-lock-configuration "ObjectLockEnabled=Enabled" --endpoint-url=<my object storage>

Now when you look at the bucket’s lock configuration, we don’t see a default object lock rule:

aws s3api get-object-lock-configuration --bucket test-bucket --endpoint-url=<my object storage>

{

    "ObjectLockConfiguration": {

        "ObjectLockEnabled": "Enabled"

    }

}

Now the bucket can be used when we create the object storage repository within Veeam Backup & Replication.

The moral of this narrative is to be careful when using an object storage platform’s native console/GUI when creating bucket(s) that you intend to use with Veeam’s software.  If the interface requires you to assign an object lock mode and/or object lock duration, you now have the knowledge required to verify that the bucket was created in a way that will work with Veeam.  If it wasn’t, you also have the steps required to create a new bucket or “repair” the existing one.

Hopefully this will prevent and/or solve this particular immutability headache.

 


8 comments

Userlevel 7
Badge +20

This is a really great article, Steve.  Hopefully this topic helps those using OBS with Immutability.

Userlevel 7
Badge +8

Very handy to know this. I’m pretty new to object storage and am slowly realizing that the command line is key to getting it right. 

Userlevel 7
Badge +6

Great article.  I haven’t run into this using Wasabi as I don’t think we can actually sent retention on the bucket level there, but it could certainly be an issue for other object storage providers/applications.  For the purposes on your demonstration, mind if I ask what you were using for the object storage?  

Userlevel 6
Badge +2

Derek,

I used Ceph 17 (Quincy Stable) for the example in this blog.  You are correct via Wasabi’s website, creating the locked bucket will not cause you any issues.

However, if after you create the bucket you were to go to the setting of the bucket and turn on/configure the “Bucket-Level Retention” shown below then you would have the issues described above. 

So avoid any such settings via any product’s interface and let Veeam do the work for you.  Hope this answers your question 👍

Userlevel 6
Badge +2

Thank you for sharing this great article @SteveF ,

Is this also work for the QNAP QuObjects?

Userlevel 6
Badge +2

@vAdmin QNAP QuObjects should work with the AWS CLI and the commands that I used in the article.  I have a QNAP device in my home lab and will load QuObjects at some point to test this.  I will post an update when that testing occurs.

Userlevel 6
Badge +2

@vAdmin QNAP QuObjects should work with the AWS CLI and the commands that I used in the article.  I have a QNAP device in my home lab and will load QuObjects at some point to test this.  I will post an update when that testing occurs.

Thank you very much, @SteveF  I'm eager to hear the QNAP testing results because I use it both at work and at home.

Userlevel 7
Badge +6

Great topic!

Comment