Block size in combination with Veeam Cloud Connect and Object Storage

We have had to move to larger blocks for Object as smaller ones don't work as well. Great post Nico. 👍

It heavily depends on which object storage you are using, it’s not a common rule for everyone. Indeed, some suffer the IO load we trow at them, and increasing the block size is an easy solution. But at expense of highly increased incremental size. @HannesK did some tests and 2x is the value we also found.

It makes total sense that a full backup has the same size, if you think about it the entire vm has always the same size, regardless if you cut it into 1mb pieces or 4mb pieces. But when it comes to incremental, a changed byte can mark an entire block as changed, so a larger block may have more chances to have changed bytes.

Some other worked by introducing trottling, either in the storage or by placing reverse proxies in front of the storage (like haproxy or others). The idea obviously is to slow down the backup process to let the storage be able to ingest data. This choice will affect the SLA, as you’re now offloading data at a slow pace.

Some providers instead went to test and select vendors that have no problem at all in ingesting our data. But as you can imagine if we need to go into these details I’d take the conversation offline.

BTW: if you go for Direct2Object IO load is even higher, as writes can now be random as in the block storage they are replacing. Capacity tier is somewhat “safe” as many writes are sequentials and so a little bit less stressful than D2O.

Keep in mind what other data you may have on your Object Storage. Eg: If you have other (non Veeam) data that is larger blocks in general, your object storage may be able to handle the smaller blocks that Veeam creates, deep in their nested folders, easier than if its all Veeam data using 1 MB blocks. 

As well, if your Object Storage is all SSD or NVMe, it might not matter at all. That said, I can’t imagine anyone deploying Object Storage as a backup landing zone using anything other than HDDs, just due to pure cost. Scaling up into PiB of data using SSDs is no small expense.

Scale of your Object Storage deployment can also have an impact to your overall performance. A million objects spanned over 8 HDDs is going to hurt. A million objects spanned over 800 HDDs is not even a concern. However, now we’re back to the first point I made - if all the data on your Object Storage is Veeam data, scale isn’t going to necessarily help you here because you’re no longer talking about a million objects. You’re talking billions if you use 1 MB blocks. As an example, a 53 TB bucket of Veeam data using 4 MB Blocks consumes 35.8 Million objects.  A 71.2 TB bucket of Veeam data using 1MB blocks contains 149 Million objects. These are real world numbers from our Object Storage. Extrapolate those numbers out for an Object Storage cluster containing PiB of data. That’s a *lot* of objects. Want to know why @Chris.Childerhose uses 4 MB Blocks? Now you know. Can you control both sides is a great question (Eg: do you manage your customer’s on premises Veeam settings), so you can set these settings?

I suspect the hyper-scalers handle this better than on premises Object Storage simply due to their scale. They can have thousands of HDDs in their Object Storage, consuming all sorts of data (not just Veeam) allowing them to spread the small object load across large swaths of HDDs. Some On Premises solutions may have a significantly sized cache tier just to help mitigate the effects of small objects.

Now, back to performance. Sure, reads/writes might be slower with 1 MB objects, but is that even *really* a problem (provided, obviously, that your object storage doesn’t fall over completely handling the ingress/egress data) and you can ingest the data fast enough to meet the customer backup job schedule?

If a customer is going to Object Storage hosted by some other vendor somewhere off in the internet, its hopefully not be their primary backup (3-2-1 anyone? Anyone? Bueller?). Generally offsite backups are not expected to have short RPOs, because there are too many variables. Internet speed, backup sizes, offsite vendor of choice, etc. If your primary backups are toast, you’re generally pretty happy just to have offsite *somewhere*. That said, there are always going to be those customers that want near zero RPOs and RTOs (that is, until they see the costs of doing so, but that’s another conversation).

All that said, that’s not specifically my point. Of course you need your Object Storage to be performant *enough* to handle ingesting data from many customers, and performant enough to handle restores. The other concern (and arguably the bigger concern) you’ll have is: What do you do when you need to rebuild that data? You lose a HDD, now what? How fast can that replacement HDD be rebuilt from your other drives in your erasure coded set? You did plan out a sizeable erasure coded set, right? Small objects can kill you here too - each object is a file on that underlying file system. Rebuild times can be quite significant. Plan accordingly.

Now, I’m not saying be scared of Object Storage with your Veeam deployments. Only just be aware it has its own set of challenges. Some Object Storage vendors may handle things better than others, but the challenges are usually pretty ubiquitous across the vendors. Some just handle those challenges differently than others. 

Setup : 

The setup of the Veeam Cloud Connect is a infrastructure that consists of several SOBR’s that consists of multiple extents (performant block-storage physical servers with REFS - 64K) in a performance tier in 1 datacenter and a capacity tier with object storage (performant physical object storage devices) in another datacenter with using the copy-mode and immutability.

Of course there is a performant interconnection between the 2 datacenters.

It heavily depends on which object storage you are using, it’s not a common rule for everyone. Indeed, some suffer the IO load we trow at them, and increasing the block size is an easy solution. But at expense of highly increased incremental size. @HannesK did some tests and 2x is the value we also found.

It makes total sense that a full backup has the same size, if you think about it the entire vm has always the same size, regardless if you cut it into 1mb pieces or 4mb pieces. But when it comes to incremental, a changed byte can mark an entire block as changed, so a larger block may have more chances to have changed bytes.

Some other worked by introducing trottling, either in the storage or by placing reverse proxies in front of the storage (like haproxy or others). The idea obviously is to slow down the backup process to let the storage be able to ingest data. This choice will affect the SLA, as you’re now offloading data at a slow pace.

Some providers instead went to test and select vendors that have no problem at all in ingesting our data. But as you can imagine if we need to go into these details I’d take the conversation offline.

BTW: if you go for Direct2Object IO load is even higher, as writes can now be random as in the block storage they are replacing. Capacity tier is somewhat “safe” as many writes are sequentials and so a little bit less stressful than D2O.