Transferring data from Linux Repo to OnPrem S3 created enormus excess space usage

Question

Data copies from the Linux hardened repo to S3 On Prem lblows up spacew up space consumption on object storage without a clue why.

In this case, the disk consumption on the Linux Repo is about 900TB (after compression & dedup) on XFS. Copying the same data to Object storage scale-out-repo (using 4MB block size as recommend in best-practices guide), object storage size is currently 1.66PB of data and the cluster is almost full (86%)

As the disk space was calculated using the VEEAM sizing tool, we calculated with about 1.1PB on object storage with additional space to grow.

And the second thing is, that the S3 object deletions object storage receives from VEEAM are much less than they should be, but the immutability time (+ 3 week) has already reached, so there must be much more objects deleted than what is actually happening.

icon

Best answer by SteveF 13 December 2023, 16:52

View original

SteveF · Accepted Answer

@DrObject, thanks for the question. Veeam’s best practice and recommended block/object size is 1MB which you can find in our HelpCenter documentation for Storage Optimization. When using larger block/object sizes like 4MB, we have seen storage increases of 2x or greater. The larger the block/object size the results from deduplication and compression will be significantly reduced.

In this scenario, you are moving from XFS who’s default block size is 4KiB (4,096 bytes) so the deduplication is expected to be greater than when using 4MB (4,000,000 bytes).

As far as the deletions are concerned, we do reuse some objects are references for future backup chains due to our use of fast-cloning to reduce duplicate objects. So if you have a 3 week retention period, it isn’t expected that on week 4 you will see an entire week’s worth of storage reclaimed. Over time, backup chains that are no longer being used will be removed.

As always if you feel that something isn’t working as designed, please contact Veeam Support and they will help you determine if there is an issue that needs to be addressed.

JMeixner · Answer

Perhaps the block cloning savings are nor preserved during the transmission between the two storages or the block sizes are not optimal?Just my first thought on this while reading it...

Transferring data from Linux Repo to OnPrem S3 created enormus excess space usage

3 comments

Comment

Comment

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded