Skip to main content
  • EN
Solved

S3 object buckets

Hello,

 

I would like to have your feedback about how to manage S3 backup.

I mean, on the service provider S3 

Créate access key with correct privilege

Then I need to create a bucket. What name do you choose ? Is there use case where several buckets are needed for end customer ?

And for the folder, same question.

I mean, it’s more about governance and to be sure to understand correctly the pro and con.

Do I need 1 bucket / job ? 

Do I need 1 bucket for all backup job and an other for backup copy ?

Do I need 1 bucket for all jobs type ?

Then, for the folder, 1 job per folder ? 1 job per job type ? 1 folder for all ? 

 

I’m totally new to the object storage and would like to understand what are the best practice for this :)

 

Thanks for your answer :) 

 

6 comments

MicoolPaul
Userlevel 7
Badge +22

Hi @damien commenge!

Aside from the obvious ‘it depends’ here’s what I think you should know:

 

Do I need 1 bucket / job ? 

No, though you should be mindful of how far your S3 storage is designed to scale to with Veeam within a single bucket. Object Storage performance and stability is heavily dependent on database performance with some storage platforms in-particular. AWS/Azure for example will scale to hundreds of TBs without breaking a sweat, but I saw within one implementation that a vendor specified to size the buckets no larger than 50TB unless using 4MB blocks.

Do I need 1 bucket for all backup job and an other for backup copy ?

Nope.

Do I need 1 bucket for all jobs type ?

You don’t have a need to create multiple buckets per job type, or anything like this. It’s down to you to decide how you would like to split any buckets however with the potential size maximums I mentioned above.

Then, for the folder, 1 job per folder ? 1 job per job type ? 1 folder for all ? 

Again, not necessarily.

 

Here’s what I would suggest you do before designing any object storage. Look at IAM/STS.

Within object storage, you have three access modes:

  • Insecure Direct Access
  • Secure Direct Access
  • Secure Indirect (Gateway) Access

 

I’m writing up my session on this into a blog but for now I did a session at the recent UK VUG that discusses this: 

 this is roughly where I started talking about IAM/STS but you might find the entire session useful as I cover the different access modes too. In essence you can have all backup data traverse one or more gateways to reach your object storage, which is supported but you could be creating centralised bottlenecks, and your gateway ensures data isolation between each tenant. Alternatively you can leverage IAM/STS to ensure that every tenant/endpoint can write their backups to object storage directly (faster, assuming the object storage is publicly accessible). IAM/STS means that whilst Veeam has the root key for the entire bucket, it generates sub folders with limited scope keys for your tenants to use, so that they can never see a neighbouring tenant’s data.

 

Luca Dell’Oca wrote a paper on this subject with service providers in mind and is also well worth a read: VBR12-and-Secure-Mode.pdf (fromthearchitect.net)

 

Hope this helps you making a start on your designs :)

Michael Paul - Opinions are my own and do not necessarily reflect the opinion of Veeam | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.com
Chris.Childerhose
Userlevel 7
Badge +21

Well I think Michael has pretty much covered it.  😂

Chris Childerhose - VMCA2024 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 4* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
D
Userlevel 6
Badge +4

@MicoolPaul

Thanks a lot for your answer.

On my scenario here it’s only for 1 customer and installed on his site. It’s not for “service provider”.
I know S3 will be provided by Netapp fas2820.

I’m going to watch your video to better understand how to use it !

D
Userlevel 6
Badge +4

OK, I just finish to look at both documents. There is something I totally don’t understand. Why they are both talking only about agent backup ? 

On my side, it’s just about VM and nas today. I don’t seen anything about “standard” backup on the PDF and on your video. Is it not compatible ? 
My object storage will be used as backup copy destination.

Backup to Hardened linux repository and backup copy to Netapp S3 compatible. 

SOBR is expected here. 

Is it applicable in my scenario too ? 

dloseke
Userlevel 7
Badge +6

OK, I just finish to look at both documents. There is something I totally don’t understand. Why they are both talking only about agent backup ? 

 

As a Service Provider, it’s possible to install the Veeam Agent on individual servers and backup direct to object.  The Service Provider Console provides the repository information including the credentials, but the data doesn’t flow through the console and the agent instead connects direct to the object repository at your provider of choice.  However, I’m not sure about doing this outside of the service provider realm, but that’s the way I view it and why it would be talked about in regards to the agent connecting to the repo in this manner.

 

On my side, it’s just about VM and nas today. I don’t seen anything about “standard” backup on the PDF and on your video. Is it not compatible ? 
My object storage will be used as backup copy destination.

Backup to Hardened linux repository and backup copy to Netapp S3 compatible. 

SOBR is expected here. 

Is it applicable in my scenario too ? 

VBR can certainly backup direct to object, or copy direct to object.  SOBR is no longer needed, but is still possible as well.  Multiple buckets or folders are not needed when it’s for a single repository.  One thing I want to note is that all data is handled by a single server, a single bucket is fine.  However, if you have multiple servers that will accessing the account, a separate bucket is recommended for each server.  This is because each server keeps a database of the data in the bucket, so if one server uploads data into a bucket, performance will be affected as other servers accessing the bucket will be surprised to see foreign data in that bucket and I believe will need to index that data.  It doesn’t sound like this is your scenario, but I just wanted to call that out.  One thing I haven't investigated is if this would have the same effect if each server used the same bucket but a different folder within the bucket.  That said, I just prefer to keep my buckets separate, using different credentials for each bucket with policies set to isolate the buckets.

I just realized that a while back I write a blog post about my setup.  I believe it’s a bit dated because I believe there’s more capability here, as well as Wasabi now supplies WACM for free (they charged $100/month when I posted this) so you can actually create separate sub-accounts for each tenant/customer/reseller/whatever you see fit.  I posted the link below for reference. 

Also, the way I’ve done this pretty well completely ignores the IAM/STS policies that Michael mentions in his response above.  Reading his response, my assumption is that I’m doing everything more manually that what it appears may be possible using IAM/STS and allowing Veeam to manage the limited scope keys to folders within a bucket.  Not that I think the way I’m doing things is wrong, but there may be a better way.

Derek M. Loseke, Senior Systems Engineer | Veeam Legend 2022-2023 | VMSP/VMTSP | VCP6-DCV | VSP/VTSP | CCNA | https://technotesanddadjokes.com | @dloseke
D
Userlevel 6
Badge +4

Thanks a lot for your answer :)

Comment


Terms & Conditions