Yes, that‘s the way, veeam implemented it:

https://helpcenter.veeam.com/docs/backup/cloud/cc_capacity_tier_download.html?ver=110
 

With the powershell command, only files from the active chain will be downloaded. That‘s much faster to get back the files for the tenant in his cloud repo:

https://helpcenter.veeam.com/docs/backup/powershell/start-vbrdownloadtenantbackup.html?ver=110

But if you need older restore Points, everything needs to be downloaded first by the service provider.

Veeam doesn‘t have a feature, to choose which vm needs to be downloaded and which restore point. That‘s nothing your or another service provider can do better here. Some service provider might to use an on premise object storage in their datacenter. That help to get lower download times, to get the data back. But not with a cloud service like wasabi.

Downloading from Wasabi could get really time intensive, because it‘s s cloud service. The service provider needs to download all data from the internet. Wasabi could be doing some limitation there. Second, there is the limitation from the internet connection. 

I would check your SLA with that service provider. Does he guarantee a time until he has that „archive“ data back to you?

@Mildur makes some fantastic points and its the downside of them offloading to another cloud, archive tier is designed not for time sensitive restores, with the benefit of utilising less performant storage.

The main questions I’d be asking (potentially if your service provider):

- Was the cloud provider being forthcoming about how much storage was being using on their performant storage vs wasabi?

Regardless of whether they called it wasabi or just capacity/archive storage. It’s not uncommon for providers to white label such services so they have the flexibility to migrate between them if they need to.

- Was there any mention of storage tiers at all?

If they haven’t communicated to you at all about any of this, directly or in a contract, sounds like a legal dispute could be underway depending on how they’ve advertised their services…

- Have you got any SLAs on recovery?

Cloud based backups inevitably have worse recovery times vs local backups (aka your primary backup repository), it’s important to identify though whether this is an aged recovery or if your cloud provider are being too aggressive offloading backups vs your requirements. Like they say “everything is negotiable” to an extent “everything is configurable”, now you know of the recovery pains, you may want to negotiate for more of their high performance storage and have backups offloaded when they’re older.

To answer your other questions, migrating between cloud providers I would 100% engage Veeam support as it’s not a typical scenario. I don’t see any reason why all your backups couldn’t be downloaded and then uploaded and remapped.

And yes the offload is reliable, you’re working with static files so it’s incredibly tolerant to poor performant networks.

I know none of my responses are a magic bullet, but hopefully they help steer you forward

Thanks. It’s good to know this is at least the normal process. Had I known this we would have implemented it differently. We questioned the provider when setting up the account if it would be better to have one cloud repo for each office location instead of setting up one cloud repo for all our offices and they said that the one repo was fine. Wouldn’t this be quicker with separate cloud repos for each office since they would only need to download a lot less data now? Also how do they get the data back to archive storage and is that a safe process? Concerned about that since we have no way to test after they are back.

Charlie

Downloading data is per tenant.

Your service Provider would need to create a Tenant per each office.

Moving back is not possible. I don‘t know, how they want todo that:

Downloaded backups remain in the performance tier and cannot be moved back to the capacity tier.

https://helpcenter.veeam.com/docs/backup/cloud/cc_capacity_tier_download.html?ver=110

Perfect explained @Mildur and @MicoolPaul . As a VCC service provider we will renew and upgrade our VCC infrastructure. We will use primary storage on regular server-hardware as being a best practice and will copy all data on the primary storage to immutable object storage appliances in another datacenter. This is in my opinion the best way : you as a service provider have full control of your VCC infrastructure, not depending on a hyperscale cloud provider with costs, limitations on bandwidth, etc...

@Nico Losschaert 

That‘s my design too.

With the copy policy, the customer has always access to all of this restore points in the cloud repo.

With the move policy, he looses his direct access to the restore points.

Indeed @Mildur, absolutely true, with the copy policy the tenant should not notice if a primary SOBR extent is out-of-business . Nice when 2 experienced Veeam engineers/lovers and legends are having the same idea 

Yeeah :-) 

Thank you for all the replies. This is our first try with cloud backup so definitely learning an important lesson here. It sounds like I would be much better off just keeping everything on local performance storage. 
 

Any additional feed back on these points would be great.

• should I be concerned about the integrity of my archive backups considering 170 TB is a pretty large amount to download?
• even though my provider says they will move it back to archive storage when I am done restoring its seems like most of you say that’s not possible. Anyone else out there ever heard of this being possible?

Thanks again,

Charlie 

Veeam will take care of the download. I am really sure that they do an integrity check after downloading the objects. If the checksum is not correct, it will be re downloaded again.

Second, not the entire 170TB will be downloaded. If the block of a restore point is already in the performance tier, this block will not be downloaded from the capacity tier.

I think, your 170TB are multiple synthetic GFS Fulls from the same vms? Then the download will most likely only be all increments. If the backup files on the performance tier were active fulls, then all data will be downloaded for sure.

For offloading the data again, the veeam guide says, that is not possible. I trust it.

Agree with @Mildur. Not the whole 170TB will be downloaded, but impossible for us to say how much. Luckily enough the SP is using Wasabi and not Azure, AWS, … otherwise they would have a lot of egress costs. At Wasabi there are normally now extra costs involved when tiering data away from the public cloud vendor. Good luck with it? Curious about the time being involved when the data is available.

Thanks I am curious about the time it will take also. It’s also going to be interesting if they now want to charge me for more performance storage when they realize it can’t be moved back.  All I need is data for one project. To be honest if I knew what was involved I probaly would not even had continued but the provider started the process before we understood all this. In any case it’s been a real eye opener.
 

If I could be so greedy to ask one more question. I am curious about @Mildur and @Nico Losschaert saying they use “primary storage on regular server-hardware as being a best practice and will copy all data on the primary storage to immutable object storage appliances in another datacenter.” Does this mean all client backups including what they consider historical like quarterly and yearly are on performance storage and the copy you do to immutable object storage is just a backup of the backup?

Thanks again it’s nice to find a forum that actually has some experts on it. Most don’t and are just a waste of time.

Charlie

The customer has his local onpremise backup Repo. He creates a backup copy job to copy his backups to our cloud connect repo.

On the Service Provider Infrastructure, a SOBR is used with a performance tier (windows or Linux server with direct attached disks, or a iscsi/FC Storage Appliance) to hold all restore Points which the customer has copied to his cloud repo.

Every daily Restore Point, every gfs (weekly, monthly, quarterly, yearly).

This SOBR is then configured to copy all restore points to the capacity tier, as soon the customer upload them.

The capacity tier on the service provider site can be a S3 compatible Object Storage in their own datacenter or it can be a cloud service like amazon, azure or wasabi. There are many options to choose from.

As you said it, it‘s a copy of the copy. Like a hardware raid, if one disk is failling, the other disks can be used to get access to the files. This design is a protection against failed storage systems. If the customer uploads his entire historical data to us, then this data needs to be protected with the 3-2-1 rule.

By the way, with Veeam V11, quarterly GFS is discontinued You won‘t have that option in the job settings.

Thanks you again incredibly helpful.

Charlie

Perfectly explained by @Mildur !