Solved

Backup Copy Job + WAN Accelerator


Userlevel 7
Badge +1


Hello to all,

I configured Backup Copy Jobs to the second location via VPN.
Two WAN Accelerators on source side and one on destination.

VM’s with low disks are going perfectly but the problem is the big ones.

Our FileServer is arround 7TB. Is it normal to take this long, its the first Backup Copy job without initial seed.


 

icon

Best answer by MicoolPaul 22 March 2023, 12:32

View original

30 comments

Userlevel 7
Badge +6

Super helpful thread.  Thanks Michael, and thanks OP for the super-fast and informational responses getting to the root of the issue.

Userlevel 7
Badge +1

Great,

I really appreciate your help today. I learned a lot. 
Thank you
 

Userlevel 7
Badge +20

Yes, now its great. Thank you really much.

I just have one more question. 
Can I configure server to be gateway and also WAN Accelerator or I need to configure 2 servers?

I think that configuring WAN Accelerator should give better performances now. 

If you do this, then definitely use High bandwidth mode based on the bandwidth metrics you provided, as a guidance:

 

<=100Mbps = Low Bandwidth mode

100Mbps< Bandwidth >=1Gbps = High Bandwidth Mode

>1Gbps = Direct

 

As long as your servers have sufficient resources to run the multiple roles, this should be fine to use gateway & wan accelerator on the same server.

Userlevel 7
Badge +1

Yes, now its great. Thank you really much.

I just have one more question. 
Can I configure server to be gateway and also WAN Accelerator or I need to configure 2 servers?

I think that configuring WAN Accelerator should give better performances now. 

Userlevel 7
Badge +20

Nice one! This would’ve also been impacting WAN Accelerator btw as WAN accelerator would've been talking to the gateway servers. Which, if you deployed a WAN accelerator at each site would’ve made the data flow:

 

Source Synology → VBR Gateway at source → WAN Accelerator at source → Wan accelerator at target → VBR Gateway at source (for target gateway though) → Synology at target.

 

So you’d have thrown traffic over the wan to the target wan accelerator, then back across the link to the vbr gateway in your source site, then back across the link again to talk to the synology at target!

 

We’ve certainly made things more efficient here!

Userlevel 7
Badge +1

Thank you really much.
This cleared my mind. Its already looking better.

 

 

Userlevel 7
Badge +20

Your data flow is currently as follows then:

 

Source Synology NAS → Source Gateway Server (Likely VBR server) → Destination Gateway Server (Also VBR server so just loopback traffic) → Destination Synology NAS.

 

Which looks simple enough, but the issue occurs with IO requests, as the VBR server is having to manipulate a large backup file on the destination Synology NAS with reads and writes both traversing the WAN.

 

If you swap it to:

Source Synology NAS → Source Gateway Server (Likely VBR Server) → Destination Gateway Server (within Destination site) → Destination Synology NAS.

 

Then the WAN is only being used to shift blocks from source to destination gateway servers that need to be committed, and the gateway server is handling the IO read/writes to Synology at LAN speed.

Userlevel 7
Badge +1

I understand now what you mean.

Yes, on the site where the files needs to be copied gateway server is VBR which is located on another site.

So to sum it up I need to chose a server which is located within the site to be gateway server?
I will configure it like that and after start start the job again

Userlevel 7
Badge +17

Concur with Michael. These Gateway Servers he suggests adding will act as Proxies to perform the heavy load & lifting of the BCJ job processing.. I suspect once implemented processing rate will increase. The Helpcenter on BCJ discusses Gateways. 

Userlevel 7
Badge +20

Hi, I mean gateway servers in a different sense, when you create an SMB share, you choose a “gateway server” to communicate to it.

 

Check the source and destination SMB Shares as to what their gateway servers are.

I suspect your gateway server is your VBR server or another server located within the source site, so you’ve got a suboptimal path for reading/writing blocks on the SMB share. If so, you can deploy the gateway server role to near enough any server in the backup copy site. This will speed things up.

Userlevel 7
Badge +1

Both sites have NAS Synology that has SMB share. 
Gateways are our Firewalls, we have firewalls on each location. Fortigate to be specific

Userlevel 7
Badge +20

Hi,

 

It says your botleneck is the target. So can I ask, how are you accessing this storage? Is it an SMB share, or NFS share, or is the storage attached to a server?

 

If it’s an SMB or NFS share, where’s the “gateway server” is it within the destination site?

Userlevel 7
Badge +1

Without any initial seed

Userlevel 7
Badge +1

I started it from the scratch with Direct mode.
This is the statistics for now

Userlevel 7
Badge +17

I don't think processing rate will be lower, but I can't definitively confirm because I don't use WAN Accelerators. Depending on your total BCJ job size, your job shouldn't take 6-7 days to run. Just use Michael's formula above, but instead of the 7TB for the 1 VM in the job, use the total job size. This will give you the time it'll take to run the whole job...approximately. Again, since subsequent runs are incremental, it won't take anywhere near as long. 

Userlevel 7
Badge +20

In your scenario, I expect it to be much faster. Low Bandwidth mode, which it sounds like you were using, is only suitable up to 100Mbps anyway. So I’m expecting much faster throughput.

Userlevel 7
Badge +1

I just have a question about the BCJ without WAN Accelerator.

Processing rate, speed will be lower, so it will take even more than 6-7 days or?

Thanks for explanation :)

Userlevel 7
Badge +17

Nice BCJ thread. And really good advice there @MicoolPaul . Just to confirm his suggestion @NemanjaJanicic , I have a BCJ with Gb WAN between the two sites & use Direct mode. There's not really much around the length of time the initial job run taking so long unless you follow Michael's advice on initial seeding. For me, I was ok with the time length for my initial run. Subsequent incremental runs do take significantly less time. If you can be patient through the 1st run you should be fine here on out. 

Cheers! 

Userlevel 7
Badge +20

If the backup is gone, don’t worry about setting up sharesync again, I’d always be more worried about data corruption on such a large backup file. Just do the full BCJ without sharesync or wan accelerator 🙂

If you’re worried about your other VMs not processing whilst this large one runs, I’d create a separate BCJ for the big VM so it doesn’t prevent another sync cycle for your smaller VMs during the initial full backup copy 🙂

Userlevel 7
Badge +1

Thank you for this explanation.

So I can try to use it without WAN Accelerator. With that option there is no need to wait for one to be over so that next can start Copying.

I dont have it anymore but I can create the ShareSync again just to have the initial seed.
After that I can start With BackupCopy Jobs. Do I need first to “populate”, “rescan” repository so that Veeam can know that full backup is already there.

For VM’s that have small disks I will continue using WAN Accelerator because its working just fine. Maybe I can enable high bandwith mode for them just to be “quicker” if it can

Userlevel 7
Badge +20

Okay, my advice would be:

  • Cancel the BCJ, change processing mode to direct.

Yes your others will be waiting whilst your full backup takes place, it would’ve been possible to use the synology sharesync backup as a ‘seed’, to avoid another full, if Veeam validated it wasn’t corrupted, if you still have this backup, you could attempt to use it as a seed for your BCJ.

Assuming everything is equal, as in no bottlenecks greater than that of your WAN connectivity, the maths should look like the below:

 

Lets call the upload 550Mbps, as you’re not likely to hit 100% utilisation all the time.

550Mbps = 68.75MBps

I’m not going to get into the exact of Gigabyte vs Gibibyte so lets just say it’s 7 Terabytes, or 7,000,000MB.

 

7,000,000 divided by 68.75MBps = 101,818 seconds, or put in minutes: 1696.96 minutes, or 28.28 hours approximately.

 

So if you don’t have a usable backup in the destination to use as a seed, you can just disable WAN accelerator, throw the data directly to your offsite repository, and within say, lets round this up to <=1.5 Days, you’ll have a full copy there, and can carry on as normal.

This doesn’t factor in exactly any production WAN use, permitted transfer periods etc, but at the high level I believe the point makes sense.

Userlevel 7
Badge +1

But the best option to have backups on both sites would be to use Backup Copy Job?
I tried with synology sharesync, I have 20 Instances with our license so Backup Copy Job seems the best.

But if it will take 6-7 days everytime its not good. Especially that I have one WAN Accelerator for 3 VM’s. 1 is 7TB and other 2 are arround 1.5TB.

Until 7TB is finished others are waiting.

Userlevel 7
Badge +20

Either don’t use wan accelerator or use high bandwidth mode 🙂 high bandwidth mode will try to give you a higher bandwidth throughput relative to your consumed bandwidth and is effective somewhat with this up to 1Gbps.

 

Personally, unless you have bandwidth constraints, I’d tell it to use direct (no WAN accelerator) and either use QoS or network traffic rules to limit the backup speed during working hours

Userlevel 7
Badge +1

I didnt enable High bandwidth mode.
This is the site I’m transfering files to.
The source site is a bit faster than this
 

 

Userlevel 7
Badge +20

Hi, if it’s 1Gbps are you using WAN accelerator in high bandwidth mode? I would either directly copy or use WAN Accelerator in high bandwidth only in that situation.

Comment