Question

Slow execution of Jobs (replications and backup)


Good evening, several of my Replications and backups take a long time to complete and I would like to know the cause and the means to resolve this.

 

 

 


9 comments

Userlevel 4

Could you share the sizing of your veeam platform components or characteristics of each veeam component of your veeam platform ?

Userlevel 4

CPU, RAM, Disk of each component

Userlevel 7
Badge +8

I would also add, that check if your backup sizes increases significantly or not.
Do you have any Proxy servers deployed? 
Or all the effort comes from the Veeam Console?

Finally, if this persist, I would also suggest to open a support case,
they can look at it, and also tell you tips and registry keys for performance and other options.

cheers.

 

Userlevel 7
Badge +17

As shared above..need significantly more info on your backup environment before we can offer suggestions on how to improve your b/u & repl performance and why it may be running sluggish. Some specific info needed is as follows:

What backup method are you using? Forward, Forever Forward, or Reverse?
What backup & replication transport mode are you using? DirectSAN, BfSS, Hotadd, or Nbd
Do you have Proxies, WAN Accelerators, and/or Gateways (if needed) at your job source & target areas?
And, as stated, what system resources do you have assigned to components?
Also, are you using the Veeam server as an all-in-one system (Basic/Simple deployment) or Advanced/Distributed (of your components) Deployement?

It also may be beneficial to look at Veeam’s Best Practice Guide here which discusses design and performance suggestions.

Userlevel 7
Badge +7

@DAMASE237 - just taking a first glance at this - I think it looks like you don’t have enough proxy/data mover processing capacity.

For component sizing you can use the VSE calculator at https://calculator.veeam.com/vse.

The recommendation for sizing the tape server can be found in the Veeam UG - https://helpcenter.veeam.com/docs/backup/vsphere/system_requirements.html?ver=120

 

also - when you perform replication it helps to make sure that the proxy count at the target side matches the proxy count at the source side.

Userlevel 7
Badge +20

Few thoughts based on the screenshots.

 

looks like a lot of cross-site replication (I notice some replication jobs say Port-Ville whilst others say Ville-Port.

 

In cross-site traffic scenarios your cross-site bandwidth will most likely be the bottleneck. How fast is the cross-site bandwidth if so? Are you using WAN accelerators or not?

 

It looks like you’ve got a lot of concurrent jobs, I’d make sure any jobs that don’t need cross-site traffic aren’t sharing any cross-site resources such as the same backup Proxies

Userlevel 7
Badge +6

Slow performance has mush factors:

Wrong Veeam components sizing

Network and Storage speed.

Veeam component settings

First thing to do is to monitor the Bottleneck in the backup job progress, which is the bottleneck, Source, target, network, or proxy. Then start troubleshooting.

Userlevel 7
Badge +8

Even if the proxies are small, and you are not running a lot of parallelly tasks, if it’s still slow, I’d look into recourses. 

Here are the things jump out to me.  The best practices guide helps with many of them.

  1. Network. Do you have the recourses to be copying these jobs remotely. What is your bandwidth, what is the latency between sites. How much data change is there per day? how much traffic is being sent? Is this the first time sending the data, and incremental, active full? 
  2. Storage. How many IOPS can your SAN handle? What is the latency in your storage? Is it fiber, Iscsi, direct attached? Is there a disk queue building? 
  3. Veeam Infrastructure. What kind of networking, CPU and Memory do you have in your proxies and repos? are they all combined on 1 or many servers? are they physical or virtual? 
  4. What backup method are you using? This could take us back to networking if it’s using the VMware management ports or your backup type is trying to use virtual proxies etc. 

 

Let us know more about your environment. Veeam just does the work, but it can only do so much with what you give it. When you look at the jobs what does it say the bottleneck is? 

 

Userlevel 7
Badge +8

I’ll add, I see your monthly tape job running. I don’t know how much data, or how many LTO drives you have, but if you are streaming to 6 or 8 LTO drives it is reading ALOT of data. When I do my Weekly tape backups my latency goes up  very high on my SAN.

 

If you have a slower repository, or lots of spinning disk, I see you are running copy jobs and incremental as well. Look at your latency on the storage. I know with high latency on my SAN full of spinning disks it slows down the copy jobs and incremental until the tape jobs are completed. 

Comment