Solved

Slow Backup Speed


Userlevel 3
Badge

I have a backup environment that consists of:

 

1 physical Backup Server with a 10gbit connection to the switch that is also the primary backup repository

1 VM for VMware backup proxy configured to use virtual appliance (8vCPUs & 16GB of RAM)

3 node vSAN cluster all connected to the same switch via 10gbit ports

 

When running backup jobs, Veeam starts running well, reaching read speeds close to 450MB/s for some jobs. After a while, the read speeds drops significantly for both incremental and full backups, going down to ~70MB/s in some cases.

The backup proxy and the backup repository are configured to process 8 concurrent tasks. 

The bottleneck in all the backup jobs show 99% in source. How/where should I look for the reason of this bottleneck and what types of tests can I run to find which part is holding the backup speed down?

icon

Best answer by Chris.Childerhose 17 May 2024, 16:08

View original

13 comments

Userlevel 7
Badge +19

Hi @FellipeCM -

Backups slowing down to that speed isn’t entirely abnormal. That being said, do you have storage latency throttling configured for your environment? 
https://helpcenter.veeam.com/docs/backup/vsphere/io_settings.html?ver=120

Generally, you’d want to keep latency lower than 15-20ms on your production datastore, or your applications (VMs) will begin to show latency.

Userlevel 3
Badge

Hi @FellipeCM -

Backups slowing down to that speed isn’t entirely abnormal. That being said, do you have storage latency throttling configured for your environment? 
https://helpcenter.veeam.com/docs/backup/vsphere/io_settings.html?ver=120

 

There is not! In this environment, we are using the old socket license in the standard edition, so the option is greyed out.

What could cause it to slow down that would be considered common?

Userlevel 7
Badge +19

Ok, so what about your repository. Do you limit concurrent tasks there? Do you use Per-VM?

https://helpcenter.veeam.com/docs/backup/vsphere/repository_repository.html?ver=120

Userlevel 3
Badge

Ok, so what about your repository. Do you limit concurrent tasks there? Do you use Per-VM?

It is using per-VM backup files. We have the same concurrent tasks limit as the proxy, 8. Also, there is not read and write data rate limit applied.

Userlevel 7
Badge +19

Ok, so all those are good. Hmm...starting to run out of ideas to check. But honestly, I think you’re doing well. Do you notice anything in your Backup Job history?

Your VM Proxy seems to be fine, although having more than 1 may give you a bit better performance, as well as redundancy. You could implement Linux and not incur OS licensing cost.

https://helpcenter.veeam.com/docs/backup/vsphere/virtual_appliance_mode_vsan.html?ver=120

Userlevel 3
Badge

Nothing that could explain why it is happening. The only consistant issue I see is the speed drop after some time, no matter the VM, host or the amount of tasks processed by the jobs.

Userlevel 7
Badge +19

That’s normal. My jobs do the same. Backup Speed isn’t a constant metric throughout your jobs.

Only other option I can suggest Fellipe is to contact Support. Another option I guess is to ping the Product Managers in the Forums. Maybe they can offer other reasons I’m not able to think of why you see a drop in Read speed.

Userlevel 7
Badge +21

The Proxy is virtual - is the VSAN cluster physical servers?  I am asking because you will have backup server physical > proxy virtual > VSAN physical - so there will be things there that will cause slowness.

Also check this page about VSAN backup with Veeam and requirements for Proxy server - Virtual Appliance Mode for VMs on VSAN - User Guide for VMware vSphere (veeam.com)

Userlevel 3
Badge

The Proxy is virtual - is the VSAN cluster physical servers?  I am asking because you will have backup server physical > proxy virtual > VSAN physical - so there will be things there that will cause slowness.

Also check this page about VSAN backup with Veeam and requirements for Proxy server - Virtual Appliance Mode for VMs on VSAN - User Guide for VMware vSphere (veeam.com)

Yes, the vSan cluster is running on physical servers.

 

 

I got the backup job running faster by having 2 proxies processing the VMs. For what I was able to test last week, that was the best outcome. Thank you guys for the tips and the help!

Userlevel 7
Badge +19

Glad to hear was able to suggest something which helped you a bit. Yeah...more Proxies = better performance 😊

Userlevel 7
Badge +21

The Proxy is virtual - is the VSAN cluster physical servers?  I am asking because you will have backup server physical > proxy virtual > VSAN physical - so there will be things there that will cause slowness.

Also check this page about VSAN backup with Veeam and requirements for Proxy server - Virtual Appliance Mode for VMs on VSAN - User Guide for VMware vSphere (veeam.com)

Yes, the vSan cluster is running on physical servers.

 

 

I got the backup job running faster by having 2 proxies processing the VMs. For what I was able to test last week, that was the best outcome. Thank you guys for the tips and the help!

Yeah more proxies the faster backups will be. They are the heavy lifters for Veeam to do the processing of the disks so when you need performance you scale out with those.  Glad to see you got things working better.  Be sure to mark the post that helped the most as the best answer to help others looking for similar problems.

Userlevel 7
Badge +19

Hi @FellipeCM -

I just wanted to follow up on your old post here to see if you had any further questions. If not, could you mark which comment best helped you as ‘Best Answer’ so others who have a similar question and come across your post may benefit?

Thank you.

Userlevel 7
Badge +8

Try using the physical server as a proxy to test. Disable the virtual one and see the difference in performance.

 

having the proxy and repo on the save box can save an extra transfer of data in some cases.

 

there are 100s of ways you can set up Veeam but read the best practice guide as you do it

Comment