Solved

Need Help Diagnosing Slow Backup Speed..


Userlevel 2

Hey Veeam Community,

I am having a problem backing up my personal workstation to an external hard drive.

I am using the free version of Veeam Agent for Windows.

When I backup, my main (C:) Internal SSD shows:

Local SSD (C:) (476.1 GB) 262.5 GB read at 142 MB/s

However, my other Internal SSD ‘terabyte’ (X:)

Terabyte (X:) (953.9 GB) 3.8 GB read at 4 MB/s.

I benchmarked both of my drives with CrystalDiskMark and they both got good similar results being Sata SSD’s, so the X: drive should not be this slow.

I’ve tried swapping sata ports, changing cables, disable throttling when system is in use, etc. Nothing seems to help.

Can anyone help me diagnose what is causing one drive to backup so much faster?

 

Thanks, Mike

 

icon

Best answer by Ben4jammin 26 December 2021, 19:31

View original

12 comments

Userlevel 7
Badge +6

This is hard to say as there could be many factors.  I have the Veeam Agent installed on my Sager laptop which has three SSD drives in it - two are MSATA SSD and the other is a regular SSD.  I get 100+MB/s on all drives in the laptop.

If you have an antivirus program installed have you ensured to put the Veeam exclusions in there?  Check this - KB1999: How to configure antivirus exclusions to prevent interaction with Veeam Backup & Replication

I know it mentions VBR but some of the directories are the same.

Also what is on the X drive in the way of data or possibly programs installed there?  If you happen to install Veeam Agent on the X drive that is where it will also put the SQL database so this could cause contention with the reads/writes.

It is very hard to speculate but another thing to check would be the logs as well under C:\ProgramData\Veeam\Endpoint.  This should also give some clues.

Userlevel 7
Badge +5

 

Hello @mikep17, your issue could be due to so many reasons as it relates to your personal workstation and some of which are as follows;

- Memory issue: The amount of available memory on your device will indirectly impact the backup speed. That is, if your device is low on Memory (RAM), it may use the pagefile, and this can hamper the performance of your backup. Also ensure there isn't an issue of memory leak as this can impact backup speed. Restart your device to fix this temporarily!

- Network connection: Please check the speed of your internet connection.

- High CPU Usage: Also, check for high cpu utilisation. This could also lead to a bottle neck.

- AntiVirus: Do you have an antivirus solution, this could also impact it. @Chris.Childerhose has also mentioned this point. I am aware that Anti-Virus scan files before backup/ restore. Check this too :) 

When you say external HardDrive, do you mean USB? If yes, what type of USB are you using?

Note: Some external disk drives will experience slow performance. Please, also perform benchmarking using the tool you have used before to determine the expected performance of the external drive both in sequential reads, and in small random reads.

Note: CrystalDiskMark may shorten SSD/USB Memory life: https://crystalmark.info/en/software/crystaldiskmark/

Are you using Windows 10? If yes, please ensure the option to control whether USB drives are configured for “Better performance” is enabled as shown below.

 

 

Userlevel 7
Badge +6

 

Hello @mikep17, your issue could be due to so many reasons as it relates to your personal workstation and some of which are as follows;

- Memory issue: The amount of available memory on your device will indirectly impact the backup speed. That is, if your device is low on Memory (RAM), it may use the pagefile, and this can hamper the performance of your backup. Also ensure there isn't an issue of memory leak as this can impact backup speed. Restart your device to fix this temporarily!

- Network connection: Please check the speed of your internet connection.

- High CPU Usage: Also, check for high cpu utilisation. This could also lead to a bottle neck.

- AntiVirus: Do you have an antivirus solution, this could also impact it. @Chris.Childerhose has also mentioned this point. I am aware that Anti-Virus scan files before backup/ restore. Check this too :) 

When you say external HardDrive, do you mean USB? If yes, what type of USB are you using?

Note: Some external disk drives will experience slow performance. Please, also perform benchmarking using the tool you have used before to determine the expected performance of the external drive both in sequential reads, and in small random reads.

Note: CrystalDiskMark may shorten SSD/USB Memory life: https://crystalmark.info/en/software/crystaldiskmark/

Are you using Windows 10? If yes, please ensure the option to control whether USB drives are configured for “Better performance” is enabled as shown below.

 

 

Some excellent points here as well including the last one about the setting on the USB drive.  That is how I have my systems configured that use the Agent.

Userlevel 7
Badge +5

 

Hello @mikep17, your issue could be due to so many reasons as it relates to your personal workstation and some of which are as follows;

- Memory issue: The amount of available memory on your device will indirectly impact the backup speed. That is, if your device is low on Memory (RAM), it may use the pagefile, and this can hamper the performance of your backup. Also ensure there isn't an issue of memory leak as this can impact backup speed. Restart your device to fix this temporarily!

- Network connection: Please check the speed of your internet connection.

- High CPU Usage: Also, check for high cpu utilisation. This could also lead to a bottle neck.

- AntiVirus: Do you have an antivirus solution, this could also impact it. @Chris.Childerhose has also mentioned this point. I am aware that Anti-Virus scan files before backup/ restore. Check this too :) 

When you say external HardDrive, do you mean USB? If yes, what type of USB are you using?

Note: Some external disk drives will experience slow performance. Please, also perform benchmarking using the tool you have used before to determine the expected performance of the external drive both in sequential reads, and in small random reads.

Note: CrystalDiskMark may shorten SSD/USB Memory life: https://crystalmark.info/en/software/crystaldiskmark/

Are you using Windows 10? If yes, please ensure the option to control whether USB drives are configured for “Better performance” is enabled as shown below.

 

 

Some excellent points here as well including the last one about the setting on the USB drive.  That is how I have my systems configured that use the Agent.

Thank you @Chris.Childerhose 

Userlevel 7
Badge +6

Terabyte (X:) (953.9 GB) 3.8 GB read at 4 MB/s.

Did your backup job finish with those stats? If so, how long did the job run?

Userlevel 2

Hey thanks for the replies.

 

Terabyte (X:) (953.9 GB) 3.8 GB read at 4 MB/s.

Did your backup job finish with those stats? If so, how long did the job run?

I cancelled the backup before it finished. Estimated would take about 40 hours so I stopped to troubleshoot.

 

As for the other points:
Memory - not capped out during backup, not using page/swap file.

Network connection - backup is happening locally from my 2 SSD’s to my External USB 3.0 HDD.

Antivirus - I use avast, and had whitelisted the paths before starting the backup.

Install location of Veeam: C drive. Issue only comes up on the X (1tb) drive.

I just enabled the better performance option and will test again. To be honest, since I am using this for personal backups, I would accept if both my drives got the same speed as the C drive.

 

Thank you for the help.

 

Userlevel 2

Decided to run winsat and compare the X and C drives. I think I found the problem, but don’t know how to go about fixing it.

```

C:\WINDOWS\system32>winsat disk -drive x
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive x -ran -read'
> Run Time 00:00:04.06
> Running: Storage Assessment '-drive x -seq -read'
> Run Time 00:00:43.97
> Running: Storage Assessment '-drive x -seq -write'
> Run Time 00:00:04.13
> Running: Storage Assessment '-drive x -flush -seq'
> Run Time 00:00:00.94
> Running: Storage Assessment '-drive x -flush -ran'
> Run Time 00:00:00.91
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       4.36 MB/s          5.0
> Disk  Sequential 64.0 Read                   191.49 MB/s          7.3
> Disk  Sequential 64.0 Write                  358.10 MB/s          7.9
> Average Read Time with Sequential Writes     0.325 ms          8.4
> Latency: 95th Percentile                     0.684 ms          8.5
> Latency: Maximum                             11.273 ms          7.9
> Average Read Time with Random Writes         0.332 ms          8.8
> Total Run Time 00:00:54.13

C:\WINDOWS\system32>winsat disk -drive c
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive c -ran -read'
> Run Time 00:00:00.34
> Running: Storage Assessment '-drive c -seq -read'
> Run Time 00:00:04.81
> Running: Storage Assessment '-drive c -seq -write'
> Run Time 00:00:03.45
> Running: Storage Assessment '-drive c -flush -seq'
> Run Time 00:00:00.66
> Running: Storage Assessment '-drive c -flush -ran'
> Run Time 00:00:00.64
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       323.02 MB/s          8.1
> Disk  Sequential 64.0 Read                   463.74 MB/s          8.1
> Disk  Sequential 64.0 Write                  438.41 MB/s          8.0
> Average Read Time with Sequential Writes     0.195 ms          8.6
> Latency: 95th Percentile                     0.520 ms          8.7
> Latency: Maximum                             10.141 ms          7.9
> Average Read Time with Random Writes         0.174 ms          8.9
> Total Run Time 00:00:10.02

```

I have a similar issue and here is what I found:

In looking at past backups, had an SSD drive that would backup at about 40 MB/sec.  Over time it dropped to 4 MB.

I installed Western Digital dashboard to check firmware, SMART, TRIM etc

What got it back up to 20 MB/sec was deleting files until it was less than 50% used.  I didn’t delete all the files at once and could see steady gains.  Went from 4 MB/sec to 9, then 20 as I kept deleting files and restarting backup.

I don’t know what else to do to help performance.  I just updated Veeam to latest free version.  Firmware is latest version.  I only had games on that HD so not an issue creating some space.  

I’m guessing that I’ll just need to replace the drive with a better one.  It is an older SSD.

Userlevel 2

I have a similar issue and here is what I found:

In looking at past backups, had an SSD drive that would backup at about 40 MB/sec.  Over time it dropped to 4 MB.

I installed Western Digital dashboard to check firmware, SMART, TRIM etc

What got it back up to 20 MB/sec was deleting files until it was less than 50% used.  I didn’t delete all the files at once and could see steady gains.  Went from 4 MB/sec to 9, then 20 as I kept deleting files and restarting backup.

I don’t know what else to do to help performance.  I just updated Veeam to latest free version.  Firmware is latest version.  I only had games on that HD so not an issue creating some space.  

I’m guessing that I’ll just need to replace the drive with a better one.  It is an older SSD.

Yes, I had to return my SSD to SanDisk. All the smart trim etc readings were normal. Only winsat was able to find the problem. The new one I received does not have the problem even after copying all old files to the new drive. 

Userlevel 7
Badge +5

I have a similar issue and here is what I found:

In looking at past backups, had an SSD drive that would backup at about 40 MB/sec.  Over time it dropped to 4 MB.

I installed Western Digital dashboard to check firmware, SMART, TRIM etc

What got it back up to 20 MB/sec was deleting files until it was less than 50% used.  I didn’t delete all the files at once and could see steady gains.  Went from 4 MB/sec to 9, then 20 as I kept deleting files and restarting backup.

I don’t know what else to do to help performance.  I just updated Veeam to latest free version.  Firmware is latest version.  I only had games on that HD so not an issue creating some space.  

I’m guessing that I’ll just need to replace the drive with a better one.  It is an older SSD.

Yes, I had to return my SSD to SanDisk. All the smart trim etc readings were normal. Only winsat was able to find the problem. The new one I received does not have the problem even after copying all old files to the new drive. 

Thank you for letting us know!

Just to follow up, so the drive in question was a Western Digital.  I have 2 of the same model in my PC.  The other works fine.  I don’t remember if there is a big difference in their age.

So I don’t know what happened to the drive, but something obviously did.  I guess they may be older drives because my OS is on an SSD (came with the system I purchased about a year ago) and in Veeam it gets backed up at 250 MB/s while the WD and the new Samsung I bought clock in at about 175 MB/s.  Meanwhile an old spindle drive is getting backed up at 150 MB/s.

 

I guess I need to educate myself on SSD performance because I don’t know why that OS drive is so much faster than the other 2.

Just today i’ve backed up my PC to external USB3 drive.
I made 2 retrys as i experienced just the same speed problem: from my internal M.2 SSD … Pro drive (2 partitions, one is pretty large) to USB3 transfer rate was only 60-40MB/s, although my PC has no any USB2 port and i know that my previous backup made a couple of month ago worked like a charm. The root of the problem in my case was POOR EXTENDING Male-Female CABLE from my USB3 port (i use for memory sticks, SD cards, ets) connected to USB3 drive’s cable. Just as i removed it and connected USB drive directly to USB port by its own normal USB3 cable the problem has gone away and trasfer speed changed to expected 200-300 MB/s.

Comment