What settings do you have enabled in the Agent - VSS, etc.?  Also which license version do you have Free, Workstation, Server?  You should also ensure the CBT driver is installed on your system.

I have a laptop with the agent and 2 NVME drives but never notice this when backups run.

These are the only settings on the agent, I have the free windows version.

I don’t have CBT either, is that available for the free version?

At very first glance here I’d suggest investigating your drive health. My boot drive had a SMART health of 60% and saw excessively high latency on light tasks, latency of seconds like you said.

I checked drive health, but nothing stood out to me, unless anyone else sees something that shows its the drive?

These are the only settings on the agent, I have the free windows version.

 

I don’t have CBT either, is that available for the free version?

 

No you actually need a licensed version for that - Workstation or Server.  I would check what Michael mentioned as well.

I checked drive health, but nothing stood out to me, unless anyone else sees something that shows its the drive?

 

The temp seems awfully high on that drive at 50C.  Is there sufficient cooling as I am wondering if the drive is getting how when backups kick off causing the latency?  Just a guess mind you.

I’d try a benchmark to see if other things cause this latency when doing read/writes. Based on the health metrics it does seem unlikely to be a failing drive

Drive performance seems fine too.. very strange.

And latency was fine during the testing? When you do the backups, what about your other system metrics? CPU for example? Wondering if the latency is instead by wasted time trying to write to your destination and that is then actually the issue instead?

CPU and RAM have no issues while running benchmarks.

While rerunning the test, latency has no issues for read: 

Latency spikes for write though:

That’s fair enough you’ll naturally get higher latency when you generate IO, was that latency from a benchmark or from a backup task that time? Can you rerun a backup and show the IO for the drive and the backup target?

The previous two were from a benchmark, I just reran a the backup and I had spikes up to 20k ms, unfortunately the latency is so bad I can’t even catch it so I had to take a picture with my phone.

This is the destination drive:

I couldn’t make the time narrower unfortunately to show better scale.

Hi @sarmen94,

I have a Rocket 4 Plus Gaming as my OS drive and have noticed the same behavior when adding my desktop to my VBR and attempting to take a backup as part of my lab setup.

It seems the only way I can replicate this behavior outside of Veeam is in Sabrent Rocket Control Panel. If I initiate a speed test with any combination of settings except file size set to “Read Only” I get about 20 MB/s read speed and 1000+ ms latency. If I change it back to the default of 1 GB, I get about 7 GB/s read speed with less than 10 ms latency.

I just opened a case with Sabrent support to see what they say, but I suspect whatever calls this control panel test and Veeam make hit a part of the IO system that is not usually accessed by most applications and system components and may have some sort of firmware bug.

I have read about failures on those drives from a few posts online.

Perhaps verify everything with a chkdsk or block level scan. 

I have also seen issues where those drives have issues when they get too hot. Monitor the cooling of that drive while running a backup.

If it’s only during writes, does it happen right at the beginning of the backup, or test, or after a while? 

A few other tests. Get something like IOMeter, and run some performance benchmarks, start with a single stream and build up. It could be too many parallel operations overloading it. Continue to watch heat as well. 

@JonahMay Those are great read speeds BTW

Yeah, great speeds until I try to run something at the volume level instead of the file level apparently. It is nifty for loading STLs and gaming at least.

In my case chkdsk and sfc both checked out without problems on Sunday. This is also my daily driver and I only seem to see this stun when running that specific test or when VBR tries to have the agent run a full. Latency starts to spike as it does index and the VSS snapshot, then only gets worse as it starts to read the disk.

It definitely isn’t inline malware detection or AAIP as I tried disabling both of those before I was able to replicate the issue with their control panel.

Based on what I’ve seen so far, I suspect it is either a Windows or Sabrent bug when reading at the volume level. From those tests I ran that seems to be the only difference between them that could explain it. I’m probably going to create a test Veeam job that reads at the file level instead of the volume and see if that makes a difference here in a few minutes.

ETA: Made a job and told it to only back up OS files and the user folders like Documents and Downloads. Latency is now below 1000ms while backups are running but read speeds are down to about 10 MB/s so Veeam must be making close enough calls in the two backup modes to not get around it like other tools and applications do. Also, Sabrent support asked for a CrystalDiskMark screenshot so I may as well provide it here too:

Once again, latency stayed below 10ms for all read tests. Meanwhile I’ve seen up to 25000 when running a Veeam backup.