Solved

Windows 10 failed to backup to NAS


Userlevel 2

Hi,

 

I tried my luck with the subreddit but I recon this is a better place for support.

 

I was recently pointed to Veeam products for backing up my local machine onto my NAS server.

The machine runs Windows 10 with no VMs, ryzen 5900x, a 2.5gb nic, and it includes 1tb ssd and 1tb nvme (OS) for backup. The pc is connected via direct ethernet to the nas (10.0.10.1)

I've gone through the job configuration OK; the backup job was created successfully.

 

 

However, when the job runs and begins to backup, it fails with an undisclosed error "28/02/2023 20:05:14 :: Error: Error in the application "

 

 

 

Upon investigation of the logs, 10mins into the backup:

 

[01.03.2023 02:10:15] < 45676> cli| ERR |Failed to dispatch pending requests

[01.03.2023 02:10:15] < 45676> cli| >> |Data error (cyclic redundancy check).

[01.03.2023 02:10:15] < 45676> cli| >> |Asynchronous read operation failed

[01.03.2023 02:10:15] < 45676> cli| >> |--tr:Error code: 0x00000017

[01.03.2023 02:10:15] < 45676> cli| >> |--tr:Failed to complete asynchronous read request (offset: 94247059456, size: 4194304)

[01.03.2023 02:10:15] < 45676> cli| >> |--tr:Failed to complete read request 0 (offset: 94247059456, size: 4194304)

[01.03.2023 02:10:15] < 45676> cli| >> |An exception was thrown from thread [45676].

 

And

[01.03.2023 02:10:36] <43> Info [AP] (0458) output: --asyncNtf:--vdisk_corrupted:\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy21

 

 

The logs also pick up what appear are the correct volumes to backup:


 

 

A few thoughts/questions:

  • Is HarddiskVolumeShadowCopy21 my NAS presumably ?

  • It should take less than 10mins to copy 90gb over 2.5gbe, let alone only 9% of the job ? I wonder if its using the 1gb connection instead, but I've configured it to have a backup point of \\10.0.10.1\ which is the 2.5g controller. 

  • The log reads:[01.03.2023 02:09:39] <11> Info Job progress: '9%', '92,657,418,240' of '2,000,386,633,728' bytes, '81,491,192,939' of '868,336,782,336' used bytes, object '0' of '1', totals calculated: No
    How should i understand the 92gb/2tb and 81gb/868gb ?

 

I can probably share the entire logs if it doesn't have sensitive info

 

One of the users recommended I try diskshadow separately to isolate the issue potentially (https://www.veeam.com/kb1980) but I can’t see how to do it on Win10.

 

Let me know if I can provide anything more from my side.

 

Thanks,

Nick

icon

Best answer by nickdems 22 March 2023, 13:31

View original

21 comments

Userlevel 7
Badge +20

First question is how are you backing up the Win10 machine?  Agent? VBR?  That will better help determine direction.

Userlevel 2

Hi Chris, I am using VBR.

Thanks

 

Userlevel 7
Badge +20

Ok - now versions of Veeam - 11a?  Latest patch?  Also, Agent version?  It seems like possible issue with the NVME drive which is the OS.  Are all the VSS Writers good?  - vssadmin list writers

Also, I know this is dumb, but have you tried the MS fix for most things - reboot the Win10 machine? 😋

Userlevel 2

Ok - now versions of Veeam - 11a?  Latest patch?  Also, Agent version?  It seems like possible issue with the NVME drive which is the OS.  Are all the VSS Writers good?  - vssadmin list writers

Also, I know this is dumb, but have you tried the MS fix for most things - reboot the Win10 machine? 😋

 

I’m using VBR 12.0.0.1420, Agent 6.0.0.960

vssadmin shows all writers with last error “No Error”, although a few of them are at a “waiting for completion” state, rest are “Stable”

I actually hadn’t done the tried and tested turn it off and on again heh.

Alas even after the restart, I kicked off another backup job with the same result.

 

Userlevel 7
Badge +20

VolumeShadowCopy is a VSS snapshot taken, if you’re unsure what that is, here’s a Microsoft link related to how it works: https://learn.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service

 

I’d suspect an unsupported OS (Windows 10 Cumulative Updates all count independently as OS’s due to Microsoft’s love of breaking things between releases). Alternatively I’d look at any antivirus you might have interfering with Veeam’s read/write. As ransomware looks for VSS and tries to disable it, could be an oversensitive AV product especially if you don’t have the AV exclusions in.

What version of Veeam Agent are you using? As you’re managing this from VBR, what VBR aversion are you on?

RE the NIC being used, if the devices are directly wired to each other, on the same subnet and you’re targeting the IP that is only accessible on that link, then it has to be using that link.

 

You’re not seeing a bottleneck as it didn’t run properly but although the link could do 2.5GbE, doesn’t mean your NAS disks can accept data at that speed. What NAS are you using?

 

EDIT: Just seen Chris had asked the same and you’d supplied versions!

Userlevel 7
Badge +20

Ok - now versions of Veeam - 11a?  Latest patch?  Also, Agent version?  It seems like possible issue with the NVME drive which is the OS.  Are all the VSS Writers good?  - vssadmin list writers

Also, I know this is dumb, but have you tried the MS fix for most things - reboot the Win10 machine? 😋

 

I’m using VBR 12.0.0.1420, Agent 6.0.0.960

vssadmin shows all writers with last error “No Error”, although a few of them are at a “waiting for completion” state, rest are “Stable”

I actually hadn’t done the tried and tested turn it off and on again heh.

Alas even after the restart, I kicked off another backup job with the same result.

 

I am out of ideas other than opening a support case.  Maybe you can do a search on the Forums as well to see if that helps - https://forums.veeam.com/

Best of luck.  Hopefully someone else here chimes in as well.

Userlevel 2

VolumeShadowCopy is a VSS snapshot taken, if you’re unsure what that is, here’s a Microsoft link related to how it works: https://learn.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service

 

I’d suspect an unsupported OS (Windows 10 Cumulative Updates all count independently as OS’s due to Microsoft’s love of breaking things between releases). Alternatively I’d look at any antivirus you might have interfering with Veeam’s read/write. As ransomware looks for VSS and tries to disable it, could be an oversensitive AV product especially if you don’t have the AV exclusions in.

What version of Veeam Agent are you using? As you’re managing this from VBR, what VBR aversion are you on?

RE the NIC being used, if the devices are directly wired to each other, on the same subnet and you’re targeting the IP that is only accessible on that link, then it has to be using that link.

 

You’re not seeing a bottleneck as it didn’t run properly but although the link could do 2.5GbE, doesn’t mean your NAS disks can accept data at that speed. What NAS are you using?

 

EDIT: Just seen Chris had asked the same and you’d supplied versions!

 

I don’t run anti virus or firewall, other than windows included.

The NIC wasn’t anything in the end, i monitored the job via the resource monitor in windows, its using the right link to transfer data.

The speed issue lies with VSS reading on the C:\ drive as its speed is only around 120mb/s, although the nvme should handle way greater speeds, and cpu isnt a bottleneck either (only saw 20% use).

 

Userlevel 2

Ok - now versions of Veeam - 11a?  Latest patch?  Also, Agent version?  It seems like possible issue with the NVME drive which is the OS.  Are all the VSS Writers good?  - vssadmin list writers

Also, I know this is dumb, but have you tried the MS fix for most things - reboot the Win10 machine? 😋

 

I’m using VBR 12.0.0.1420, Agent 6.0.0.960

vssadmin shows all writers with last error “No Error”, although a few of them are at a “waiting for completion” state, rest are “Stable”

I actually hadn’t done the tried and tested turn it off and on again heh.

Alas even after the restart, I kicked off another backup job with the same result.

 

I am out of ideas other than opening a support case.  Maybe you can do a search on the Forums as well to see if that helps - https://forums.veeam.com/

Best of luck.  Hopefully someone else here chimes in as well.

 

Thanks Chris, I’ll look at the forum and if not i’ll open a case.

Userlevel 7
Badge +20

How much space is consumed on the NVMe? And do you have anything else running that could be using VSS? If disk space is low it could be the snapshot is getting removed, likewise if something else is trying to manipulate it. I’d try to browse the VSS snapshot during the backup when it starts to stall. I’ve also seen another thread of someone seeing pipeline stalled error, just in case it’s an application error, can you uninstall the v6 agent and install the v5 agent? VBR v12 can manage v5 and v6 agents

Userlevel 2

I’d try to browse the VSS snapshot during the backup when it starts to stall.

 

Hi Michael,

The nvme has 300gb free so shouldn’t be a problem.

Nothing else is running at the same time, definitely nothing vss related. How can i browse the snapshot during backup ?

Are there are publicly available v5 agent binaries ? When i go through the Veeam (https://www.veeam.com/download-version.html?ad=downloads&tab=previous) it asks for a business account for previous versions.

Thanks

Userlevel 7
Badge +20

https://www.veeam.com/download-version.html?tab=previous Is the right link, you just need a Veeam account. I have no licenses on my personal account and I can still get the binaries. Just register for a free account. 🙂 thanks for confirming other details. Definitely test v5. If you find that it’s fine in v5 but not in v6 then the R&D forums will likely accept a case raised even with community support as it’s a regression/bug

Userlevel 2

https://www.veeam.com/download-version.html?tab=previous Is the right link, you just need a Veeam account. I have no licenses on my personal account and I can still get the binaries. Just register for a free account. 🙂 thanks for confirming other details. Definitely test v5. If you find that it’s fine in v5 but not in v6 then the R&D forums will likely accept a case raised even with community support as it’s a regression/bug

 

i’m afraid it won’t let me with my free account, and i don’t have a business one i can link either.

Userlevel 7
Badge +12

Do you get an error when trying to download the agent? It shouldn't make a difference whether you have a license or not.

Userlevel 2

Do you get an error when trying to download the agent? It shouldn't make a difference whether you have a license or not.

Not an error, but it went straight to a page that prompts for adding a business email to the account (ie not my gmail) and won’t continue without verifying the address.

Anyway i’ve added one and downloading agent v5, will update thread with results

Thanks

Userlevel 2

Ok so v5 agent installed and running backup with it also fails around the same mark 8-9%, but no gives a BSOD instead of dying in the background ?

Might be something broken/corrupt with my C:\ or windows ?

Userlevel 7
Badge +20

Ok so v5 agent installed and running backup with it also fails around the same mark 8-9%, but no gives a BSOD instead of dying in the background ?

Might be something broken/corrupt with my C:\ or windows ?

That is possible or very slight chance the actual drive maybe?  I know it is NVME, but you never know.

Would suggest running system file checker to ensure no problems.

Userlevel 2

Ok so v5 agent installed and running backup with it also fails around the same mark 8-9%, but no gives a BSOD instead of dying in the background ?

Might be something broken/corrupt with my C:\ or windows ?

That is possible or very slight chance the actual drive maybe?  I know it is NVME, but you never know.

Would suggest running system file checker to ensure no problems.

 

Yep that was my first thought also, but apparently all good.

Any other checks i can run ? I dont fancy doing a reinstall any time soon :)

Userlevel 7
Badge +20

Ok so v5 agent installed and running backup with it also fails around the same mark 8-9%, but no gives a BSOD instead of dying in the background ?

Might be something broken/corrupt with my C:\ or windows ?

That is possible or very slight chance the actual drive maybe?  I know it is NVME, but you never know.

Would suggest running system file checker to ensure no problems.

 

Yep that was my first thought also, but apparently all good.

Any other checks i can run ? I dont fancy doing a reinstall any time soon :)

I would just run the usual SFC and DISM commands for checking things in Windows.  The only one I recall off hand is - SFC /scannow.  The DISM ones just google them.  LOL

Also, would ensure Win10 is fully patched too.

Userlevel 7
Badge +20

What error do you get with the BSOD? Might help narrow the search.

Userlevel 7
Badge +20

Any updates on this one @nickdems?

Userlevel 2

Hi @MicoolPaul, I forgot to update the thread, thanks for the prompt.

 

I installed the older agent and still had the same issues, the BSOD had no error number presented, and also didn't stay on the screen long enough for me to write anything down. If i remember correctly it was something relating to corrupt memory or out of bounds ? Sorry i can’t recall it whole.

 

I was concerned there was something corrupt on the OS drive, so ran a few windows disk tests but nothing conclusive.

In the end i started manually copying files across to the NAS as a dummy backup, and the copy failed! Upon further investigation, there were some corrupt files that kept failing the copy process. Once these files were deleted the veeam backup worked fine.

It’s just unfortunate that there wasn’t an easier way to figure out what the corrupt files where; in retrospect i should have googled some corrupt scanner app or something but arent aware of any.

 

All good in the end, it now runs daily.
I think it did a complete backup first and subsequently its doing incremental changes backup ?

Comment