• EN
Question

Backup hangs at 2%, says corrupt volume shadow copy?

  • 13 February 2024
  • 9 comments
  • 215 views
L
Userlevel 2
  • LTS
  • New Here
  • 2 comments

Running Veeam agent for Windows free on an intel nuc, image level backup to external USB SSD.

Over a week ago we were getting the following in the backup failure reports;
 

One or more bad blocks were detected and skipped and skipped in the source disk
Error: Cannot process pending I/O request (offset: 28058845184, size: 65536): async reader terminated Corrupted block processing thread failed unexpectedly Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: Cannot process pending I/O request (offset: 28058845184, size: 65536): async reader terminated Unable to retrieve next block transmission command. Number of already processed blocks: [9390]. Failed to download disk '50f5cc69-4da2-47ad-a384-fe605dfacfaf'

Ran all sorts of tools, benchmarks, SMART checks, chkdsks, etc and found no issues with C: (OS) or D: (eSSD). Noticed Veeam was due for an update, installed the new version, rebooted, started getting backups hanging with this in the logs;
 

[AP] (2f76) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[AP] (b90b) output: --asyncNtf:--busy: Target backup repository is overloaded.

After a reboot, backup jobs now hang at 2%, roughly 14.5-14.9gb processed, and will sit there indefinitely.  In the log file, once the backup gets stuck at 2%, I see this;
 

[AP] (5c0d) output: --asyncNtf:--vdisk_corrupted:\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5

I have cleared out all the shadow copies on this device, thinking one was corrupted. Re-ran the backup, same issue, but now it's citing HarddiskVolumeShadowCopy1. If I just let the backup job sit, it keeps producing these logs;
 

[AP] Waiting for completion. Id: [0x12538bc], Time: 00:30:00.0122884

At this point, the only thing I haven't tried is doing a clean uninstall of Veeam and starting over but based on the errors I'm seeing in the logs I suspect that won't fix the issue. Not sure what to do at this point, other than switch to another local backup solution because I can't seem to prove that there is anything wrong with the machine or either of its disks.

9 comments

wesmrt
Userlevel 7
Badge +4

If it's a corruption in the backup chain you could try run an Active Full. This would create a new backup chain. 

But if the problem is something in the external disk, it's better try a different target for the backup to isolate the problem. 

Wesley Martins @ Specialist Enginner at Veeam Technical Support | My blog: https://itproland.com.br (PT-BR)
L
Userlevel 2

I wiped all the backups so it would start a new chain, and according to the logs it seemingly sees that I did that and acts accordingly, but then it always leads to that same vdisk_corrupted/HarddiskVolumeShadowCopy5 error in the log and the backup stops processing at that point.  That seems to indicate that the shadow copy created by the backup job is getting corrupted, and when the backup finally starts, it fails because of that.  I can try an active full, just waiting for the current stuck backup to finish cancelling which can take a while.

Chris.Childerhose
Userlevel 7
Badge +20

Definitely try the full backup, but if that fails the same way you may be forced to troubleshoot the disks within the machine as something is causing the snapshots to corrupt based on your error messages.

It would be interesting to see if you are using the latest agent and sending direct to Object Storage like Wasabi or something as a test works.  If that fails, then there is definitely something going on in the OS.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
L
Userlevel 2

Tried the active full backup, same issue, same logs.

Chkdsk can’t find anything wrong with the OS drive or the external SSD.  CrystalDiskInfo also reports 100% health on both drives.  Not seeing anything out of the ordinary in event viewer either.  By all appearances, this Intel Nuc and its two SSDs are healthy.

Not sure if the system volume where shadow copies are stored can be corrupted, but not be caught by chkdsk.

Chris.Childerhose
Userlevel 7
Badge +20

Unsure at this point and would suggest opening a Support ticket with Veeam to get them to help troubleshooting as this is beyond the scope of the community now.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
dips
Userlevel 7
Badge +7

You can check the status of the writers by running 

vssadmin list writers

That should give an indication of the state of the writers. Check if any have errors logged for them.

Dipen N. K. | IT Security Specialist | Veeam Legend 2022 - 2023 | www.dipen.co.uk
dloseke
Userlevel 7
Badge +6

I have no idea if maybe the shadow copies can be disabled and re-enabled in order to reset it or something along those lines?

Derek M. Loseke, Senior Systems Engineer | Veeam Legend 2022-2023 | VMSP/VMTSP | VCP6-DCV | VSP/VTSP | CCNA | https://technotesanddadjokes.com | @dloseke
dips
Userlevel 7
Badge +7

I have no idea if maybe the shadow copies can be disabled and re-enabled in order to reset it or something along those lines?

Worth a try together with deleting existing Shadow Copies. 

Dipen N. K. | IT Security Specialist | Veeam Legend 2022 - 2023 | www.dipen.co.uk
E
Userlevel 4
Badge +2

Is there another backup solution or other application that is using VSS shadowcopy.
I saw this issue once that a other solution deleted the vss snapshot during Veeam backup and got this issue.

Also a question, is the storage a RAID array?

Comment


Terms & Conditions