Skip to main content
  • EN

Hi all,

I’m currently running Ubuntu 20.04.6 in a Server 2012 R2 Hyper-V VM. Veeam B&R CE 12 is installed on the VM host, which is scheduled to backup the VM every so often. I’ve been struggling for quite awhile now trying to troubleshoot some instability in the VM. It appears that applications start crashing following a Veeam backup opeartion on the VM. Multiple different applications will seg fault, including systemd. In doing this troubleshooting, while watching journalctl, I see several I/O related errors scroll by during a Veeam backup:


Oct 16 16:12:43 ubuntush multipathda600]: sdc: path already removed
Oct 16 16:12:42 ubuntush multipathda600]: uevent trigger error
Oct 16 16:12:20 ubuntush multipathp16675]: sdc: can't store path info
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Attached SCSI disk
Oct 16 16:12:19 ubuntush kernel: sdc: detected capacity change from 167772160 to 0
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] 4096-byte physical blocks
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] 0 512-byte logical blocks: (0 B/0 B)
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Sense not available.
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Read Capacity(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#256 cmd 0x25 status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#315 cmd 0x25 status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#313 cmd 0x25 status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Sense not available.
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Read Capacity(16) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#312 cmd 0x9e status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#310 cmd 0x9e status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#308 cmd 0x9e status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Test Unit Ready failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Oct 16 16:12:19 ubuntush kernel: Buffer I/O error on dev sdc, logical block 4, async page read
Oct 16 16:12:19 ubuntush kernel: blk_update_request: I/O error, dev sdc, sector 32 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] tag#297 CDB: Read(10) 28 00 00 00 00 20 00 00 08 00
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] tag#297 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#297 cmd 0x28 status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Mode Sense: 0f 00 00 00
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Write Protect is off
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] 4096-byte physical blocks
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] 167772160 512-byte logical blocks: (85.9 GB/80.0 GiB)
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: Attached scsi generic sg3 type 0
Oct 16 16:12:19 ubuntush kernel: scsi 0:0:1:0: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 4
Oct 16 16:12:17 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=THAW: succeeded
Oct 16 16:12:17 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#300 cmd 0x28 status: scsi 0x2 srb 0x82 hv 0xc0000001
Oct 16 16:12:17 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#299 cmd 0x2a status: scsi 0x2 srb 0x82 hv 0xc0000001
Oct 16 16:12:16 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=FREEZE: succeeded
Oct 16 16:12:14 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=CHECK HOT BACKUP
Oct 16 16:12:10 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=CHECK HOT BACKUP
Oct 16 16:12:10 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=CHECK HOT BACKUP

 

Not all of those lines are errors, but that is the block containing the majority of them. Veeam does not seem to pick up that anything is going wrong during this process, and it completes successfully. Does anything there look concerning and possibly related to these crashes that I’m seeing?

My guess based on those lines is something within the Hyper-V stack and since it is 2012 that could be part of the problem as well.  You might want to think about upgrading and since Veeam is completing at least you have backups.

I would also suggest testing the backups to ensure the server can be restored as a safety precaution.

Hi @selfhostr -

In looking through the HV Guide, it appears there may be a hardware limitation for HV VMs. So make sure your VM virtual h/w version meets the requirement:
https://helpcenter.veeam.com/docs/backup/hyperv/platform_support.html?ver=120#hyper-v-vms

Other than that...I see everything else you shared should be supported:
https://helpcenter.veeam.com/docs/backup/hyperv/platform_support.html?ver=120

We can’t really help much here regarding Linux errors you’re getting..only Veeam. As Chris shared, one option you could try is upgrade your HV environmnet to current. You could see if anyone on the Veeam Forums has experienced similar errors in a Ub Lnx VM, but they typically want a case ID over there. I don’t have any other suggestions.

Thanks for your responses everyone. I was afraid the bottom line would be that an upgrade is in order. I’m in the process of completely rebuilding the system from the ground up so that may eliminate any issues in the stack.

@coolsport00 It looks like the Hyper-V requirements all seem to be in line for what I have.

As far as a short-term fix, is it possible to completely disable the live backup functionality so that Veeam freezes the VM during the backup? I feel like these live backups may be adding some additional complication that I may want to avoid here.

@selfhostr - I’m not sure what you mean by “live” backups? Veeam uses Integration Services (IS) to quiesce the VM’s OS to keep apps on the VM at a app-consistent state. I don’t use HV, so not sure if IS works with Linux. If it does, then in theory, you shouldn’t be having app crashes. But I don’t have HV and thus not had to troubleshoot the issue you’re experiencing. If you disable the option to use IS in the Backup Job, then the backup is what’s called ‘crash-consistent’. You could try that if you haven’t already. Worth a shot. Otherwise, opening a case with Tech Support would be your best bet….which is allowed even for the CE version.

@coolsport00 I think you’ve got the gist of it! I’ve read a couple of different articles such as this one and this to better understand quiescence. There was a Veeam blog post I read several years ago which guided me towards writing some pre- and post- freeze scripts, and did a pretty good job of explaining the different types of quiescence that Hyper-V/Veeam can do, and I think I ended up going for the middle road, which was that it’s supposed to pause an app or two (MySQL I think it was) during the backup procedure, then resume it. I think the Integration Services are configured as well as can be (all components show in the MS Event Log as connected when the machine boots up, save for one complaining about being out of date which MS says is “fine”.) I can also see different HV components talking in the VM’s log during backup so it looks like there’s communication going on. I’m just starting to feel more and more that something isn’t working as expected and leading to these issues during the backup.

At this point, I’ve disabled the Backup Integration Service in Hyper-V which I assumed meant that the VM would be paused during the backup procedure, although I was still able to work in a remote terminal of the VM during the backup (and see some logs about VSS activating) so it seems what I was expecting didn’t really happen. I’ll try activating the crash-consistent option and see how that goes, thanks!

Edit: I’ve just realized that I didn’t have the crash-consistent option checked in the Hyper-V settings on the backup job, so I’m not quite sure what it was doing this last run. Didn’t seem to pause the VM either way.

@coolsport00 I think you’ve got the gist of it! I’ve read a couple of different articles such as this one and this to better understand quiescence. There was a Veeam blog post I read several years ago which guided me towards writing some pre- and post- freeze scripts, and did a pretty good job of explaining the different types of quiescence that Hyper-V/Veeam can do, and I think I ended up going for the middle road, which was that it’s supposed to pause an app or two (MySQL I think it was) during the backup procedure, then resume it. I think the Integration Services are configured as well as can be (all components show in the MS Event Log as connected when the machine boots up, save for one complaining about being out of date which MS says is “fine”.) I can also see different HV components talking in the VM’s log during backup so it looks like there’s communication going on. I’m just starting to feel more and more that something isn’t working as expected and leading to these issues during the backup.

At this point, I’ve disabled the Backup Integration Service in Hyper-V which I assumed meant that the VM would be paused during the backup procedure, although I was still able to work in a remote terminal of the VM during the backup (and see some logs about VSS activating) so it seems what I was expecting didn’t really happen. I’ll try activating the crash-consistent option and see how that goes, thanks!

Edit: I’ve just realized that I didn’t have the crash-consistent option checked in the Hyper-V settings on the backup job, so I’m not quite sure what it was doing this last run. Didn’t seem to pause the VM either way.

The VM will only pause for a few seconds while the snapshot is taken and removed typically.  So that is as expected regardless of the crash-consistent being on/off.

Hi @selfhostr - no...the IS just pauses the I/O to the VM apps, not ‘freeze’ the VM itself. So, it works a bit different than what you thought (if I read your comment properly). So, during backup, you can use it like it wasn’t being backed up...ssh to it, etc. 

Keep us posted on things.

Comment


Terms & Conditions