Hi all,
I’m currently running Ubuntu 20.04.6 in a Server 2012 R2 Hyper-V VM. Veeam B&R CE 12 is installed on the VM host, which is scheduled to backup the VM every so often. I’ve been struggling for quite awhile now trying to troubleshoot some instability in the VM. It appears that applications start crashing following a Veeam backup opeartion on the VM. Multiple different applications will seg fault, including systemd. In doing this troubleshooting, while watching journalctl, I see several I/O related errors scroll by during a Veeam backup:
Oct 16 16:12:43 ubuntush multipathda600]: sdc: path already removed
Oct 16 16:12:42 ubuntush multipathda600]: uevent trigger error
Oct 16 16:12:20 ubuntush multipathp16675]: sdc: can't store path info
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Attached SCSI disk
Oct 16 16:12:19 ubuntush kernel: sdc: detected capacity change from 167772160 to 0
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] 4096-byte physical blocks
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] 0 512-byte logical blocks: (0 B/0 B)
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Sense not available.
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Read Capacity(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#256 cmd 0x25 status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#315 cmd 0x25 status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#313 cmd 0x25 status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Sense not available.
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Read Capacity(16) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#312 cmd 0x9e status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#310 cmd 0x9e status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#308 cmd 0x9e status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Test Unit Ready failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Oct 16 16:12:19 ubuntush kernel: Buffer I/O error on dev sdc, logical block 4, async page read
Oct 16 16:12:19 ubuntush kernel: blk_update_request: I/O error, dev sdc, sector 32 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] tag#297 CDB: Read(10) 28 00 00 00 00 20 00 00 08 00
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] tag#297 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
Oct 16 16:12:19 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#297 cmd 0x28 status: scsi 0x0 srb 0x20 hv 0xc0000001
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Mode Sense: 0f 00 00 00
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] Write Protect is off
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] 4096-byte physical blocks
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: :sdc] 167772160 512-byte logical blocks: (85.9 GB/80.0 GiB)
Oct 16 16:12:19 ubuntush kernel: sd 0:0:1:0: Attached scsi generic sg3 type 0
Oct 16 16:12:19 ubuntush kernel: scsi 0:0:1:0: Direct-Access Msft Virtual Disk 1.0 PQ: 0 ANSI: 4
Oct 16 16:12:17 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=THAW: succeeded
Oct 16 16:12:17 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#300 cmd 0x28 status: scsi 0x2 srb 0x82 hv 0xc0000001
Oct 16 16:12:17 ubuntush kernel: hv_storvsc d309a2a0-91ff-4664-ae4d-bb42094a0206: tag#299 cmd 0x2a status: scsi 0x2 srb 0x82 hv 0xc0000001
Oct 16 16:12:16 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=FREEZE: succeeded
Oct 16 16:12:14 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=CHECK HOT BACKUP
Oct 16 16:12:10 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=CHECK HOT BACKUP
Oct 16 16:12:10 ubuntush hv_vss_daemonu694]: Hyper-V VSS: VSS: op=CHECK HOT BACKUP
Not all of those lines are errors, but that is the block containing the majority of them. Veeam does not seem to pick up that anything is going wrong during this process, and it completes successfully. Does anything there look concerning and possibly related to these crashes that I’m seeing?