Skip to main content
  • EN

Hi everyone,

I encountered a strange behavior while migrating a customer this week and would like to share my experience and possibly get answers about the root cause.

 

Scenario: Veeam Instant Recovery from Hyper-V to VMware

 

Source: Hyper-V host (WS2019 Datacenter) and WS2019 Standard guest with 2 disks (OS=NTFS / DATA= ReFS)

Destination: Esxi v8 (vSphere Enterprise plus)

Backup Server: WS2022 standard with Veeam B&R Enterprise 12.1.2.172

 

What I did for many VMs with success and with a very short downtime :

  1. Backup source (running VM) to a local repository
  2. Stop source VM
  3. Backup (incremental) to the same repository
  4. Instant Recovery to vSphere
  5. Install VMware Tools
  6. Reboot VM
  7. Check if everything is working
  8. Migrate to production

But on this WS2019 guest with a ReFS data volume, at step 4, just after the guest boot, the volume showed as RAW…

 

I found this in the ReFS log on the guest :

Event ID 7: "ReFS failed to mount the volume. Version 3.7 doesn't match expected value 3.4"

 

After googling for a while, I found that ReFS Version 3.7 is supported with WS2022 but not WS2019.

So I attached the RAW disk to a WS2022 guest, and managed to read the data correctly.

After that, I attached a new disk on my WS2019 to recover data from the WS2022.

 

The ReFS volume has been updated from v3.4 to v3.7, but when did that happen ?

 

The only WS2022 involved in these steps is the VBR server itself, so I wonder which steps are really done while running the Instant Recovery.

Is the guest ReFS volume mounted by the VBR server which would explain this unexpected upgrade of ReFS ?

 

Thanks for your attention.

 

Regards

Nico

 

I believe the root cause of the issue is the ReFS version mismatch. Windows Server 2019 supports ReFS up to version 3.4, while Windows Server 2022 introduces ReFS 3.7. When a volume formatted with 3.7 is accessed from a 2019 system, it will appear as RAW, since the older OS can't interpret the newer metadata structure.

This kind of version upgrade can happen when the VBR server (running WS2022) mounts the volume during Instant Recovery or related operations. Mounting a ReFS volume in WS2022 might trigger a silent upgrade to 3.7, even if the original OS was WS2019.

Hmmm....that's certainly interesting behavior. ​@matheusgiovanini brings up a good potential suggested cause of what the issue could be but am not sure. I'd take that info to Support & see what they say. Or...maybe even ping the Product Managers at the Forums. 

Thanks for your contributions  ​@coolsport00 ​@matheusgiovanini 

I must add that I am stuck with Veeam 12.1 because of multiple Veeam infrastructure components used by 2 Veeam B&R instances (I can’t update Veeam to 12.3 as the source infrastructure hosts ans repos would update their components and cause future failed backups & replications).

So I don’t know if 12.3 would produce the same.

It’s about the recovery process and mounting the disks to the server for the V2V conversion.

https://gist.github.com/XenoPanther/15d8fad49fbd51c6bd946f2974084ef8

When some ReFS versions get mounted in a writable way, the Windows server will upgrade the ReFS version to the current version for that Windows Server.

During V2V, if the target is HyperV or VMware, Veeam will ensure correct drivers and tools (e.g., Vmware tools) are installed during the restore process, so likely that is what did it.

I couldn’t find an official source from Microsoft but the behavior is reported here as well

It’s really best if your backup environment matches same windows versions as your production workload if ReFS is in play as ReFS has some tricky behaviors you need to watch out for.

Does this happen when you do an Entire VM Restore instead of an Instant Recovery? That should confirm whether this is caused by Veeam mounting the drive or not.

In an Entire VM Restore, no OS other than the restored OS will mount the drive, so the REFS version should stay the same.

Good points ​@ddomask ​@Tommy O'Shea . This is a learning experience for me re: ReFS behavior. Personally...I’ve never been a fan cuz of all the “quirkiness” of it I’ve heard over the yrs. Linux XFS FTW! 🤣

​@Tommy O'Shea  I didn’t try the entire VM restore but I agree, I don’t think ReFS would be upgraded in this case.

As ​@ddomask points it out, I think the upgrade occurs at the V2V conversion. Unfortunately during a migration with multiple guest OSes at the source, it’s quite difficult to match OS version between backup environment and guests… I will carefuly check if ReFS is used next time and adapt in consequence.

​@coolsport00 I personally choose XFS over ReFS for Repo, but in that scenario ReFS was used at the guest level… 

Anyway, hard lesson learned but everything is working at the end :)

 

Glad to hear all is now well ​@Telnet23 😊

Comment


Terms & Conditions