Pre-Check for update VEEAM to Version 12 using VMware


Userlevel 4

Hello together,

before you update VEEAM to Version 12 using VMware vCenter/ESXi please check VMs HardDisks for duplicate UUID. In Version 11 this was never a problem. In the new Version VEEAM checks the UUID and backup will fail.

VMware has no problem with that most time. But they know the Problem - https://kb.vmware.com/s/article/2006865


This command will find any HardDisk with duplicate UUID - please note that this is VMware PowerCLI command

Get-VM | Get-HardDisk | Select @{N='VM';E={$_.Parent.Name}}, @{N='Uuid';E={$_.ExtensionData.Backing.Uuid}} | Group-Object -Property Uuid | ?{ $_.Count -gt 1 }


As Service Provider such changes in VEEAM are a total disaster!!


33 comments

Userlevel 4

@vAdmin You said it :(

I found over 300VMs over across multiple customers with duplicate UUIDs.

I understand that the main problem comes from VMware - but VMware has no problems with that at the moment, only VEEAM in Version 12 has it.

Userlevel 7
Badge +17

I will discus this issue with our VMware admins next week.

Userlevel 4

@vAdmin No, this will not work - I tested it

 

You have to identify the Disk, shutdown the VM and change UUID over the shell

vmkfstools -J setuuid <vmname>.vmdk

 

Userlevel 4

Inside the forum there is a workaround to change transport mode to NBD (network) - https://forums.veeam.com/post479533.html#p479533

But this is not a workaround for us - because we have a separate Backup VLAN/Network/Uplink

 

I created a Powershell script to Change Disk UUIDs without to reboot a vm - https://github.com/claranet/ClaranetHub/tree/main/VMware/Change-DiskUUID

Currently we test it on some vms with failed backups

Userlevel 7
Badge +17

Another question:
Is there an official statement from Veeam about this problem/feature (😎)?

Userlevel 7
Badge +17

Another question:
Is there an official statement from Veeam about this problem/feature (😎)?

Fabian ( @Mildur ) has sent an interesting VMware KB article to me:

https://kb.vmware.com/s/article/1021189

This is a possible explanation of the problem. Disks with the same UUID must not be mounted to the same VM. The Veeam backup proxys are mounting a lot of disks and there is a great chance that disks with the same UUID are among these.

Userlevel 4

I found a blogpost - I will review it and test it… If it work, i will reply the result here

 

https://blog.milla-online.de/duplicate-disk-uuids-and-how-to-get-rid-of-it-hopefully

Userlevel 7
Badge +7

Ouch! 😮

Thanks for sharing Martin..

Userlevel 7
Badge +2

Inside the forum there is a workaround to change transport mode to NBD (network) - https://forums.veeam.com/post479533.html#p479533

But this is not a workaround for us - because we have a separate Backup VLAN/Network/Uplink

 

I created a Powershell script to Change Disk UUIDs without to reboot a vm - https://github.com/claranet/ClaranetHub/tree/main/VMware/Change-DiskUUID

Currently we test it on some vms with failed backups

Wow, without a reboot 😃that sounds promising,

Thank you for sharing it here @Martin Weber 

Userlevel 7
Badge +20

Thanks for sharing this. Will come in handy to add to my script library.

Userlevel 1

Anyone know why it has become a problem now in V12? Why do we have the UUID checks in v12 and not in v11? Can’t we just leave the check out of v12 like it was in v11?

Userlevel 4

OK, found new issue on this Problem - Backup over HotAdd will not work with “Linked Clone” as the created in vCloud Director if enabled or VM created in Virtual Desktop Environment.

Reproduce

$VM = Get-VM "<my vm>"
$Snapshot = $VM | New-Snapshot -Name "BASE"
New-VM -Name "$($VM.Name) - Linked Clone" -VM $VM -VMHost $VM.VMHost -LinkedClone -ReferenceSnapshot $Snapshot

Get-VM -Name "$($VM.Name)*"| Get-HardDisk | Select @{N='Name';E={$_.Parent.Name}}, @{N='Uuid';E={$_.ExtensionData.Backing.Uuid}}

 

Userlevel 7
Badge +14

That's interesting and I've never heard of the issue with duplicate UUIDs. So if you have an affected disk, the backup job via hotadd/virtual appliance mode will fail?

According to the article only vSphere 6.7 and below have this issue. But if you upgraded to 7.0 or higher, it wouldn't solve already duplicated UUIDs ?

Userlevel 7
Badge +17

No, I have checked our 7 U3 clusters and there are duplicate UUIDs...

Userlevel 7
Badge +14

@JMeixner Ok, so this is something which needs to be corrected manually after upgrading.

@fspadaro Only V12 will have this issue.

Userlevel 7
Badge +7

Another question:
Is there an official statement from Veeam about this problem/feature (😎)?

I was wondering the same thing...

Userlevel 7
Badge +12

No, I didn’t had the chance yet to clarify the behavior. 

I suspect it's because VmWare does not support mapping discs with the same UUID to the same VM.
If you use a HotAdd proxy, the VM's discs are mapped to our proxy. If multiple VMs use discs with the same UUID, mapping to a Veeam HotAdd Proxy is not supported by VmWare.
The solution is to resolve the issue with duplicated UUIDs as recommended in the VmWare KB article.

https://kb.vmware.com/s/article/1021189

 

 

I’ll update this topic, when I have an “official” answer from my team.
 

Best,

Fabian

Userlevel 7
Badge +8

Answer from the support:

“In deploying vm from template they have duplicate vmdk UUID's.
This is known in vmware and at the moment there is no solution only a workaround, by deploying in same host and moving them after to different ones so no duplicates are created.
Here is where you are affected. In the past v11 did not check UUID's, it checked only moref ID's meaning that in theory you could have duplicate vddk UUID's. V12 is still checking moref id's but also UUID's in order to better handle metadata.”

Userlevel 7
Badge +2

@vAdminNo, this will not work - I tested it

 

You have to identify the Disk, shutdown the VM and change UUID over the shell

vmkfstools -J setuuid <vmname>.vmdk

 

Wow, that’s massive outage :-| for a big and busy VM.

Thank you for the sharing @Martin Weber 

Userlevel 4
Badge +2

Watching….

Userlevel 7
Badge +8

Thank you for sharing this, we discovered plenty of duplicate UUID. We will fix it before v12 update which is planned next week 🙄

Userlevel 4

@wouter.oltlammers Yes, to be compatible with vmware they added a DiskUUID check in Version 12. The Problem on VMware side is older - and maybe it can also have some effects there.

I changed 200VMs with this script last days and Backups worked fine again.

Pleas note, if you use Network Transportmode this does not affect - It only affect if you use HotAdd-Mode.

Userlevel 3
Badge +1

@JMeixner you check the issue also on 7.0u3, but is it still a problem with V12 hot-add jobs?

Userlevel 7
Badge +17

The problem is not VMs with the same UUID, but disks with the same UUID…

 

Userlevel 7
Badge +2

Hi @Martin Weber , 
Thank you for sharing.

So, the work around is to do the Storage vMotion after Deploying the VM from the Template.

What if the VM has already been deployed from Template and running as Production VM?

Comment