• EN

Pre-Check for update VEEAM to Version 12 using VMware

Martin Weber
Userlevel 4

Hello together,

before you update VEEAM to Version 12 using VMware vCenter/ESXi please check VMs HardDisks for duplicate UUID. In Version 11 this was never a problem. In the new Version VEEAM checks the UUID and backup will fail.

VMware has no problem with that most time. But they know the Problem - https://kb.vmware.com/s/article/2006865


This command will find any HardDisk with duplicate UUID - please note that this is VMware PowerCLI command

Get-VM | Get-HardDisk | Select @{N='VM';E={$_.Parent.Name}}, @{N='Uuid';E={$_.ExtensionData.Backing.Uuid}} | Group-Object -Property Uuid | ?{ $_.Count -gt 1 }


As Service Provider such changes in VEEAM are a total disaster!!

Dont' drink and root! | Social: arogarth.net

33 comments

BertrandFR
Userlevel 7
Badge +8

Answer from the support:

“In deploying vm from template they have duplicate vmdk UUID's.
This is known in vmware and at the moment there is no solution only a workaround, by deploying in same host and moving them after to different ones so no duplicates are created.
Here is where you are affected. In the past v11 did not check UUID's, it checked only moref ID's meaning that in theory you could have duplicate vddk UUID's. V12 is still checking moref id's but also UUID's in order to better handle metadata.”

JMeixner
Userlevel 7
Badge +17

By any chance, do you all use linux proxies with lvm?
I ran a test on request of support without lvm and the VM's we have issues with processed successfully.

Oh, is support looking for this problem after all? I had this problem some time ago with LVM and nothing useful was found at this time...

 

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
fspadaro
Userlevel 3
Badge +1

By any chance, do you all use linux proxies with lvm?
I ran a test on request of support without lvm and the VM's we have issues with processed successfully.

No I’m using a windows proxy

Federico S. - VMCA2022 | VMCE2021 | VCAP-DCV/VCP-DCV | VCAP-NV/VCP-NV
J

By any chance, do you all use linux proxies with lvm?
I ran a test on request of support without lvm and the VM's we have issues with processed successfully.

fspadaro
Userlevel 3
Badge +1

The problem is not VMs with the same UUID, but disks with the same UUID…

 

Hi @JMeixner,

yes, didn't come out right.

I’ve backup with the same proxy in [HotAdd] mode two VMs with disks with the same UUID export with command on the top of the topics:

Get-VM | Get-HardDisk | Select @{N='VM';E={$_.Parent.Name}}, @{N='Uuid';E={$_.ExtensionData.Backing.Uuid}} | Group-Object -Property Uuid | ?{ $_.Count -gt 1 }

 

Regards

Federico S. - VMCA2022 | VMCE2021 | VCAP-DCV/VCP-DCV | VCAP-NV/VCP-NV
Mildur
Userlevel 7
Badge +12

No, I didn’t had the chance yet to clarify the behavior. 

I suspect it's because VmWare does not support mapping discs with the same UUID to the same VM.
If you use a HotAdd proxy, the VM's discs are mapped to our proxy. If multiple VMs use discs with the same UUID, mapping to a Veeam HotAdd Proxy is not supported by VmWare.
The solution is to resolve the issue with duplicated UUIDs as recommended in the VmWare KB article.

https://kb.vmware.com/s/article/1021189

 

 

I’ll update this topic, when I have an “official” answer from my team.
 

Best,

Fabian

Product Management Analyst @ Veeam Software
BertrandFR
Userlevel 7
Badge +8

not a funny discover specially from people which provides VMs from template with ansible but code will be fixed to avoid that symptom.

@HannesK @Mildur Could you confirm this is a known and expected behavior in v12? i did a support case to have an official answer today :)

@yohan CHABE

JMeixner
Userlevel 7
Badge +17

The problem is not VMs with the same UUID, but disks with the same UUID…

 

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
fspadaro
Userlevel 3
Badge +1

Hi to all,

I’m testing in my lab:

  • 2 VMs with the same UUID
  • Veeam DP v12
  • VMware 7.0U3

The proxy server process all disks (simultaneously) in Virtual appliance mode without problems.

 

Federico S. - VMCA2022 | VMCE2021 | VCAP-DCV/VCP-DCV | VCAP-NV/VCP-NV
fspadaro
Userlevel 3
Badge +1

Another question:
Is there an official statement from Veeam about this problem/feature (😎)?

Fabian ( @Mildur ) has sent an interesting VMware KB article to me:

https://kb.vmware.com/s/article/1021189

This is a possible explanation of the problem. Disks with the same UUID must not be mounted to the same VM. The Veeam backup proxys are mounting a lot of disks and there is a great chance that disks with the same UUID are among these.

 

 

 

 

 

 

I will check on my lab environment, where I’ve found 4 VMs with the same UUID. 

Federico S. - VMCA2022 | VMCE2021 | VCAP-DCV/VCP-DCV | VCAP-NV/VCP-NV
JMeixner
Userlevel 7
Badge +17

Another question:
Is there an official statement from Veeam about this problem/feature (😎)?

Fabian ( @Mildur ) has sent an interesting VMware KB article to me:

https://kb.vmware.com/s/article/1021189

This is a possible explanation of the problem. Disks with the same UUID must not be mounted to the same VM. The Veeam backup proxys are mounting a lot of disks and there is a great chance that disks with the same UUID are among these.

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
HunterLAFR
Userlevel 7
Badge +8

Inside the forum there is a workaround to change transport mode to NBD (network) - https://forums.veeam.com/post479533.html#p479533

But this is not a workaround for us - because we have a separate Backup VLAN/Network/Uplink

 

I created a Powershell script to Change Disk UUIDs without to reboot a vm - https://github.com/claranet/ClaranetHub/tree/main/VMware/Change-DiskUUID

Currently we test it on some vms with failed backups

thanks Martin!
this is gold!

Luis F.- HunterLF - HunterLAFR - lfconsulting.org
marco_s
Userlevel 7
Badge +7

Another question:
Is there an official statement from Veeam about this problem/feature (😎)?

I was wondering the same thing...

Marco Sorrentino - Italy | System Engineer | VMCE & Veeam Lover
JMeixner
Userlevel 7
Badge +17

Another question:
Is there an official statement from Veeam about this problem/feature (😎)?

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
JMeixner
Userlevel 7
Badge +17

I will discus this issue with our VMware admins next week.

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
regnor
Userlevel 7
Badge +14

@JMeixner Ok, so this is something which needs to be corrected manually after upgrading.

@fspadaro Only V12 will have this issue.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
fspadaro
Userlevel 3
Badge +1

@JMeixner you check the issue also on 7.0u3, but is it still a problem with V12 hot-add jobs?

Federico S. - VMCA2022 | VMCE2021 | VCAP-DCV/VCP-DCV | VCAP-NV/VCP-NV
JMeixner
Userlevel 7
Badge +17

No, I have checked our 7 U3 clusters and there are duplicate UUIDs...

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
regnor
Userlevel 7
Badge +14

That's interesting and I've never heard of the issue with duplicate UUIDs. So if you have an affected disk, the backup job via hotadd/virtual appliance mode will fail?

According to the article only vSphere 6.7 and below have this issue. But if you upgraded to 7.0 or higher, it wouldn't solve already duplicated UUIDs ?

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
Martin Weber
Userlevel 4

OK, found new issue on this Problem - Backup over HotAdd will not work with “Linked Clone” as the created in vCloud Director if enabled or VM created in Virtual Desktop Environment.

Reproduce

$VM = Get-VM "<my vm>"
$Snapshot = $VM | New-Snapshot -Name "BASE"
New-VM -Name "$($VM.Name) - Linked Clone" -VM $VM -VMHost $VM.VMHost -LinkedClone -ReferenceSnapshot $Snapshot

Get-VM -Name "$($VM.Name)*"| Get-HardDisk | Select @{N='Name';E={$_.Parent.Name}}, @{N='Uuid';E={$_.ExtensionData.Backing.Uuid}}

 

Dont' drink and root! | Social: arogarth.net
Martin Weber
Userlevel 4

@wouter.oltlammers Yes, to be compatible with vmware they added a DiskUUID check in Version 12. The Problem on VMware side is older - and maybe it can also have some effects there.

I changed 200VMs with this script last days and Backups worked fine again.

Pleas note, if you use Network Transportmode this does not affect - It only affect if you use HotAdd-Mode.

Dont' drink and root! | Social: arogarth.net
wouter.oltlammers
Userlevel 1

Anyone know why it has become a problem now in V12? Why do we have the UUID checks in v12 and not in v11? Can’t we just leave the check out of v12 like it was in v11?

BertrandFR
Userlevel 7
Badge +8

Thank you for sharing this, we discovered plenty of duplicate UUID. We will fix it before v12 update which is planned next week 🙄

Chris.Childerhose
Userlevel 7
Badge +20

Thanks for sharing this. Will come in handy to add to my script library.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
vAdmin
Userlevel 7
Badge +2

Inside the forum there is a workaround to change transport mode to NBD (network) - https://forums.veeam.com/post479533.html#p479533

But this is not a workaround for us - because we have a separate Backup VLAN/Network/Uplink

 

I created a Powershell script to Change Disk UUIDs without to reboot a vm - https://github.com/claranet/ClaranetHub/tree/main/VMware/Change-DiskUUID

Currently we test it on some vms with failed backups

Wow, without a reboot 😃that sounds promising,

Thank you for sharing it here @Martin Weber 

Microsoft Azure Solutions Architect Expert, VMware Certified Professional - DCV 7, Citrix Certified Professional Virtualization (CCP-V), ITIL Practitioner

Comment


Terms & Conditions