• EN
Question

Stuck VMDK From the Veeam Proxies

Stabz
Userlevel 7
Badge +7

Hey guys,
I have the same random problem with my Proxies (Windows), VMDK are still attached to my proxies randomly once the job finish. 
The proxies are not backuped during the different backup jobs. 


The Vms that Veeam processed is found to require consolidation.

On the vmware side I found a log " Unable to access file since it is locked An error occurred while consolidating disks: Failed to lock the file. Consolidation failed for disk node 'scsi0:0': Failed to lock the file." at the end of the backup job.

But I don’t find the root cause. Have any of you ever encountered this problem?

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/

31 comments

Chris.Childerhose
Userlevel 7
Badge +20

On occasion I have seen this happen but not that often to be honest.  We are switching to Linux Proxies and disabling multipathing as noted for them which seem to work much better with the HotAdd functionality.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Stabz
Userlevel 7
Badge +7

At every backup I have between 1-10 vms with this problem. First time I m facing this problem with Windows Proxies. Last time was cause Replication Job and Backup job was in parallel at the same time.

I have opened a support case.

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Chris.Childerhose
Userlevel 7
Badge +20

Yeah, hopefully Support can get it sorted out and determine the cause of it.  Keep us posted.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
coolsport00
Userlevel 7
Badge +17

Hi @Stabz - as Chris shared...I have this problem on occasion as well. It’s not just on Windows VM Proxies...but Linux as well. This can happen when you have multiple jobs attempting to back up a VM at the same time. For example, I have a CO-side VBR server which takes care of solely Backups. Then I have a DR-side VBR server which takes care of solely Replication jobs. Though I try my darndest not to have jobs on the 2 server overlap, they sometimes do, and this behavior can happen.

Other times, I’ve had this happen when jobs haven’t overlapped. This behavior has gone on for years (several)...at least for me.

Curious if Support has a resolution to it.

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
MarkBoothman
Userlevel 7
Badge +6

I found that by disabling automount on our Windows proxies the issue became less frequent.

From and elevated cmd prompt > Diskpart 

Automount Disable

Exit

exit

 

 

VMCA2022 | VMCE2021 | Veeam Legend | Twitter: @Boothy71
Chris.Childerhose
Userlevel 7
Badge +20

I found that by disabling automount on our Windows proxies the issue became less frequent.

From and elevated cmd prompt > Diskpart 

Automount Disable

Exit

exit

 

 

Yes, disabling this does help and is best practice for sure.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Stabz
Userlevel 7
Badge +7

I found that by disabling automount on our Windows proxies the issue became less frequent.

From and elevated cmd prompt > Diskpart 

Automount Disable

Exit

exit

 

 

Yep, normally since the v10 this is disble automatically, I try to force it but the problem persist.
(https://www.veeam.com/kb1882)

Here a part of the support answers:
 

“As Veeam only request to take a snapshot when backup start, & again request vcenter/esxi host to delete the snapshot once backup completes or fail. So we need to check why the esxi host is not deleting the snapshot when backup completed or fail.

Regarding VM disk are getting attached to proxy vm and not getting released need to check that in the esxi host as well, why it is not removing the disk after backup finished, it could be proxy not functioning properly or the esxi host is causing something here.

You can try migrating the vm to other esxi host & datastore see if it works here.”

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Chris.Childerhose
Userlevel 7
Badge +20

Do you have the ability to test a Linux Proxy to compare it to a Windows one?  If you do ensure to disable multipathing in Linux as per best practice.  This might be the way to help solve this.

That reply from support is very vague and leads me to think that you need a VMware support case too.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Stabz
Userlevel 7
Badge +7

Yes I just set a Linux proxy, the first test finished without error.  I ll see tomorrow morning the result with all my job.

If the error continue I ll ask to my customer to open a case to VMware too 


 

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
MarkBoothman
Userlevel 7
Badge +6

It would be interesting to see if the proxy or VMware can vmotion fine. I’ve seen issues in the past where the host has locks on files and a restart of the management services fixes it or worst case a host restart.

VMCA2022 | VMCE2021 | Veeam Legend | Twitter: @Boothy71
coolsport00
Userlevel 7
Badge +17

Just as I shared above, I use Linux Proxies & the problem still arises on occasion. Just FYI. 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
Chris.Childerhose
Userlevel 7
Badge +20

Yes I just set a Linux proxy, the first test finished without error.  I ll see tomorrow morning the result with all my job.

If the error continue I ll ask to my customer to open a case to VMware too 


 

Well hopefully the Linux proxies lessen the frequency over Windows which would be ideal.  Keep us posted.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
coolsport00
Userlevel 7
Badge +17

For me, it's been hit or miss. I can go a week without issue, or I can have a couple days in a row where I have sporadic occurrences 😕

I think overall it's probably been less occurrence though. 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
Moustafa_Hindawi
Userlevel 7
Badge +6

Check Veeam Proxy design best practices:

vSphere Proxy | Veeam Backup & Replication Best Practice Guide

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/
Andanet
Userlevel 7
Badge +10

Has the VM other active snapshot? Normally you can check the lck file doing a browse on folder datastore where VM reside. 

 

Antonio D'Andrea | VCP6-DCV | VMCE 2021 | VMCA 2022 | VMTSP2023 | Veeam Legend 2023 | VUG Italy Leader | I want it written on my tombstone "Please start a restore to the last valid backup"
Stabz
Userlevel 7
Badge +7

Hey, 
Thanks for all this answers. 

The Vmotion works properly, I have migrated my proxies to an anothers hosts and datastores.

This morning I have only two Vms with this problem (always the same), I asked to my customer to open a case to Vmware support.

And there are no snapshots on the production Vms or on the proxies.  

 

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
JMeixner
Userlevel 7
Badge +17

We have disabled Multipathing and LVM for all our Linux proxies (I think I have written a thread about this here in community). We had massive problems with LVM - not the exact error you are describing, but errors about accessing VMDKs, too.

After disabling LVM and Multipathing we have nearly no problem with the proxies anymore...

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
coolsport00
Userlevel 7
Badge +17

Glad to hear @Stabz . Just know it more than likely will reoccur, but I have found Linux to be not as bad as Windows, & I do not have LVM or Multipathing enabled on any of mine. 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
Stabz
Userlevel 7
Badge +7

Yes for Linux proxy I always disabled Multipathing, but in my case I have Windows Proxy, cause I have a remote site with an Exagrid appliance so I need a Windows Proxy for specific restore.

I ll still investigate.

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Stabz
Userlevel 7
Badge +7

Hey folks!

A little update we opened a Vmware support ticket, and without surprise here a part of the answer
“During the backup process, Veeam will temporarily mount the VMDK files associated with your VMs. This allows Veeam to access the data necessary to create a complete backup. Once the backup finishes successfully, Veeam will automatically release the mounted VMDK files.

If you encounter a situation where a VMDK file remains mounted after a backup job completes, this may indicate an issue with Veeam itself. Hence, the issue is not with vCenter, and we recommend contacting Veeam support for additional assistance.”

Well, I ll continue to make some tests :)

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Andanet
Userlevel 7
Badge +10

Hi @Stabz I had a hunch: are proxies present from previous installations? For example, v11 --> v12 update?

Have you tried to add a new fresh proxy and tested it to failed backup job? 

 

Antonio D'Andrea | VCP6-DCV | VMCE 2021 | VMCA 2022 | VMTSP2023 | Veeam Legend 2023 | VUG Italy Leader | I want it written on my tombstone "Please start a restore to the last valid backup"
Chris.Childerhose
Userlevel 7
Badge +20

Hey folks!

A little update we opened a Vmware support ticket, and without surprise here a part of the answer
“During the backup process, Veeam will temporarily mount the VMDK files associated with your VMs. This allows Veeam to access the data necessary to create a complete backup. Once the backup finishes successfully, Veeam will automatically release the mounted VMDK files.

If you encounter a situation where a VMDK file remains mounted after a backup job completes, this may indicate an issue with Veeam itself. Hence, the issue is not with vCenter, and we recommend contacting Veeam support for additional assistance.”

Well, I ll continue to make some tests :)

So, point the finger back to Veeam LOL

Hopefully after more testing you figure this one out as I am curious now.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Stabz
Userlevel 7
Badge +7

Hi @Stabz I had a hunch: are proxies present from previous installations? For example, v11 --> v12 update?

Have you tried to add a new fresh proxy and tested it to failed backup job? 

 

Hey !

I installed a fresh Linux Proxy on one site and a new fresh Windows proxy on another.

It seems that I never have any stuck disk on my Linux proxy. On my new Windows  proxy without any hardening etc  I got the problem again....

Another interresting thing the problem seems to affect only production Windows virtual machines. 

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Chris.Childerhose
Userlevel 7
Badge +20

That is interesting the Linux proxy does not have the issue over Windows as I would figure the other way around.  At this point I might suggest a support case if you don’t already have one to get to the bottom of this.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
E
Userlevel 4
Badge +2

Curious, what version of windows are you running on the proxies?

Comment


Terms & Conditions