• EN
Question

Stuck VMDK From the Veeam Proxies

Stabz
Userlevel 7
Badge +7

Hey guys,
I have the same random problem with my Proxies (Windows), VMDK are still attached to my proxies randomly once the job finish. 
The proxies are not backuped during the different backup jobs. 


The Vms that Veeam processed is found to require consolidation.

On the vmware side I found a log " Unable to access file since it is locked An error occurred while consolidating disks: Failed to lock the file. Consolidation failed for disk node 'scsi0:0': Failed to lock the file." at the end of the backup job.

But I don’t find the root cause. Have any of you ever encountered this problem?

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/

31 comments

tarik.yenisey
Userlevel 5
Badge +1

Hey guys,
I have the same random problem with my Proxies (Windows), VMDK are still attached to my proxies randomly once the job finish. 
The proxies are not backuped during the different backup jobs. 


The Vms that Veeam processed is found to require consolidation.

On the vmware side I found a log " Unable to access file since it is locked An error occurred while consolidating disks: Failed to lock the file. Consolidation failed for disk node 'scsi0:0': Failed to lock the file." at the end of the backup job.

But I don’t find the root cause. Have any of you ever encountered this problem?

I used to encounter this problem very often.

- Option 1 may be due to insufficient resources on your Veeam server or proxies. You can increase resources.

- As for option 2, I experienced this problem on 4 proxy servers at the same time. The problem was caused by the transport service. Even if I upgraded or deleted it and reinstalled it, it did not solve it. I set up a new server and selected it as proxy again and it worked.

Chris.Childerhose
Userlevel 7
Badge +20

I have been running into this issue with Linux proxies and have found that while there are no snapshots visible when looking at snapshots of the target vm, there are snapshots if you look at the diskpath and the vmdk files. I was able to remove the stuck disk from the proxy by editing the proxies settings, and was then able to successfully consolidate the vm. For us the problem doesn’t happen often, my biggest issue is that while Veeam recognizes the target vm “has stuck VM snapshot”, it is listed as a success or info item in the log and there does not seem to be a way to monitor for these issues, no alarms in Veeam ONE or the VSPC.

Be sure with Linux Proxies to disable multipathing as that is the recommended setting from Veeam.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
B

I have been running into this issue with Linux proxies and have found that while there are no snapshots visible when looking at snapshots of the target vm, there are snapshots if you look at the diskpath and the vmdk files. I was able to remove the stuck disk from the proxy by editing the proxies settings, and was then able to successfully consolidate the vm. For us the problem doesn’t happen often, my biggest issue is that while Veeam recognizes the target vm “has stuck VM snapshot”, it is listed as a success or info item in the log and there does not seem to be a way to monitor for these issues, no alarms in Veeam ONE or the VSPC.

E
Userlevel 4
Badge +2

you said you also created a Veeam support case.
Is there any progress there?
Is there anything in the logs on the proxy (eventlogs and or veeamlogs) than could explain this behavior.

 

Stabz
Userlevel 7
Badge +7

We are using a windows with desktop experience
 

 

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Chris.Childerhose
Userlevel 7
Badge +20

Curious, what version of windows are you running on the proxies?

Actually that is a good question as we tend “if” using Windows to use Core not desktop experience.  But not sure it would make a difference in this case.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
E
Userlevel 4
Badge +2

Curious, what version of windows are you running on the proxies?

Chris.Childerhose
Userlevel 7
Badge +20

That is interesting the Linux proxy does not have the issue over Windows as I would figure the other way around.  At this point I might suggest a support case if you don’t already have one to get to the bottom of this.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Stabz
Userlevel 7
Badge +7

Hi @Stabz I had a hunch: are proxies present from previous installations? For example, v11 --> v12 update?

Have you tried to add a new fresh proxy and tested it to failed backup job? 

 

Hey !

I installed a fresh Linux Proxy on one site and a new fresh Windows proxy on another.

It seems that I never have any stuck disk on my Linux proxy. On my new Windows  proxy without any hardening etc  I got the problem again....

Another interresting thing the problem seems to affect only production Windows virtual machines. 

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Chris.Childerhose
Userlevel 7
Badge +20

Hey folks!

A little update we opened a Vmware support ticket, and without surprise here a part of the answer
“During the backup process, Veeam will temporarily mount the VMDK files associated with your VMs. This allows Veeam to access the data necessary to create a complete backup. Once the backup finishes successfully, Veeam will automatically release the mounted VMDK files.

If you encounter a situation where a VMDK file remains mounted after a backup job completes, this may indicate an issue with Veeam itself. Hence, the issue is not with vCenter, and we recommend contacting Veeam support for additional assistance.”

Well, I ll continue to make some tests :)

So, point the finger back to Veeam LOL

Hopefully after more testing you figure this one out as I am curious now.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Andanet
Userlevel 7
Badge +10

Hi @Stabz I had a hunch: are proxies present from previous installations? For example, v11 --> v12 update?

Have you tried to add a new fresh proxy and tested it to failed backup job? 

 

Antonio D'Andrea | VCP6-DCV | VMCE 2021 | VMCA 2022 | VMTSP2023 | Veeam Legend 2023 | VUG Italy Leader | I want it written on my tombstone "Please start a restore to the last valid backup"
Stabz
Userlevel 7
Badge +7

Hey folks!

A little update we opened a Vmware support ticket, and without surprise here a part of the answer
“During the backup process, Veeam will temporarily mount the VMDK files associated with your VMs. This allows Veeam to access the data necessary to create a complete backup. Once the backup finishes successfully, Veeam will automatically release the mounted VMDK files.

If you encounter a situation where a VMDK file remains mounted after a backup job completes, this may indicate an issue with Veeam itself. Hence, the issue is not with vCenter, and we recommend contacting Veeam support for additional assistance.”

Well, I ll continue to make some tests :)

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Stabz
Userlevel 7
Badge +7

Yes for Linux proxy I always disabled Multipathing, but in my case I have Windows Proxy, cause I have a remote site with an Exagrid appliance so I need a Windows Proxy for specific restore.

I ll still investigate.

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
coolsport00
Userlevel 7
Badge +17

Glad to hear @Stabz . Just know it more than likely will reoccur, but I have found Linux to be not as bad as Windows, & I do not have LVM or Multipathing enabled on any of mine. 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
JMeixner
Userlevel 7
Badge +17

We have disabled Multipathing and LVM for all our Linux proxies (I think I have written a thread about this here in community). We had massive problems with LVM - not the exact error you are describing, but errors about accessing VMDKs, too.

After disabling LVM and Multipathing we have nearly no problem with the proxies anymore...

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
Stabz
Userlevel 7
Badge +7

Hey, 
Thanks for all this answers. 

The Vmotion works properly, I have migrated my proxies to an anothers hosts and datastores.

This morning I have only two Vms with this problem (always the same), I asked to my customer to open a case to Vmware support.

And there are no snapshots on the production Vms or on the proxies.  

 

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Andanet
Userlevel 7
Badge +10

Has the VM other active snapshot? Normally you can check the lck file doing a browse on folder datastore where VM reside. 

 

Antonio D'Andrea | VCP6-DCV | VMCE 2021 | VMCA 2022 | VMTSP2023 | Veeam Legend 2023 | VUG Italy Leader | I want it written on my tombstone "Please start a restore to the last valid backup"
Moustafa_Hindawi
Userlevel 7
Badge +6

Check Veeam Proxy design best practices:

vSphere Proxy | Veeam Backup & Replication Best Practice Guide

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/
coolsport00
Userlevel 7
Badge +17

For me, it's been hit or miss. I can go a week without issue, or I can have a couple days in a row where I have sporadic occurrences 😕

I think overall it's probably been less occurrence though. 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
Chris.Childerhose
Userlevel 7
Badge +20

Yes I just set a Linux proxy, the first test finished without error.  I ll see tomorrow morning the result with all my job.

If the error continue I ll ask to my customer to open a case to VMware too 


 

Well hopefully the Linux proxies lessen the frequency over Windows which would be ideal.  Keep us posted.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
coolsport00
Userlevel 7
Badge +17

Just as I shared above, I use Linux Proxies & the problem still arises on occasion. Just FYI. 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
MarkBoothman
Userlevel 7
Badge +6

It would be interesting to see if the proxy or VMware can vmotion fine. I’ve seen issues in the past where the host has locks on files and a restart of the management services fixes it or worst case a host restart.

VMCA2022 | VMCE2021 | Veeam Legend | Twitter: @Boothy71
Stabz
Userlevel 7
Badge +7

Yes I just set a Linux proxy, the first test finished without error.  I ll see tomorrow morning the result with all my job.

If the error continue I ll ask to my customer to open a case to VMware too 


 

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/
Chris.Childerhose
Userlevel 7
Badge +20

Do you have the ability to test a Linux Proxy to compare it to a Windows one?  If you do ensure to disable multipathing in Linux as per best practice.  This might be the way to help solve this.

That reply from support is very vague and leads me to think that you need a VMware support case too.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Stabz
Userlevel 7
Badge +7

I found that by disabling automount on our Windows proxies the issue became less frequent.

From and elevated cmd prompt > Diskpart 

Automount Disable

Exit

exit

 

 

Yep, normally since the v10 this is disble automatically, I try to force it but the problem persist.
(https://www.veeam.com/kb1882)

Here a part of the support answers:
 

“As Veeam only request to take a snapshot when backup start, & again request vcenter/esxi host to delete the snapshot once backup completes or fail. So we need to check why the esxi host is not deleting the snapshot when backup completed or fail.

Regarding VM disk are getting attached to proxy vm and not getting released need to check that in the esxi host as well, why it is not removing the disk after backup finished, it could be proxy not functioning properly or the esxi host is causing something here.

You can try migrating the vm to other esxi host & datastore see if it works here.”

Philippe DUPUIS |Veeam Certified Engineer | https://original-network.com/

Comment


Terms & Conditions