Skip to main content
Question

Issue with worker on a Proxmox node

E
  • Edy
  • Comes here often
  • 14 comments

Hello All,

I'm testing Veeam with my homelab Proxmox cluster. I have an issue with a worker on a node that is not reachable every other day. The backup gives a warning message. When I test the worker in the Veeam console I'm getting this message

 

<<<<<<

19/01/2025 20:42:53 Error Failed to synchronize configuration settings of the worker XXX-worker can't lock file '/var/lock/qemu-server/lock-101.conf' - got timeout

19/01/2025 20:42:53 Error Worker XXXworker test failed: can't lock file '/var/lock/qemu-server/lock-101.conf' - got timeout

Here is the warning message from the email

Failed to prepare the worker XXX-worker: Failed to power on the worker VM: Failed to wait for the task UPID:XXX:0006259E:0080C27A:678CCD30:qmstart:101:root@pam: to complete; There are no available workers in the cluster. Performance may be affected; Failed to prepare the worker worker: Failed to power on the worker VM: storage 'datastore' does not exist; Job finished with warning at 1/19/2025 1:25:27 PM

<<<<<<

I tried to resolve it with ChatGPT using the Proxmox node shell

  • there is a lock for the VM, but removing it, won’t help
  • VM status is stopped
  • in the GUI I see a task trying to start the VM.

What resolved the issue is when I use

  • systemctl restart pvedaemon

But what happens is the next Veeam backup will run fine and the following fails again. It’s a pattern of every two days. I upgraded VBR from 12.2 to 12.3 with the hope it will resolve the issue.

 

Any idea what is going on. I have another Proxmox node and I don’t see the issue there.

 

Thanks,

Edy

 

 

9 comments

Andanet
Forum|alt.badge.img+11
  • Andanet
  • Veeam Legend
  • 362 comments
  • January 27, 2025

Hi ​@Edy 

I’m not an expert in Proxmox but I start to work with it in my lab, and I checked differents issue due to hardware compatibility and configurations.  Other than that, there are also a few suggestions on the Proxmox forum you might find useful. 

you can try with this command

qm unlock 101

or continue reading here

https://forum.proxmox.com/threads/task-error-cant-lock-file-var-lock-qemu-server-lock-xxx-conf-got-timeout.146330/

https://forum.proxmox.com/threads/trying-to-aquire-lock-task-error-cant-lock-file-var-lock-qemu-server.16235/

I think, considering that this is not a Veeam issue, best way is to delete the worker node and recreate it. 

Please let us know it is going. 

 

 

Antonio D'Andrea | VMCE 2021 | VMCA 2022 | VMTSP2024 | Veeam Legend 2023/2024 | VUG Italy Leader | VCP6-DCV | I want it written on my tombstone "Please start a restore to the last valid backup"
Chris.Childerhose
1 person likes this
  • Chris.Childerhose
    Chris.Childerhose
lukas.k
Forum|alt.badge.img+10
  • lukas.k
  • Veeam Vanguard
  • 230 comments
  • January 27, 2025

Hi Edy,

Are there any other tasks that may block resources or processes on the Proxmox node? I know similar issues from vSphere and AHV (Nutanix) environments and I had several issues when two solutions (2x Veeam or Veeam and Cohesity) are performing concurrent tasks.

Did an upgrade to v12.3 solve your issue?

 

Best

Lukas

LK | Veeam Vanguard | VUG Leader - Cyber Security Space | Security Specialist
E
  • Edy
  • Author
  • Comes here often
  • 14 comments
  • January 27, 2025

Hi Lukas,

 

No the upgrade to 12.3 didn’t resolve the issue. I’m performing also a vzdump on the node, but not at the same time when the Veeam backup is running. Maybe I should temporarily disable the vddump and see if it makes a difference.

lukas.k
Forum|alt.badge.img+10
  • lukas.k
  • Veeam Vanguard
  • 230 comments
  • January 27, 2025

Yes maybe try that. Basically this is an approach to avoid ressources to be addressed by different processes.

 

I’m not yet very experienced with Proxmox so in case this doesn’t help maybe you should redeploy the workers and see if that helps.

LK | Veeam Vanguard | VUG Leader - Cyber Security Space | Security Specialist
E
  • Edy
  • Author
  • Comes here often
  • 14 comments
  • January 27, 2025

Thanks Lukas - I have halted now the vzdump and I will see in two day how it goes.

Chris.Childerhose
lukas.k
2 people like this
  • Chris.Childerhose
    Chris.Childerhose
  • lukas.k
    lukas.k
R
  • robin.milk
  • New Here
  • 2 comments
  • January 28, 2025

I'm having the same issue. Maybe not every other day, but it happened like 5-6 times since I configured it 2 months ago.

My solution is to go into the Task Queue and kill the Task via the “Stop” button.


​​​​

After that, the backup runs fine. The Problem is to notice it, that usually happens when my monitoring tells me that a backup job hasn't finished for 3 days.

E
  • Edy
  • Author
  • Comes here often
  • 14 comments
  • January 28, 2025

Thanks for the information and what your solution is.

 

I’m curious how the worker works. In the affected node I have 3 VMs. Does the worker start/stop for each VM is backs up or once the worker runs it stays on until the backup job is finished?

R
  • robin.milk
  • New Here
  • 2 comments
  • January 28, 2025

veeam starts the Worker-VM via the Proxmox API and also stops it, once it's done. You can see the start and stop request in the “Cluster log”s

E
  • Edy
  • Author
  • Comes here often
  • 14 comments
  • February 1, 2025

A few days later my issue still persists. What I tried in the meantime

 

  • I stopped vzdump to use less resource
  • I redeployed the worker

What happens the task to stop the worker is running. I cannot stop the tast as someone mentioned. What stops the task in my case is

systemctl restart pvedaemon

 

 

Comment


Terms & Conditions