Veeam Proxy for Linux has Problems with LVM


Userlevel 7
Badge +6

We had some problems with restoring VMs in the last time.

After some research we found that the Proxy on Linux has a problem with the assignments of physical volumes. It seems that there are PVs of target machines are preserved in the LVM archive files of the proxy… This is causing problems at restore time – leaving the restored VMs unbootable.

I have double checked the Veeam documentation about prerequisites (https://helpcenter.veeam.com/docs/backup/vsphere/backup_proxy_requirements.html?ver=110) and found no hint that LVM is a problem with Linux proxy.

The problem occurred with Suse Linux only, we are in the process to check if there is the same problem with RedHat Linux.

 

To visualize the problem, I will show it from the Veeam Proxy perspective and from the target VMs perspective.

 

Veeam Proxy

List the physical volumes with pvs. One device is not found…
Grep for the UUID of the missing volume in the directory /etc/lvm/archive

In this case the UUID is found in three files. At these three occasions this volume was seen as a local physical volume of the proxy and saved into the LVM archive…

To which target VM belongs this device originally?

Our Linux administration team has a central index of the PVs of all VMs. So, they can search for the UUID.

Now we know that the target VM with the number 121 is the one who owns the original PV.

 

Target VM

On the target VM a physical volume with the offending UUID can be found (this is the intended location of this PV 😊).

 

Conclusion

So, it seems that the usage of LVM on the Veeam proxy can damage the management of the PVs on the proxy. In this case the target VM was not bootable after restoring, because the described PV was not correctly assigned to the VM.

There is no error in Veeam at backup or restore, all is looking good. But the restored VM is not bootable. So, this is very good argument to use Sure backup and Data Labs to make sure your backups are restorable and functional.

We had this problem on Suse Linux only up to now. But we will setup all our Linux proxy VMs with LVM now, regardless if they are running on Suse or RedHat Linux.

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Don’t use LVM on any Linux proxy. Deactivate it, or better, don’t install it on the proxy VM.


22 comments

Userlevel 7
Badge +5

Great walkthrough and very informative conclusion… Thank you @JMeixner 

Userlevel 3
Badge

thanks for sharing @JMeixner 

Userlevel 7
Badge +6

Interesting to learn new things about Linux especially something affecting the proxy role. Thanks for sharing this.

Userlevel 7
Badge +2

Hey @JMeixner - I’m sorry you ‘discovered’ this in this manner. I do like this part however:

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Userlevel 7
Badge +6

Hey @JMeixner - I’m sorry you ‘discovered’ this in this manner. I do like this part however:

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Hi Rick,
is this a known issue? I did not find anything about this...

Userlevel 7
Badge +2

Hey @JMeixner - I’m sorry you ‘discovered’ this in this manner. I do like this part however:

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Hi Rick,
is this a known issue? I did not find anything about this...

Well if you confirmed with support - it is known. I’ll see if we can get the docs updated.

Userlevel 7
Badge +6

Hey @JMeixner - I’m sorry you ‘discovered’ this in this manner. I do like this part however:

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Hi Rick,
is this a known issue? I did not find anything about this...

Well if you confirmed with support - it is known. I’ll see if we can get the docs updated.

😂😂😂 ok… yes, if you see it this way the issue is known now...

Userlevel 7
Badge +7

Thanks @JMeixner for your post! Very good to know!

Userlevel 7
Badge +3

Thanks @JMeixner 

Userlevel 7
Badge +4

Thanks for your sharing @JMeixner.🙇

Userlevel 7
Badge +3

Thank you for sharing @JMeixner , interesting fact to know

Userlevel 7
Badge +2

I have raised it to product management @JMeixner  - so stay tuned!

Userlevel 7
Badge +6

Thanks, @Rick Vanover 

The LVM docs say that the archive/backup functionality can be disabled in /etc/lvm/lvm.conf - any idea if that would solve the issue?

We’re using Red Hat 8 proxies with LVM enabled and surebackup on a subset every night - we’ve not noted any issues so far with restores and LVM ourselves.

Is your LVM setup particularly complex?

Userlevel 7
Badge +6

I had the problem primarly with SuSe.

But as LVM is not needed for the proxy functionality the most save solution is to deactivate or deinstall it.

We have newly installed our Linux proxys without LVM and the problem did not reoccur up to now.

Userlevel 3
Badge

Thanks for the great and detailed post. 
Is there also an issue if you use LVM on the target VM’s? All are systems are setup with LVM for the data disk.

 

Userlevel 7
Badge +6

I have not seen problems with LVM on the target machines. The problem is with LVM on the Veeam proxy machines only.

Userlevel 7
Badge +4

@JMeixner thx for share info!

Thanks. Good to know.
We are using a mix of Proxy + Repo on CentOS and Rocky Linux (all from the RHEL family), and LVM is everywhere excluding disks where backups are stored. 
There are daily surebackup jobs, no one VM failed within two years. 

 
Userlevel 7
Badge +6

Interesting issue, thanks for posting it @JMeixner. Are the backups itself ok and is only the restore not working in that case? 

Userlevel 1
Badge

@JMeixner : before changing the user guide (or maybe even the product), we need to be sure what the real reason is. I got a case number from @Rick Vanover , but that does not mention anything about restore issues.

So my question is: is there a case number for the restore issues and could I please get that case number? 

Userlevel 7
Badge +6

@JMeixner : before changing the user guide (or maybe even the product), we need to be sure what the real reason is. I got a case number from @Rick Vanover , but that does not mention anything about restore issues.

So my question is: is there a case number for the restore issues and could I please get that case number? 

Hello @HannesK , this should be the correct case. I will check again.

Comment