Skip to main content
  • EN

We had some problems with restoring VMs in the last time.

After some research we found that the Proxy on Linux has a problem with the assignments of physical volumes. It seems that there are PVs of target machines are preserved in the LVM archive files of the proxy… This is causing problems at restore time – leaving the restored VMs unbootable.

I have double checked the Veeam documentation about prerequisites (https://helpcenter.veeam.com/docs/backup/vsphere/backup_proxy_requirements.html?ver=110) and found no hint that LVM is a problem with Linux proxy.

The problem occurred with Suse Linux only, we are in the process to check if there is the same problem with RedHat Linux.

 

To visualize the problem, I will show it from the Veeam Proxy perspective and from the target VMs perspective.

 

Veeam Proxy

List the physical volumes with pvs. One device is not found…
Grep for the UUID of the missing volume in the directory /etc/lvm/archive

In this case the UUID is found in three files. At these three occasions this volume was seen as a local physical volume of the proxy and saved into the LVM archive…

To which target VM belongs this device originally?

Our Linux administration team has a central index of the PVs of all VMs. So, they can search for the UUID.

Now we know that the target VM with the number 121 is the one who owns the original PV.

 

Target VM

On the target VM a physical volume with the offending UUID can be found (this is the intended location of this PV 😊).

 

Conclusion

So, it seems that the usage of LVM on the Veeam proxy can damage the management of the PVs on the proxy. In this case the target VM was not bootable after restoring, because the described PV was not correctly assigned to the VM.

There is no error in Veeam at backup or restore, all is looking good. But the restored VM is not bootable. So, this is very good argument to use Sure backup and Data Labs to make sure your backups are restorable and functional.

We had this problem on Suse Linux only up to now. But we will setup all our Linux proxy VMs with LVM now, regardless if they are running on Suse or RedHat Linux.

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Don’t use LVM on any Linux proxy. Deactivate it, or better, don’t install it on the proxy VM.

Great walkthrough and very informative conclusion… Thank you @JMeixner 

thanks for sharing @JMeixner 

Interesting to learn new things about Linux especially something affecting the proxy role. Thanks for sharing this.

Hey @JMeixner - I’m sorry you ‘discovered’ this in this manner. I do like this part however:

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Hey @JMeixner - I’m sorry you ‘discovered’ this in this manner. I do like this part however:

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Hi Rick,
is this a known issue? I did not find anything about this...

Hey @JMeixner - I’m sorry you ‘discovered’ this in this manner. I do like this part however:

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Hi Rick,
is this a known issue? I did not find anything about this...

Well if you confirmed with support - it is known. I’ll see if we can get the docs updated.

Hey @JMeixner - I’m sorry you ‘discovered’ this in this manner. I do like this part however:

In the meantime, Veeam support has answered and confirmed that LVM should not be used on Linux Proxys. Unfortunately, this  is not reflected in the documentation. I hope this will be updated shortly.

Hi Rick,
is this a known issue? I did not find anything about this...

Well if you confirmed with support - it is known. I’ll see if we can get the docs updated.

😂😂😂 ok… yes, if you see it this way the issue is known now...

Thanks @JMeixner for your post! Very good to know!

Thanks @JMeixner 

Thanks for your sharing @JMeixner.🙇

Thank you for sharing @JMeixner , interesting fact to know

I have raised it to product management @JMeixner  - so stay tuned!

Thanks, @Rick Vanover 

The LVM docs say that the archive/backup functionality can be disabled in /etc/lvm/lvm.conf - any idea if that would solve the issue?

We’re using Red Hat 8 proxies with LVM enabled and surebackup on a subset every night - we’ve not noted any issues so far with restores and LVM ourselves.

Is your LVM setup particularly complex?

I had the problem primarly with SuSe.

But as LVM is not needed for the proxy functionality the most save solution is to deactivate or deinstall it.

We have newly installed our Linux proxys without LVM and the problem did not reoccur up to now.

Thanks for the great and detailed post. 
Is there also an issue if you use LVM on the target VM’s? All are systems are setup with LVM for the data disk.

 

I have not seen problems with LVM on the target machines. The problem is with LVM on the Veeam proxy machines only.

@JMeixner thx for share info!

Thanks. Good to know.
We are using a mix of Proxy + Repo on CentOS and Rocky Linux (all from the RHEL family), and LVM is everywhere excluding disks where backups are stored. 
There are daily surebackup jobs, no one VM failed within two years. 

 

Interesting issue, thanks for posting it @JMeixner. Are the backups itself ok and is only the restore not working in that case? 

@JMeixner : before changing the user guide (or maybe even the product), we need to be sure what the real reason is. I got a case number from @Rick Vanover , but that does not mention anything about restore issues.

So my question is: is there a case number for the restore issues and could I please get that case number? 

@JMeixner : before changing the user guide (or maybe even the product), we need to be sure what the real reason is. I got a case number from @Rick Vanover , but that does not mention anything about restore issues.

So my question is: is there a case number for the restore issues and could I please get that case number? 

Hello @HannesK , this should be the correct case. I will check again.

Has there ever been a result of this? I’m now busy for 5h to restore files from a Linux LVM filesystem but Veeam fails miserably in helping me doing this.

If I use one if our physical Linux proxies as helper, I get warnings like “Unable to mount the filesystem on one or more volumes”, Skipped “Mounting of lvm snapshots is not supported”. But there was no lvm snapshot, we don’t use that at all. The physical proxy has lvm filesystems as all our servers have.

If I use a Veeam temp helper appliance I don’t get that warning but when I navigate to the desired location in the Backup Browser, the directory is just empty. This is a directory with many files, 1-2 TB of small files. I waited for some time, but the directory stays empty. 

I hoped the files are just not displayed, but a restore attempt of the whole directory ended with only the empty directory restored. 

So now I have to do an IR of the VM, stop a bunch of services (hopefully I can identify all that talk to the outside), change IP and hope for the best in using rsync to bring the data back from IR VM (which will take ages). I’ve created 05971011 for this.

 

 

 

 

 

We have removed LVM from all out Linux proxys and didn't have any problems since then. It was a problem with LVM on the proxy, not with LVM on the target systems.

Veeam support did not find a definitive cause for the behaviour.

 

Last week or so was another topic in community with a similar problem...

We have removed LVM from all out Linux proxys and didn't have any problems since then. It was a problem with LVM on the proxy, not with LVM on the target systems.

Veeam support did not find a definitive cause for the behaviour.

 

Last week or so was another topic in community with a similar problem...

 

I checked documentation if there is now anything mentioned regarding lvm on proxy, but didn’t find anything. Not using lvm on proxies would not be an easy option as this is default for all Linux systems here.

Comment


Terms & Conditions