There has been an increase in interest from our customers and partners in alternative platforms and there’s no shortage of options in the hypervisor and cloud market. While many of the underlying virtualization components like KVM/Qemu/etc might be common to most of the platforms, the storage and management layers can vary quite a bit. Storage choices can affect the ability to take snapshots and capture changed block data between backup points via a host API and each management plane that sits above the compute/storage/networking can expose features and capabilities via the their APIs differently. KVM and Xen are Linux based hypervisor technologies that allow VM processes to access hardware through the host. The virtual disk data can be stored on a large variety of options such as local filesystems and disks, NFS, iSCSI/FC, LVMs, and distributed storage options like CEPH and Gluster or even proprietary options. Platforms can choose which software/hardware components to pair with their offering to allow VMs to migrate easily between hosts without downtime, and offer features like HA, snapshots and backup integrations depending on their target audience.
Some Veeam customers have virtual machines they want to protect on platforms that Backup & Replication doesn’t currently support for host-based backup integration. I was looking through a list of hypervisors that came up on the Veeam Forums and noticed there were a handful of customers asking about XenServer and XCP-ng. It’s been a while since I’ve seen Xen come up so I started looking into it and reading about Xen Orchestra and Vates XCP-ng. Xen Orchestra is a centralized, web-based management plane to manage XenServer and XCP-ng environments. You can deploy it as a prepackaged VM to a Xen server or manually deploy it somewhere to manage and monitor Xen-based environments. It is similar to vCenter for VMware and lets you manage and monitor hosts and clusters, storage, networking, images and virtual machines. Additional Information about Xen Orchestra can be found here: Xen Orchestra ✦ Web Interface for XenServer (xen-orchestra.com). There’s also a CLI that can be deployed via NPM (NPM is a package management tool that’s included with Node.JS): Architecture | XO documentation (xen-orchestra.com)
Veeam Agent for Windows and Veeam Agent for Linux are components of the Veeam Data Platform that can be used to protect Windows and Linux servers or desktop that run on bare-metal hardware or virtualization platforms. Because they are deployed on the operating system you wish to protect, they can be used even on hypervisor platforms where a native host-based integration isn’t available. I was easily able to deploy agents to a few machines and take backups and perform restores, but I wanted to see if I could use the data from the hypervisor to extend the offering a bit further.
I wanted to start by getting a list of VMs with their hardware configuration, name, IP address and virtual disk information. I started looking at the xo-cli tool that comes with the Xen Orchestra management server offering. After some research I found the following gave me what I needed:
- xo-cli list-objects type=VM
- xo-cli list-objects type=VDI
- xo-cli list-objects type=SR
- xo-cli list-objects type=network
- xo-cli list-objects type=VM-template
The type=VM returned the metadata for the VM itself which I will need later for restores, but also the IP address as long as the guest tools are installed. Having a list of IP’s will let me build a CSV file to create a protection group. I can get the OS type (as long as it’s set correctly), so I know if it’s a Windows or Linux VM. The type=VDI returns Virtual Disk information. Xen uses VHDs and RAW formats typically. I can also get a list of ISO’s that are stored in ISO Storage Repositories. VMs connect to the VDI’s through a VBD. I can match up which disks are attached to which VM by looking for objects with matching VBDs. The output is in a JSON format which can be imported into powershell easily. On my Linux VM with Node.JS and the xo-cli npm package installed, I ran the following commands to generate the files I wanted:
printf -v date '%(%Y-%m-%d_%H-%M)T\n' -1
xo-cli list-objects type=VM > vmlist_$date
xo-cli list-objects type=VDI >> vdilist_$date
xo-cli list-objects type=SR >> SRlist_$date
xo-cli list-objects type=network >> Netlist_$date
xo-cli list-objects type=VM-template
Then I used a Veeam File Copy job to copy these to my VBR server by pointing at this folder. You probably want to use some mv or rm commands to clean up older files periodically. Once I had this data on the VBR server, I could start to import it into Powershell with something like below:
$path = "E:\xoa_vmlist\vmlists"
$vmfile = Get-ChildItem -Path $path |where {$_.name -like "vm*"}| Sort-Object -Property name -Descending | Select-Object -First 1
$vmcontent = gc $path\$vmfile | Out-String
$vmdata = (ConvertFrom-Json $vmcontent)
$filtVMData = $vmdata |select type, name_label, addresses, CPUs, memory.static, powers_state, VIFs,UUID, @{name="VBDs";Expression= {$_.'$VBDs'}},os_version, network, VDIs, restorecmd
$vdifile = Get-ChildItem -Path $path |where {$_.name -like "vdi*"}| Sort-Object -Property name -Descending | Select-Object -First 1
$vdicontent = gc $path\$vdifile | Out-String
$vdidata = (ConvertFrom-Json $vdicontent)
$filtVDIData = $vdidata |Where-Object {$_.name_label -notlike "*.iso"} |select type, name_label, size, UUID, @{name="VBDs";Expression= {$_.'$VBDs'}}, @{name="SR";Expression= {$_.'$SR'}}, device
$srfile = Get-ChildItem -Path $path |where {$_.name -like "sr*"}| Sort-Object -Property name -Descending | Select-Object -First 1
$srcontent = gc $path\$srfile | Out-String
$srdata = (ConvertFrom-Json $srcontent)
$filtSRData = $srdata |select SR_type, name_label, size,physical_usage, allocationStrategy, shared, UUID, VDIs
$netfile = Get-ChildItem -Path $path |where {$_.name -like "net*"}| Sort-Object -Property name -Descending | Select-Object -First 1
$netcontent = gc $path\$netfile | Out-String
$netdata = (ConvertFrom-Json $netcontent)
$filtnetData = $netdata |select name_label, bridge, uuid, VIFs
This gives me a few variables I can use: $filtVMData $filtVDIData $filtSRData. Information from these variables can be joined to a single powershell object with something like this:
for ($i = 0; $i -lt $filtVMData.Count; $i++){
$vm = $filtVMdata0$i]
$vif = $vm.vifs
$vbd = $vm.VBDs
$netarray = foreach ($nic in $vif){
$filtnetData |where {$_.VIFs -like $nic}}
$filtVMDatar$i].network = $netarray.uuid
$repo = foreach ($d in $vbd){
$filtVDIData |where {$_.VBDs -like $d}}
$filtVMData=$i].VDIs = $repo
}
#add device id's for later restores
ForEach ($v in $filtVMData){
for ($x = 0; $x -lt @($v.VDIs).Count; $x++){
if ($v.VDIs -eq $null){
continue}
else{
$v.VDIs9$x].device = $x }
}
}
From here, you can get the list of IPs for Windows VMs and Linux VMs and create CSV files. You can compare an existing CSV with the latest data to see if you need to add any new machines that might have been created since the last time the script was run.
The New-VBRCSVContainer and Set-VBRCSVContainer Veeam PowerShell cmdlets can be used. More details on these commands can be found here: New-VBRCSVContainer and creating Agent Backup jobs information can be found here: Working with Veeam Agent Backup Jobs and Policies
Now that we have a way to automate backups, what can we do for restores? Veeam Agents have several restore options available including:
- Application Item Recovery (Microsoft Active Directory, SQL, Exchange, SharePoint, Oracle Database, etc)
- File Level Recovery for Windows, Linux and other operating systems.
- Volume Recovery which can restore non-OS volumes to a previous restore point
- Bare-metal Recovery: Using a Recovery ISO to restore to an empty VM/Physical server
- Exporting Volumes as Virtual Disks, including VHD’s which are the default virtual disk type for XenServer and XCP-NG.
This last option could be incredibly useful for automated larger scale recovery. By adding one of the physical hypervisor servers as a managed Linux machine, the Export as Virtual Disk feature allows you to directly export a disk/volume as a VHD that can be restored to the Storage Repository location such as an NFS share mounted to the host. After the disk is restored, it will need to be renamed by generating a new UUID on the Xen server and using a mv command to replace Veeam name with the new UUID. VHD’s have a size limitation of 2TB, but OS partitions are typically smaller than this. If additional disks are required that leverage the RAW format, those can be created and attached to the VM and a Volume Recovery can be used to restore the other virtual disks via the agent.
The xo-cli list-commands command showed me there is a xo-cli vm.create. All of the data I need to create this command was in my $filtVMData variable, so I used the following to generate the command as a property of each VM
#Create Restore Command
Foreach ($vm in $filtVMData){
$netid = @($vm.network)c0]
$rest_name = $vm.name_label + "_restored"
$rest_vdi = $vm.VDIs |Select-Object device, name_label, SR, size, type
$first = "xo-cli vm.create name_label=`"$rest_name`" template=`"UPDATE_ME`" VIFs=`'json:={`"network`":`"$netid`"}]`' VDIs=`'json:`"
$second = ""
foreach($d in $rest_vdi){
$random = -join ((97..122) | get-random -count 5 | %{achar]$_})
$rest_vdiname = $rest_name + "_" + $random
$dev = $d.device
$srid = $d.SR
$sz = $d.size
$t = $d.type
$rest_vdistr = "{`"device`":$dev, `"name_label`":`"$rest_vdiname`", `"SR`":`"$srid`", `"size`":$sz, `"type`":`"$t`"},"
$second = $second + $rest_vdistr
$newsecond = $second -replace ".{1}$", "]`'"
#$secnd = $second + "]`'"
$vm.restorecmd = $first + $newsecond
}
}
If VMs are attached to multiple networks, you will want to modify this script to loop through the VIFs. You will want to either automate the Template ID or copy it from the command up above that matches the OS of the server you’re restoring. I chose to run the following and copy/paste the UUID before restoring: xo-cli list-objects type=VM-template |grep "testvmcli" -A 40
The above command will create an empty VM that you can resize, or you could further enhance the script to add CPU count, memory and other parameters if needed and attach your Recovery ISO that was exported from the Veeam Server and uploaded to the Xen Orchestra ISO SR to and perform a restore. There also options to attach existing VDI’s with the xo-cli vm.restore command if you want to further script the VHD restore to attach it, or just leave out the VDI section to create an empty VM and attach the restored disk via the UI.
More information about the Bare-metal Recovery process can be found here:
- Windows Recovery Media: Restoring from Veeam Recovery Media - Veeam Agent for Microsoft Windows Guide
- Restoring from Veeam Recovery Media - Veeam Agent for Linux User Guide