Veeam Agent for Linux DKMS vs Prebuilt | Veeam Community Resource Hub

During the latest local Veeam User Group in Belgium, we presented a Veeam Agent for Linux deep dive specifically around the CBT driver. In the past, I’ve seen a lot of confusion about this topic leading to people installing just everything. In this small post, I just wanted to summarise some of the findings that were presented and hopefully it will help somebody in the future.

So first of all, what is the CBT driver or Kernel module that we ship. Well the CBT driver allows us to do two things

Create snapshots on block devices
Create a tracking of the blocks that are changed between backups. This improves incremental backup as we only need to read the changed data

There are two kinds of drivers. The reason being that some of the kernel functionality that the initial driver, “veeamsnap” was using have been deprecated. Therefore as a company, Veeam has decided to add new functionality in the Linux kernel (there is an ongoing effort to get it in the main kernel sources). The new driver “blksnap” in a way is split in 2 parts. First is the “blkfilter”, this is a generic mechanism that we are trying to contribute that allows any other company to capture io while passing to the io device. For example, an antivirus company could use this to scan newly written blocks only. The second is “blksnap” itself which uses this framework to hook into the IO path. Which one should you choose? Well in generally, kernels lower then 5.10 can use veeamsnap. Kernel higher or equal can use blksnap. Unfortunately, it is not that as black and white since for example Redhat backports fixes to older kernels for stability. The best thing you can do is validate via https://www.veeam.com/kb2804 . Most likely, if you just installed a clean system with the latest version, you will probably need blksnap at this point.

But now the bigger question arrives, which path do you want to go? DKMS or Prebuilt? And what is the difference. Well prebuilt means that Veeam compiles the kernel modules as a binary and ships this package. DKMS stand for Dynamic Kernel Module Support and it means that the package contains source code. The system will automatically compile the code when a new kernel is installed. This is great because kernel modules must roughly match the kernel they are running. So what are the advantages and disavantages of both

DKMS

You can recognise the packages because the modules contains “no arch” or “all”. Most of the time the package is smaller as it is just source code.

Advantage

“Always ready” for the newest kernel

Disadvantage

Requires build tools (gcc, kernel headers, dkms, etc). This can take a lot of additional place and packages. This might upset the security people. You might enable extra repositories such as peel-release on rocky to download dkms.
The build process should go smooth but it is harder to troubleshoot

Prebuilt

You can recognise the packages because they contain “x86_64” in the name or “kmod” “kmp”

Advantage

Easy to install (just install the package and you are done)
Doesn’t have dependencies

Disadvantage

You might want to make sure before you upgrade the kernel that Veeam has shipped a matching module

So which one do you choose? Well typically I say to customers, if it is Redhat alike, you might want to go with prebuilt. If you are Debian/Ubuntu alike, since they update the kernel more often, you might want to go with DKMS. Important here is that you have a choice when you are doing a manual install, so this is not a fixed rule. Eg in the RHEL 9 repo, https://repository.veeam.com/backup/linux/agent/rpm/el/9/x86_64/, you can see both blksnap-6.3.1.1016-1.noarch.rpm (dkms) vs kmod-blksnap-6.3.1.1016-1.el9.x86_64.rpm (prebuilt)

Finally if you install a kernel module and you have Secure Boot enabled, you have to enrol the public key of the instance that compiled the software in your UEFI by importing with mokutil, rebooting and accepting it. Here I see a lot of confusion. Which key do you need to enrol? Well if you use prebuilt kernel modules, you need to use enrol the public key of Veeam R&D. This because it is Veeam who compiled it. The key sits in a package uefi-certs. If you didn’t get an error installing this package, the root password hash is used during installation and asked at reboot. I did get an error on my rocky tryout (presumably because I didn't have a root password), so I had to manually import the certificate after installation from “/etc/uefi/certs/blksnap-ueficert” with mokutil.

If you are using DKMS however, you don’t need the uefi-certs package. Who compiled the module? DKMS did, so you have to enrol the DKMS key (“/var/lib/dkms/mok.key” on rocky for example).

Once you have loaded the key, you need to reboot and approve the key via UEFI before the os boots. If you don’t do this, you might get an error such as “failed to load module veeamblksnap” as the kernel refuses to load an unrecognised kernel module.