So I had a unique support case recently, a customer using Veeam Agent for Linux with mixed success, they had one particular server that worked fine during implementation and then after a week the backups had started to immediately fail. The server was unable to load the veeamsnap module.
Veeam themselves have a great article dealing with this in general and it helps resolve the majority of the cases, link here.
One point I believe this article doesn’t properly discuss though is why this can happen, especially when it previously worked properly. To understand this we need to know a bit about how Veeam works with its Linux Agent.
To fully understand how the Veeam Agent for Linux works, the first question we need to answer is, what is “veeamsnap”? veeamsnap is a Linux kernel module used by Veeam to create volume snapshots. Due to the huge variety of Linux systems and updates supported by Veeam, there are some pre-requisites that must be met for veeamsnap to function, these are documented here.
The key pre-requisite that caused the issue for my customer was “linux-headers”. Veeam documentation does a good job is highlighting that you need these but stops short of telling you clearly how to find this. To find out what kernel you’re using just run the following command:
uname -srmYou’ll get output like this:
Linux 4.15.0-120-generic x86_64The key piece of information here is the final suffix in the number “generic”. This means we’re using the generic Linux kernel and need the Linux Generic Headers.
So, what are these headers? At a technical level, kernel headers provide the various function/structure definitions required when compiling code that will interact with the kernel. At an overview level, we require the matching kernel headers to the exact kernel build we are using. This means when Veeam builds its veeamsnap module for our kernel, we have a high degree of confidence that whilst Veeam can’t test every possible kernel in existance, we can be sure that the structure being used for the invoking of functions and parameters being passed are compatible with what the kernel is expecting.
So now we know what headers are, how do we install them? The Veeam article I linked above includes commands for installing the specific kernel headers required, for archival purposes I’ve included the commands below:
Debian:
sudo apt-get install linux-headers-$(uname -r)
RHEL/OEL/Centos:
sudo yum install kernel-headers-$(uname -r)
Fedora:
sudo dnf install kernel-headers-$(uname -r)My one issue with this is that if you enable automatic updates, this issue will reoccur the next time your kernel is updated as we’ve installed a specific version of the kernel headers. To resolve this permanently you need to install the main package of your relevant headers, for example to install the generic headers we can type in:
sudo apt-get install linux-headers-genericNow there are other alternative kernel builds out and I won’t attempt to document them all here, but if you’ve inherited a system whereby someone deployed a different kernel and you’ve been hitting a brick wall, hopefully this will help. Once we’ve installed the main package instead of a specific version, when you run your updating, automatically or manually, this will be included.
Fun fact to close on, did you know there are numerous reduced latency kernels within Linux, mainly from the days of trying to achieve low/no latency audio processing? For example: -preempt, -lowlatency, -rt, -realtime. They rely on more aggressive context switching to reduce latency mainly with various tweaks and reduction in the use of power-saving techniques for responsiveness.
