Malware Detection, Inline Entropy Scan – A Deeper Dive

With the release of Veeam Backup & Replication v12.1 came one of several anticipated security features – Malware Detection. Its ability to proactively detect ransomware potentially before it engages in malicious intent within an organization cannot be overstated. It's an invaluable feature! The downside to this feature is it's a "v1" release. And with it being initial inception comes some growing pains.

Since enabling Malware Detection in my environment, I'll share some concerns I've encountered, specifically towards the Inline Entropy scan engine; although I'll touch a little bit on File System Analysis as well. I'll also discuss what I've learned in how to interpret some of what Veeam flags as suspicious, how to prevent future systems from being flagged, what's lacking, and what the future enhancements roadmap looks like (or may look like..it's always a "tbd" thing with feature enhancements)… based off some research and reviews I've found.

 

ENABLE MALWARE DETECTION

So let's begin. First off, how do you enable Malware Detection since it's not enabled by default? This is easy enough. You do so from the Veeam Console > Malware Detection menu option:

Then, you select which engine to enable, or both – Inline Entropy analysis, File System analysis, as well as optionally enable Incident API (2nd tab) to perform Quick Backups.

When you enable the Inline Entropy scan, Backup Jobs, upon first run after enabling, will not use CBT so as to complete a full disk read on each machine disk. Subsequent Job runs will then start using CBT. The full disk read creates a RIDX file on the Backup Proxy and is used for comparison purposes to detect potential malicious activity. Inline Entropy and File System scans warrant a bit more description, so I'll share more on the differences and what they do from a high-level standpoint next.

 

PRE-REQUISITE INFORMATION

Let's cover some pre-requisite information on how Malware Detection works within Veeam.

• First, Malware Detection is managed by the Veeam Data Analyzer service which runs on the VBR Server. It restarts once a day, at 12:00am. At midnight, a new Malware Detection session starts, performing the following actions
  1. Checks updates for the list of known suspicious files and extensions
  2. Sends email notification about all Malware Detection events created within the last 24hrs
  3. Initiates a scan session using a specific Malware Detection Method – Inline Entropy analysis scan, File System analysis scan, YARA Rule Scan Backup, A/V Scan Backup, or 3rd Party Malware Protection solution
  4. Each Detection Method investigates a specific potential malware activity. But I'll just cover the 2 main ones, Inline Entropy & File System Analysis. You can read about the others in the Guide:
  > File System Analysis – simply, this engine detects the Malware activity shown below via guest indexing data during a Backup job. For this to function, you must have Guest Indexing enabled in your Jobs. If events are detected, a log is created in a specific location: C:\ProgramData\Veeam\Backup\Malware_Detection_Logs folder

  

  
  > Inline Entropy Analysis – this engine detects the Malware activity shown below during a Backup job via blocks in a data stream from the Backup Proxy; it doesn't require Guest Indexing to be enabled for it to function. If events are detected, no dedicated log is created for this engine, but logging for this scan is placed in a general log, located at: C:\ProgramData\Veeam\Backup\Svc.VeeamDataAnalyzer.log 

  

• If Malware is detected, the Veeam Data Analyzer service:
  1. Creates a Malware Detection event
  2. Marks the machine and restore point where malware activity was detected the first time as Suspicious or Infected. All following restore points created by the Job, and any additional Job (Backup, Copy, etc) which includes the scanned machine, will also be marked as Suspicious or Infected until the machine is Marked as clean.

MALWARE DETECTION MANAGEMENT

There are a few locations you can view and/or manage your Malware Events.

• History node – History > Malware Detection > Malware Events, to view Detail events, a couple of which look like below:
  (NOTE: the data displayed is for Inline Entropy scans; File System scans will give more details)

  

  

   

• Inventory node – Inventory > Malware Detection, to manage the Malware events. You can mark a machine as clean and/or exclude it from future scans if you're sure what Veeam has detected is a false positive
  Rt-click on a machine and select from one of the options listed, or click on it and choose from the Ribbon – Mark as Clean, Show History (takes you to the History node), or Scan Backup (more info on this at the end)

  

  When choosing Mark as clean, you have a couple of choices to decide upon – "mark restore points…as clean" and also to "exclude…from detection". If you only mark restore points detected during the previous backup run as clean, the next run of your job will more than likely flag this machine again as Suspicious because the Veeam Analysis engine will again detect the 'culprit' it found previously. As such, you have a few options – if you're sure your machine is clean (by successful A/V or YARA scans), you can exclude restore points or your machine (this option is not recommended unless the machine is, for example, an appliance machine) from future scans. Be aware, If you choose to exclude restore points and you're unable to find the 'culprit file(s)' and remove them, you'll keep getting notified of infections with susbsequent Backup runs; at the very least, you can disable Malware notifications until you're able to determine the cause of the detection.
   

  

   

• Home node – Home > Backups > DIsk, to perform one of 2 Malware management operations: manage individual VM restore points from the Backup Job properties, or Scan Backup of an individual machine. When would you mark an infected VM's restore point as clean? If, for example, you know the VM, at a certain time, was not infected, yet Veeam detected it as Suspicious or Infected. You could run an A/V or YARA Scan Backup operation against the VM for a given date range to determine when certain restore points marked as infected may actually have been clean.

  

   

SO WHAT DOES THIS ALL MEAN (INVESTIGATION)??

When performing Malware event investigation and forensic analysis of what Veeam has determined to be Suspicious or Infected , at least for Inline Entropy scans, you can run into a bit of frustration; File System Analysis does have its own frustrations as well. I just won't cover what all those are here.

The first place you should look when notified of a Malware event is at the Details event window in the History node. For Inline scans, this lacks needed details, and there may not be any way around this because of how Inline scanning works – they run at the block level. But, according to Veeam Product Management, this is going to be looked into to see what can be done to provide more information for what is detected.

Additionally, unlike File System Analysis, Inline Entropy scans lack dedicated logging. There is a log file you can peruse to see if anything definitive stands out on what may have been detected (location shared above). But, the log is more of a general log for the Data Analyzer service and not solely for Inline scans. And, in the few machine investigations I've done, this log sadly shares much the same vague information as shown in the Details window from the History node. So, determining what is or isn't malicious is difficult.

So how does one proceed with the flagged Veeam Malware events? Keep in mind, Veeam's Malware Detection is not an isolated tool to detect potentially malicous vectors on your machines. Rather, it's a complimentary tool to other security systems and protocols you should already have in place. As such, regardless of what Veeam deems as potentially malicious, you should always perform forensics on your machines by running A/V and YARA scans. You then base your Malware management and configuration decisions based off your visual inspections and scan results.

 

LIMITATIONS (CAVEATS) TO VEEAM'S MALWARE DETECTION

As mentioned earlier, Veeam's Malware Detection does have some limitations and caveats to be aware of. I'll discuss some of them below:

• The Scan Backup operation currently can only be run against Windows machines. To be fair, this is no different than running Secure Restore. Still, I would've liked to have seen a Linux version of this available even in the 1st iteration of the Malware Detection feature. According to Veeam Product Managers, a Linux scan option should be available in a future VBR release or update
• Inline Entropy scan support – this engine can only be run on machines with Simple Volumes, and which use NTFS or ext2,3,4 filesystems. This may be a limitiation for organizations which use storage saving filesytems like ReFS and XFS (think file servers). Dynamic Volumes or disks using encrypted Bitlocker are also not supported. Additionally, text "artifacts" are only detected if they: have a 4KB block size, have the UTF-8 encoding, and it is not stored in the Master File Table. Sleeping malware detection is also not supported. On Linux, some packages with LZMA compression, files encrypted with Windows EFS, some ISOs, etc may be unintentionally marked as Suspicious during Inline scans. This may be a reason for receiving the "encryption data" event as shown above. If you do, you can mark such Malware events as false-positive
• Details of Malware events can only be seen for File System Analysis scans (e.g. file extensions and/or paths & directories). Inline Entropy Details are vague, as shown in the screenshots above. This is because the Inline Entropy Analysis scan analyzes block-level binary data in flight while being transferred by the Backup Proxy, so it's (currently) not possible to determine file names, extensions, or the location of events detected by Inline Entropy Analysis scan types -> Encrypted Data, Ransomware Notes, or Onion Links . This makes narrowing down the cause of malicious events difficult. Veeam PMs state they are also looking into this behavior to determine if more detail can be shared in events and logs to assist with forensics
• Logging – as I shared above, though there is a log file which can be viewed for the Inline Entropy Analysis scan, it's as vague as the Details window. In my opinion, a dedicated log file for Inline Entropy scans should be created and, if possible, provide more detail with regard to Malware events. This, in conjunction with manual forensics, can assist with proper Malware management and configuration decisions on machines which have been flagged by Veeam as potentially malicious
• This one is a bit hidden, but YARA scans are not available for Veeam Data Platform Editions below Advanced Edition. It doesn't matter if you have socket or VUL-based Enterprise Plus licensing, your Veeam Edition needs to be either Advanced or Premium to perform YARA scans. A/V scans are available for all Editions though. Personally, I don't like this stipulation for YARA. Maybe just make it an Ent+ offering. What are your thoughts on this?
• I don't know if it can be viewable for investigative purposes, but, according to the User Guide for Inline scans, Veeam places a RIDX-formatted file in a temporary folder (no path is given) on the Backup Proxy,. I think it would be relevant to 1. know the path location, and 2. if this file can be viewed, to then be able to use it for Malware investigative purposes. But, if it's just a hash-based file created for Malware comparison and determination, then it would be of no value
• If you mark restore points as clean, currently future restore points are not marked as clean for the same file or entity detection. This then leads to continual Malware Detection events for the same 'culprit' file(s). What should happen is, for a given event, if you mark a restore point as clean, future Backups should not invoke a Malware event for the same 'culprit'. If future scans do prompt an event, it should be for a different cause. Thankfully, I think Veeam PMs have also indicated this behavior will be changed in a future release

POSTS TO MONITOR

This is Veeam's first iteration of its Malware Detection engine. It's expected to bring about some lack of features, unexpected and/or unwanted results at times, and even some headaches. That being said, I still think it brings way more value than negativity to an organization's Backup/DR environment, and it will only get better.

In this last section I wanted to bring to everyone's attention a couple of Forum posts from which I garnered some of the information in this blog. You may have noticed I stated possible feature enhancements and changes in future releases. The below Forum posts are where I got this information from, as shared by Veeam Product Manager comments in the posts. The first post shared below is mostly informational for Inline Entropy scan operation, whereas the second post discusses Malware Detection as a whole, and specifically the File System scan engine and some features users feels it lacks..rightfully so. The 2nd post is also a place where many future feature enhancements I think will come from. So, if you've been playing around with Veeam's Malware Detection lately, and have some valuable ideas you feel would make it better, I highly recommend providing your suggestions in this 2nd post. I also recommend keeping an eye on it in general in case any relevant configuration or modification information is shared. It seems to be updated quite often from Anton and Dimitry

> Malware Detection Ransomware Note Found
> Veeam v12.1 Suspicious Files

There you have it…a discussion of my experience so far in using Veeam's Malware Detection feature. I'd be curious to hear your experience of it so far in the comments below.