Deep dive Inline Malware Detection


Userlevel 7
Badge +8

First of all : Wishing you a happy new year with the hope that you will have many blessings in the year to come.

 

1. Objective

The objective of this deep dive is to detail the steps to follow to activate the Inline Malwares Detection feature.

2. Introduction

2.1 Description of Inline Malwares Detection

Veeam Backup & Replication 12.1 introduces a new security feature called “Encryption Detection”. It is an online entropy analysis that uses artificial intelligence and machine learning (AI/ML) to detect previously unencrypted data that becomes encrypted in processed disk images.

This allows for the detection of a potential ransomware attack. “Inline Entropy” analyzes data on the fly during backups.

Veeam is not intended to replace detection tools (EDR/XDR, Antivirus) that must be installed and configured on production environments, but rather complements them to facilitate forensic analysis and determine healthy backup points.

Detecting data encryption as quickly as possible is a key element in your protection against ransomware. Online entropy analysis as well as suspicious activity detection features are of great help in combating ransomware and ensuring that you are not backing up systems that are already partially corrupted.

2.1.1 Malware Detection Events

  • Files encrypted by malware: Online analysis triggers a malware detection event if the encrypted data exceeds predefined analysis sensitivity limits.

Text artifacts created by malware:
- Detection of V3 onion addresses (56 symbols in the format [a-z2-7]{56}.onion) and ransomware notes triggers a malware detection event, providing a proactive defense mechanism against potential threats.
- A malware detection event will be created if at least one ransomware note is found (Clop, Medusa)

3. Activation and Configuration

Note: An Enterprise Plus or Veeam Universal License (VUL) is required.

By default, online analysis is disabled in Veeam Backup & Replication 12.1, given its potential resource consumption. Users wishing to take advantage of this feature should evaluate their system resources accordingly. 25 to 30% additional CPU load per proxy.

3.1 Support Compatibility and Limitations

This feature can be used to analyze backups of: 

  • VMware virtual machines, including VMware Cloud Director virtual machines
  • Hyper-V virtual machines 
  • Machines with Veeam Agent for Microsoft Windows in managed mode (volume-level backup only)

Supported file systems include NTFS, ext4, ext3, and ext2.

Text artifacts are detected only if the following conditions are met: 

  • The file system block size is 4 KB.
  • The text file size is not less than 4 KB.
  • The text file has UTF-8 encoding.
  • The text file is not stored in the Master File Table (MFT).
  • Dynamic disks and BitLocker-encrypted disks are not supported.

3.2 How it Works?
 

  1. Data analysis during backup: Veeam Backup & Replication analyzes metadata of data blocks during backup task, recording ransomware data in a temporary folder on the backup proxy.
  2. Data Storage: A RIDX file is created for each disk and contains the following information: o Disk metadata (disk name, creation time, disk size, used size, sector size, partition table)
    o Ransomware data for each data block (amount of encrypted data, abnormal file type magic numbers, onion addresses, and ransomware notes)
  3. Finalizing the backup: After the backup task, ransomware data is transferred to the VBRCatalog folder on the backup server. The Veeam Guest Catalog Service informs the Veeam Data Analyzer service of new data requiring analysis.
  4. Launching the analysis: The Veeam Data Analyzer service checks the results of the last analysis in the RansomwareIndexAnalyzeState.xml file located in the VBRCatalog folder and starts a new analysis session. The Veeam Data Analyzer service compares the latest and previous RIDX files and updates the RansomwareIndexAnalyzeState.xml file. If malware activity is detected, the service will create a malware detection event and mark the objects as suspicious. Note: Malware metadata in the VBR catalog is relatively small. Less than 1 MB for 1 million files.

3.3 Inline Scan Configuration

The setup is relatively simple:

  1. In the main menu, select Malware Detection > General
  2. In the Encryption detection field, check the Enable inline entropy analysis box
  3. Specify the analysis sensitivity according to your backup data and infrastructure capabilities. The default value is Normal. There are 5 levels.
  4. Once validated, a pop-up mentions that activation requires a complete read of the disks to establish the reference, the backup window will therefore be larger. This will only be done the first time or when adding a new disk to a machine.

    Attention: activating this feature activates it for all backup jobs. In case of an important production environment and to avoid an overload due to a complete read of the disks, it is possible to exclude machines from the scan.

    Workaround: add all machines to malware exclusions and remove (a few) hundreds per day.

     

  5. In Powershell

    • Add-VBRMalwareDetectionExclusion

    • Get-VBRMalwareDetectionExclusion

    • Set-VBRMalwareDetectionExclusion

    • Remove-VBRMalwareDetectionExclusion

3.4 Results

Once activated, a new line appears in the backup job logs:

During the next backup, only the catalog is sent for comparison.

To test the detection, we will encrypt a backed-up VM (using "Wanacry").

Once the VM is encrypted, launch a new backup. The catalog data is compared and Veeam raises a warning.

Once the backup is complete, if notifications are configured, an email is sent, otherwise you can find the information at several levels:

In the “Malware Events” view in the history

You can obtain details on the event

In Inventory, in the “Malware Detection” tab, you have the complete list of VMs with an alert.

In the Backups view, a job with a potentially infected machine is easily identifiable using a logo.
 

If you look at the details, you can see all the backup points that have an anomaly or not.

4. Manage the status of malware alarms

4.1 Mark machine as “clean”

All machines marked by malware detection as suspicious or infected can be found in the Inventory view in the Malware Detection tab. If you have cleaned the machine of malware or if the malware detection event was a false positive, you can mark the machine as clean.

Specify a reason. In case of false positives, it is possible to mark all the points in the chain as clean by checking the box “Mark restore points affected by corresponding detection events as clean”.

It is also possible to exclude the machine from future scans if necessary by checking “Exclude the workload from malware detection”. The machines will be added to the exclusion list.

4.2 Managing Malware Status of Specific Restore Points

To manage the state for specific restore points, if you know that a specific machine is infected but the malware detection analysis has not detected any suspicious activity, you can manually modify the malware state of the specific restore point. Go to the “Home” view and then the “Backup” tab. In the properties of a job, select the point in question and right-click or use the drop-down menu with the “Mark as infected” option.
 

Conversely, if a point has been marked but it is a false positive, it is possible to mark it as clean by choosing “Mark as clean”
 

 


22 comments

Userlevel 7
Badge +21

Really liking this feature of 12.1.  Still planning our rollout but cannot wait to see how this works once we hit 12.1.

Userlevel 7
Badge +19

Nice, detailed writeup @Stabz ! I learned something new today..wasn’t aware (or forgot?) the ‘Mark as Infected’ option for a restore point. Good stuff!

Userlevel 7
Badge +6

Great demonstration, @Stabz! Congrats and thanks for sharing. 👏🏻

Userlevel 7
Badge +8

I’m getting more and more excited to install this in production. Hopefully not too much longer.

I’m really curious to see the CPU consumption increase when it’s doing huge scans on my servers. 

Userlevel 7
Badge +19

I’m looking forward to this as well. I’ll probably get it installed before the end of the month.

Userlevel 1

Hi there,

Any updates on how hard this hits your VBR / proxy?  We’re hearing from our SA / account manager that this takes a LOT of CPU for inline scan? 

Jonathan

Userlevel 7
Badge +19

@jonathan.storey - for best response, I’d ping PMs on the Forums. I’m sure this does take a lot of CPU, similar to virus scans. But, for a more detailed response on what is happening, best to ping PMs I think.

Userlevel 7
Badge +21

Hi there,

Any updates on how hard this hits your VBR / proxy?  We’re hearing from our SA / account manager that this takes a LOT of CPU for inline scan? 

Jonathan

I have only tested in my lab, and it is not too bad with the setting on Medium and not Extreme.  I need to wait until we start rolling it out to our Production servers.

Userlevel 7
Badge +14

@jonathan.storey It will depend on your environment, amount of VMs/data, proxies, etc. But in general you could plan with 20-30% CPU increase initially. For testing you can exclude all VMs from malware detection as described in the blog post above, and only enable it for some of them to see the performance impact.

Userlevel 7
Badge +9

This is a great piece... Cheers @Stabz

Userlevel 7
Badge +12

Awesome post @Stabz, will look into detail in the next weeks! Thx for sharing.

Userlevel 7
Badge +6

That’s interesting, thank you @Stabz 

@Stabz Does section 3.2 also apply to the Veeam O365?

Userlevel 7
Badge +21

@Stabz Does section 3.2 also apply to the Veeam O365?

No, it is not part of the VB365 product as yet.

Userlevel 7
Badge +8

This is a great write up. It’s a good reminder I’ll have to be slow about implementing some of these features as resetting CBT on my severs would take days to backup.  

 

Does it do an active full, or just require reading the full disk while it does an incremental? I’ll have to test that too as I wouldn’t want it to replicate 100’s of TB to other sites 

I enabled this feature but the .RIDX files are overloading my B&R server. For 1 of my VMs in particular the .RIDX files are 2.5GB for each day’s restore point for just this 1 VM. A fairly unremarkable 20TB windows file server.

How many days of this data does Veeam keep on the B&R server? 

Userlevel 4
Badge +1

Great writeup.  I want to emphasize the section @Stabz has on the “Malware Exclusion” list under “Global Exclusions”.  I’ve already heard from several customers that they do not want to enable inline scanning because they think it will tank their environement on that first full scan of the workloads as VBR gathers baseline data.  Exclude all of your workloads and then allow the scan to run on a couple to determine the proxy resources needed.  We show 20%-30% increase in CPU utlization on the proxy during that initial run, but results may vary.  This is a remarkable feature that every customer can take advantage of.

Userlevel 4
Badge +1

Also, to note, the analysis sensitivity setting does not impact performance.  This setting is a sensitivity setting.  You would adjust this setting, depending on the amount of encrypted data you have in your environment, to avoid false positives.

Userlevel 7
Badge +8

 Thank everyone for your answers and feedback!

In my lab I didn’t notice a big change in the CPU consumption but I have only few VMs, I need to compare that to a real production infrastructure.

 @Scott It’s not an active full, Veeam read the whole disk for create the baseline of metadatas.

I enabled this feature but the .RIDX files are overloading my B&R server. For 1 of my VMs in particular the .RIDX files are 2.5GB for each day’s restore point for just this 1 VM. A fairly unremarkable 20TB windows file server.

How many days of this data does Veeam keep on the B&R server? 

​The RIDX are saved on the proxy server in a temporary folder, for me this files should be deleted once the data transferred to the VBRCatalog. @Mildur r  have you more informations about this?

This is one of those writeups that while shows how to enable it (which is fairly easy)  but doesn’t show you how to get any actionable file level detail once you have a detection.    

I have a detection that I think is a false positive, but I can find no way to have Veeam tell me what file is being flagged.  As in the above examples there isn’t any detail beyond a server reference that has 1000’s of files on it.     

Userlevel 7
Badge +8

@BobLeg you could maybe find some informations here :https://community.veeam.com/cyber%2Dsecurity%2Dspace%2D95/malware%2Ddetection%2Dinline%2Dentropy%2Dscan%2Da%2Ddeeper%2Ddive%2D6820

The Inline Entropy module remains quite vague in terms of information and requires further investigation.
If you want more detail you could use the guest indexing option.

Userlevel 7
Badge +19

This is one of those writeups that while shows how to enable it (which is fairly easy)  but doesn’t show you how to get any actionable file level detail once you have a detection.    

I have a detection that I think is a false positive, but I can find no way to have Veeam tell me what file is being flagged.  As in the above examples there isn’t any detail beyond a server reference that has 1000’s of files on it.     

Bob - there is a Forum post regarding the Inline Entropy Scan engine, and it’s lack of visibility for detections. It appears there is an update on the way to help show either file and/or location (or both 🙏🏻 ). You can follow the post from the link below. If you have any positive feedback you’d like to share/offer, I recommend doing so there as well.

https://forums.veeam.com/veeam-backup-replication-f2/malware-detection-ransomware-notice-found-t91360.html

@Stabz - thanks for the share 😊

Comment