We follow veeam AV KB article for the exclusions.

But thinking about the latest attack, it would be good to use a standalone anti virus software and not something centrally managed.

https://www.veeam.com/kb1999

We just use Windows Defender. No exclusions. Previous job we used Symantec then changed to Kaspersky..and used exclusions as issuggested by Veeam. 

We normally follow the vendor guidelines and configure the exclusions; same goes for Veeam. But I always have a bad feeling that someone could utilize those exclusions for bad stuff. I don’t have a problem to exclude certain file endings, but process names or directories are pretty open for anything.

The story with Kaseya is really bad and can cost them much of their reputation and trust. Although nothing is 100% safe, such attacks should never be possible in the first place.

Same here even with Windows Defender.

Hardening OS, for microsoft :

https://www.microsoft.com/en-us/download/details.aspx?id=55319

AV then EDR agent on it, at the moment both we have specific rules on backup infra.

Use the same kB as @Mildur 

Normally I recommend the installation of AV software on Backup components and to follow @Mildurs link. 

Would such an attack not work like this if AV would be configured without exceptions?

Thanks everyone for the responses so far. The attack information is still slowly being released but it appears this was a supply chain attack and I’ve heard conflicting reports that this was a zero day on the Kaseya VSA and that the criminals had pushed a malicious update to them. If it’s the latter and Veeam suffered from such a breach then by trusting the application we’d open ourselves up to vulnerabilities.

It’s worth highlighting that at present Veeam don’t deploy updates remotely and automatically but if malicious code was injected into an update, signed by Veeam and then made available within one of their CUs we’d all be vulnerable because we tend to trust the Veeam executables due to the requirements of KB1999.

We have AV Software on our Veeam servers, mostly without exceptions.

So far we had problems with the AV software at update time only. So, we pause the AV software while the update package is unpacking itself and activate it right after that again.

OK, if there is malware injected in a Veeam Update, we would have a problem….

I'm not sure if an AV could prevent such an attack. At least a traditional anti-virus wouldn't be able to detect the malicious code inside an update as it would be an unknown threat. And probably no security solution could block a zero day exploit where an attacker could try/error everything until he/she has full control.

Agree @regnor. I’m pretty sure A/V would not prevent ransomware of any kind, except those which have been around a couple years. New?...definitely not. I’ve not heard of any case where it has, being they are generally newly written code aimed towards a specific vulenerability which A/V solutions tend to be unaware of.

There are solutions such as Sophos Intercept X that use machine learning to monitor entire process chains and file touches to look for malicious activity but it’s interesting how quickly these definitions are updated. In the event of a zero day, sure you’re relying on ML, but signatures could also quickly highlight a dormant threat, until we exclude the processes entirely.

This is of course purely theoretical and going on a slight detour to my original intent, and that is, viruses hitting your Veeam backups, typically something targeting the folders won’t be noticed as they’re set to be excluded, which increases risk, but we also don’t want anti virus messing with Veeam’s job. I’d like to see the exclusions reduced so we just trust the Veeam processes specifically and then any other code touching Veeam repos gets scanned for malicious intent.

Agree with @regnor too, legacy AV software will not prevent ransomware attacks - probably not even old ones. You would need next gen AV software, @MicoolPaul’s Sophos Intercept X probably is one of these. With such a AI/ML software, I think, you would not define any exclusions. 

That’s where Sophos Intercept X is interesting, it’s actually a hybrid, there’ve been examples of AI/ML software being tricked as they tend to use a reputation scoring system (example of Cylance being tricked here). Sophos work around this by utilising both their traditional signature driven engine combined with ML and related technologies to create a more complete security offering. Whilst I’m talking about Sophos, they also have a technology called “security heartbeat” whereby the agent can deny connections from devices that are missing the agent or in a poor health state, and their firewall can cut off outbound communications based on the same criteria.

I promise I’m not sales pitching for Sophos here, I just implement their stuff a lot and think these features are great in the battle against these kinds of evolving threats.

sounds promising! Although I am glad that this is not my field of application 

@vNote42 agreed, Infosec is difficult but it’s certainly a shared responsibility since it impacts us all!

I think we’re going to see more of a push towards application whitelisting as this continues. It’d be great to see Veeam’s push towards hardened repositories extend to validation against security templates (definitely a wish list item but unlikely to happen!)

I agree that a more modern approach is needed because the traditional signature based way isn't working anymore (or better since long long time). Many years ago I went with Webroot; they were completely based on behavioral analysis combined with  whitelisting/blacklisting if files. Although this way also has it's disadvantages and Webroot lacked of some enterprise features, it worked very well at that time.

The question is; when someone is actively attacking you, will any kind of Anti-virus be able to prevent a successful attack? If the attackers has enough time, then he can just try anything until he's able to enter the system. I think with a remote execution or privilege escalation, you'll always lose. Then it's rather about how secure are your other systems and how well planed is your network, security and authentication.

There’s certainly more bad actors out there than known zero days and unless it’s part of a wide campaign, most small SMBs won’t have the misfortune of dealing with a zero day (emphasis on most, it’s still not impossible).

We certainly need to evolve our defence in depth, @BertrandFR highlighted the Microsoft hardening baselines and other security organisations have their own recommendations as well, for example in the U.K. we have the “National Cyber Security Centre” (NCSC) who offer their own baselines to complement Microsoft’s own. This is great for reducing the attack surface especially when we see new threats such as “PrintNightmare”, it’s worth highlighting this to everyone now, the print spoiler service is enabled by default but there’s an unpatched vulnerability that allows you to run code as SYSTEM, which again undermines most security software due to the level of privilege it’s running at.

I mentioned earlier about application whitelisting and if your AV doesn’t support whitelisting (another Sophos highlight here, they have a server lockdown mode whereby once enabled no new applications are allowed to run), alternatives such as Microsoft’s AppLocker can be used to achieve the same goal.

We certainly need to remain vigilant against these types of threats and continue to restrict what code is able to run on our systems as this could undermine immutability (and another good reason to use third party immutability services to avoid a common vulnerability set being able to compromise all backups).

Anyone that saw the word from Gostev this week would have seen him talking about this subject too, I was actually involved in the same Reddit conversation he was referring to and a user was adamant they would be protected by the immutability protections provided by their vendor and we had to start explaining, those immutability protections are assuming the person trying to delete the immutable data is playing by the rules and hasn’t got a zero day or unpatched vulnerability.

use sep with exclusions.

Regarding the “PrintNightmare” I applied the GPO on a server where there is no print server or print-related services see Xenapp etc ..

As per CVE-2021-34527 | Windows Print Spooler Remote Code

Disable inbound remote printing through Group Policy

You can also configure the settings via Group Policy as follows:

Computer Configuration / Administrative Templates / Printers

Disable the “Allow Print Spooler to accept client connections:” policy to block remote attacks.

You must restart the Print Spooler service for the group policy to take effect.

Impact of workaround This policy will block the remote attack vector by preventing inbound remote printing operations. The system will no longer function as a print server, but local printing to a directly attached device will still be possible.

 

 

Hi @MicoolPaul , great post! Thx for sharing. We are installing always AV (several brands : Windows Defender - BitDefender - SEP - Kaspersky - ...), decided by the customer) on all Veeam components. As @Mildur already mentioned, we are using the exclusions described by Veeam : https://www.veeam.com/kb1999

July 6, 2021-KB5004947 (OS Build 17763.2029) Out-of-band (microsoft.com)

Microsoft Update Catalog

https://msrc-blog.microsoft.com/2021/07/06/out-of-band-oob-security-update-available-for-cve-2021-34527/

With the patch comes a feature (which can be enabled or not) to further secure the installation of new printer drivers (KB5005010)

This is a big one, good post @MicoolPaul 

A few other points - I love this idea as a reminder to use the Data Integration API after backups are taken to do scans. Scan with the latest updates, no exclusions and even different (more aggressive) tools. This is for the systems Veeam is backing up. I want to evangelize more of this, but this was a nice reminder.

We have running AV on all servers including Veeam. We followed the KB article but we regularly see errors regarding directories on VBR server that can't be removed. At this time AV is scanning the directory. 

Error: boost::filesystem::remove: The directory is not empty: "C:\windows\TEMP\{ffecfddb-e365-42ac-806e-58c1b974b14d}\" Agent failed to process method {FileSystem.DirectoryRemove}.

Excluding C:\windows\temp is not a good idea and the subdirectories are all more or less random. Would be nice if this directory would be configurable. 

@MicoolPaul : Excellent post buddy !