Great question. There’s of course no single answer and it will vary greatly based on what you/your organisation require for data loss protection. As before you even get to your immutability, your production environment and backup infrastructure need to be able to deliver the necessary performance to back up the data to the timescales needed. I also argue that whatever your business says they need for RPO, you should cut it in half, so you have N+1 redundancy on job failures, buys you time to fix the problem before violating your RPO.

You’ve got different layers you need to protect, immutable storage is great to protect your primary storage from compromise however consider any other threats to the storage such as is there an out of band management option available such as iDRAC or iLO on your server that could be used to trigger a wipe of the array.

Based on this I’d then argue you want an immediate copy to an offsite provider. This could be achieved via S3 compatible storage or a Veeam Cloud Connect Service Provider. Full disclosure I’m a service provider myself but I recommend the service provider route as it’s managed infrastructure that you don’t control. You just pay a bill whereas with S3 you tend to be the administrator of general storage, cloud connect is purpose defined. Which means no insider threat and alarming is possible when RPOs are violated, S3 doesn’t care what’s going to it for contrast. This provides off site data protection and Cloud Connect has insider threat protection to secure you from loss of backups being deleted. Finally agree what a final tier of DR should look like for an RPO when you’re trying to recover and someone took out your production + your service provider bill didn’t get paid. Tape is probably your best bet here, you could ship tape to another supplier to keep it out of your insider threat’s hands too and you’ve gained yet more geo redundancy of your data.

Final comment, encrypt ALL your backups. We’re seeing a shift from ransomware to data extortion as businesses become better prepared to recover from ransomware, what better option to steal a copy of all your data than a pre compressed and deduplicated, unencrypted backup file!

Hi JayST,

In addition to what MicoolPaul wrote, I recommend and implement the following with my customers:

• Have a good detection system.
• Using Veeam ONE alarms is a big plus.
• Next-gen EDR solutions with automated isolation
• Monitoring tools "security oriented"

And the most important part:

• Define an answer protocol, with clear processes and identified responsibilities and trigger points.
• Test it !
• Veeam infrastructure protection is often left out of the customers response protocol, it must be part of it.
• Test it !
• Escalation possibility (Service provider with a SLA contract)
• Test it !
• Having a drilled team change everything.
• Test it !

It's all of the above that will contribute to define if your RPO with immutability is enough.

(test it)

Great answers! Great points.

There are a lot of things to care about regarding protection for the scenario arround data compromise. 

But i'm still struggling to come up with proper immutable retention times with proper arguments. For example, when i try to lookup some consensus about avarage compromise detection times, this could lead to some arguments to provide for at least 14 - 31 days immutable retention.

For example: let's say an organization adds an Veeam immutable hardened repository to his existing Veeam environment, no other plans other than that. There was no such thing before in this environment. Sizing comes up. What sizing procedure would make sense and what arguments whould make sense to suggest an immutable retention period, only caring about this immutable repository. Would 7 days be too short according to the proper arguments? 

I'm think longer immutable retention time (as a standalone measure) to protect against compromise are less of value than one might think. If compromised, detection takes (for example) 3 weeks and what follows is the decision to restore with a RPO of +3 weeks RPO, i think it quickly becomes an option to consider paying the ransom for example. 

So advising the combination of the immutable storage (using the hardened repository) storing the latest backup chain and improving the detection tools seems the best option in my opinion. One without the other is of much lesser value or maybe even (close to) worthless in the real world.

Mhh, I don’t think you can give an answer to this question that is correct for all cases.

The authors of malware are - sadly - not dumb and adjust their “wait” time.

So, if you suspect some malware activity, use the virus detection option within the restore process. This is not a 100% protection, but it is far better than nothing.

Yes i agree. That's why i'd argue 7 days of immutable storage as a standalone measure is not that valuable (due to the mentioned “wait” time), although beter than not having anything immutable. But the same could be said about having 31 days immutable as standalone measure, although increasing the chance to beat “wait times”, but then there are other “concerns” when having to use that old data for restore when compromised. 

I definetly would agree there's no answer like “28 days is best in most cases”. It depends on a lot. But,  the correct questions to ask to see what period does fit best, is an interesting part of the discussion.

Lots of great answers already here! I think XFS repo is a good point to have performance and minimum security from ransomware and side attack. I saw this like a plus not the finale shield from any attack.

Hi @JayST -

In the Vanguard Slack channels, we discuss ransomware every so often (the more discussions, the better). Several of the ‘Vannies’ are SPs and shared it has been determined ransomware/malware files seem to lay dormant up to around 45 days before they activate. So SPs have increased their retention for immutable storage to 60 days. They didn’t share the source of where they learned about the dormancy days for ransomware so I can’t provide it. Point is, as was stated previously, hackers do think about such things and can (& probably do) modify their code accordingly. Having several different layers of recoverability will lessen your chances of not being able to recover any data. 

GFS retention could be a good one for space saving without increasing the retention. Ok you can lose some data points but security and exploitability is always a balance (cost etc...)

I’ve heard of instances where they restored at the file level to just prior to when the payload activated, recovered critical file data, then restored the system to prior to the infection, and updated the cleaned data to bring the system to near current RPO.