I took advantage of my holidays to step away from screens for a while. Unfortunately, the contest has now come to an end but I still wanted to share my experience with the community.
So, what will you find in my toolbox? Well, before answering that, I think it's essential to cover a few other points first because being ready for a disaster is about more than just technology.
Experience Is Your Best Ally
I fully agree with what others have already mentioned: experience is your ally. The more you dig into topics and use your tools on a regular basis, the more comfortable you become and the day a problem hits, you'll be far more confident in how you approach it.
Take the different restore options Veeam provides as an example: it is absolutely essential to test each of them so you understand *when* to use them. Knowing how and when to leverage Instant Recovery, for instance, can save you precious minutes when every second counts.
You Need a Big-Picture View of the Infrastructure
You need a holistic view of the infrastructure: *Compute, Storage, Network, and Backup*.
I've been implementing backup and disaster recovery infrastructures for several years now. That experience has exposed me to a wide variety of environments, each with its own challenges. As a result, even when I don't know a customer's infrastructure inside out, I can find my bearings fairly quickly and rebuild their information system while respecting their constraints.
The Psychological Factor: Managing Emotions in a Crisis
Another point, a psychological one this time is knowing how to manage your emotions. During cyber crises, we've seen key people completely lose their composure, to the point of fainting or worse! If you're prone to stress, I can only recommend training for these kinds of situations: build scenarios and run simulations. This will help you identify the weaknesses in your recovery plan outdated or missing documentation, at-risk individuals, and so on.
That's also why you shouldn't hesitate to surround yourself with trusted partners who can support you and help you get your business back up and running.
The Technical Side: Documentation Is King
Now, on the technical side, it is imperative to keep your technical documentation up to date: architecture diagrams, recovery plans, and quick-reference runbooks. The more information you have at your fingertips, the better off you'll be.
The Crisis Management Document
It's important to establish a document that identifies the crisis management location in the event of a disaster, the role of each member of the crisis/investigation team, and a procedure describing the actions required to fail over to the DR infrastructure or to rebuild from scratch.
Effective crisis management requires setting up an essential body: the crisis cell (crisis management team), which allows you to manage the situation efficiently. Defining this team is a must, and it should include the following information:
- Name, role, and designated backup (alternate)
- Roles and responsibilities
The crisis cell could be composed of:
- Crisis Cell Lead
- Head of Information Systems & Business Continuity (BCP)
- Logistics & Relocation Lead
- Business Operations Lead
The crisis cell drives the channel related to bringing production back online (the restart). It's important to set up a secure communication solution avoid using your company's Teams, which may itself be compromised.
The goal of the crisis cell is to keep the business running in degraded mode, in parallel with the work of the investigation cell (CERT).
The investigation cell handles the forensics channel. Its number-one objective is to determine the date of the intrusion (or the root cause of the outage) and then clear the markers. This work is what allows you to lift any doubt and safely restart the critical machines.
Both cells work in parallel and communicate constantly. One cannot function without the other.
What's Actually in My Toolbox
As an external consultant, here's what I always recommend keeping on hand:
- Veeam ISOs the latest few versions, not just the very latest one.
- Windows Server and Linux ISOs.
- Pre-built templates for rebuilding a Veeam Backup & Replication (VBR) server or proxies. When you have to rebuild from zero, having ready-to-go images saves a huge amount of time getting your restore capabilities back.
One last thing: even in degraded mode, don't overlook the security and hardening layer. Having templates lets you roll out and bring the Veeam infrastructure back up quickly so you can start restoring without cutting corners on security.
Thanks for reading!
