At the beginning of the year, we faced a critical scenario with one of our clients: a ransomware attack directly compromised the company's main file server. The impact on operations was immediate, as multiple departments depended on these spreadsheets and shared files in real time to keep daily processes running.
When a crisis of this magnitude happens, the initial instinct of any technical team is to start restoring data as quickly as possible. However, without organization, the work ends up being duplicated, files may become recontaminated, and the time until full recovery only increases. I have been through this before.
In this specific scenario, we had two teams working together: our team and the client's internal team. To avoid conflicts in priorities or misaligned actions between the teams that could compromise the restoration window, incident governance was our first step before executing any command.
Below, I share the practical workflow that we aligned on to organize the situation, calm the client, and ensure an efficient RTO/RPO.
The step-by-step governance and operational process was as follows:
Crisis Organization and Alignment
We created a crisis room to centralize all communication and actions carried out by our team together with the client's team.
Role Definition: Clearly defining who decides priorities, who validates the data, and who executes the procedures in the environment.
Attack Vector Mapping: Quick identification alongside the security team of the source of the intrusion in order to isolate the entry point and contain new risks.
Immediate Actions in the VBR Environment
Pause Backup and Retention Jobs: We temporarily froze schedules and retention policies to avoid contamination of new restore points or accidental overwriting of healthy data until the restore process was completed.
Validation of a Safe Destination: Together with the security team, we defined an isolated and clean location to perform the restores, avoiding returning files to a repository that was still considered suspicious.
Restoration Strategy and Continuous Validation
We aligned directly with the client on which folders were critical to operations and started with those.
We executed the restore in short blocks/checkpoints, validating each stage and sharing progress through a spreadsheet within the crisis room. This avoids wasting time if any batch needs to be reassessed, ensuring the best possible RTO and RPO.
We configured an isolated backup routine for newly restored data. This ensured the integrity of older restore points while the environment was still undergoing stability testing.
Delivery and Closure
We used the incident window to review essential security features in VBR that could have been pending, such as MFA and immutable repositories. (This point needs to be checked regularly before an attack occurs, as it can be crucial during critical moments and may have a major impact if the attack reaches the backup environment.)
Documentation and Handover
Creation of the incident report, formal validation of the data with the client, sending all actions taken via email, and normalization of the regular backup jobs.
Conclusion and Lesson Learned
From my point of view, I believe that during an attack, governance is what transforms chaos into a structured and predictable process. More than simply restoring files, establishing control of the situation brings the confidence that the client's leadership needs to see during a crisis.
For SysAdmin Day Contest
