Key Takeaways from the El Segundo Zero Day Resilience Workshop
Cyber resilience is not proven with a static Disaster Recovery plan. It is tested in advance through training and practice. It is proven in the moments when operations fail and an organization can respond with speed, clarity, and confidence.
I recently had the opportunity to help host a Veeam Zero Day Resilience Workshop in El Segundo, California, where IT and security leaders from education, health, manufacturing, marketing, and public services came together to share how they would respond to a fast-moving cyberattack. The viewpoints from CISOs, CTOs, security analysts, system administrators, IT directors, and more enabled valuable discussions from many angles on how to prepare in advance to successfully counter a cyberattack.
If a Veeam-sponsored security workshop is available in your area, I would highly recommend attending for the insight provided. Here are the highlights and key takeaways.
Workshop Keynote Highlights
- 69% of organizations were attacked in the past year
- 89% of organizations had their backup repositories targeted
- 24 hours on average for threat actors to gain access, exfiltrate data, and demand a ransom
- 22 days on average for an organization to be recovered to a state of Minimum Viable Business (MVB)
The speed of impact is outpacing the speed of recovery. While some degree of business disruption is expected during a cyberattack it must be minimized. The workshop highlighted the need not only to equip your organization to prevent attacks, but also to equip your organization to contain threats, communicate effectively during attacks, and recover successfully when an attack inevitably happens.
After the keynote, the workshop focused on tabletop discussions of a scenario with three phases.
Scenario Phases
- The discovery phase. The scenario outlined several indicators of potential malicious activity happening in an organization. Example indictors were elevated outbound network traffic to an unknown IP, suspicious PowerShell commands, and newly created accounts in Active Directory.
- The next phase outlined that operations had been interrupted. Several VMs had become unresponsive and file shares were inaccessible. A ransom note had also been discovered…
- The final phase focused on recovery paths and key considerations for each recovery option.
Scenario Phase 1 Takeaways
The first scenario began by introducing indicators that could be mistaken for normal IT activity. The workshop participants at each table discussed what tools/information could be used to correlate these events to more accurately determine if the activity was normal or malicious. If it was determined to be malicious, what steps would be taken to immediately contain the breach? Workshop participants had varying approaches with some as drastic as immediately cutting off all network access to all systems while others had a more surgical approach, choosing instead to contain affected systems and striving to keep as many of the critical business systems operational.
The delta in approach highlighted that organizations have different requirements and the importance of categorizing business systems by criticality. This was especially true of organizations such as healthcare where downtime can have a severe impact on the treatment and care healthcare professionals can provide.
A final topic of discussion for Phase 1 revolved around communication of the incident both internally and externally.
From the table discussions these points emerged:
- Establish “normal” behavioral baselines in your environment so anomalies are easier to identify
- Correlate alerts across identity, endpoint, network, and infrastructure using SIEM/SOAR tools
- Contain compromised user and administrative accounts quickly
- Strive to preserve logs and data that may be needed for forensic investigation
- Empower leaders to make time-sensitive decisions
Scenario Phase 2 Takeaways
In Phase 2 of the scenario, high CPU and disk utilization alerts appeared across multiple virtualization hosts late on a Friday afternoon (Threat actors often take advantage of weekends and holidays when fewer staff are available to counter/detect an attack). Snapshot files were modified, security agents were unexpectedly disabled, virtual machines became unresponsive, and shared drives were inaccessible. Late Friday evening is when the ransom note was discovered.
Discussions shifted from detection to containment. Tables were debating how to assess the blast radius, how to contain the threat while prioritizing availability of critical services, how to communicate with stakeholders, and determine how decisions about a ransom demand would be made.
A notable key point was how critical out-of-band communication is. During a cyber incident, normal email, identity, collaboration, and messaging platforms will likely be unavailable. Or worse, if communication platforms are operational, they might be monitored by the threat actor. Employees and cyber incident responders need a secure, approved, and tested way to communicate that does not rely on the affected production environment.
Workshop participants shared several approaches they practice, including:
- Predefined cell phone call trees
- Approved messaging applications that operate independently of corporate systems
- Separate identity providers that do not depend on the production identity platform
- Printed hard copies and offline versions of incident response plans contact lists
Additional considerations discussed were knowing when to move to out-of-band communication channels, how to authenticate incident responders, where decisions will be documented, and how legal, regulatory, and records-retention requirements will be maintained.
Another topic discussed was how to effectively communicate with stakeholders without being overwhelmed with frequent requests for status updates. One idea discussed involved setting up a dashboard with metrics and objectives that key stakeholders could reference as needed.
Other key lessons discussed:
- Aim to isolate affected systems while preserving evidence and data integrity.
- Aggressively contain both standard and privileged accounts that may be compromised.
- Secure backup infrastructure before an attack to prevent tampering. (See Veeam’s Best Practice Guide on immutability and the 3-2-1-1-0 rule.)
- Identify priority-zero and priority-one services before an incident occurs and determine realistic RPO and RTO.
- Know when to engage legal counsel, cyber insurance providers, incident response specialists, law enforcement, and other external partners.
Scenario Phase 3 Takeaways
The final scenario phase now shifted the focus to recovery with four main paths:
Each option was discussed beyond just a technical perspective. Participants considered the recovery time, cost, data integrity, compliance exposure, downstream dependencies, and the risk of reintroducing compromised systems to production.
At the table I hosted we discussed the concept of “dwell time”, the length of time an attacker remains undetected inside a network. Mandiant’s M-Trends 2025 report put the global median dwell time at 11 days. With that in mind the table discussed how to determine if backup data is safe to restore from and if it might contain tools/resources that the threat actor could use to regain access in the future.
The possibility of paying the ransom to obtain a decryption key was discussed. The table determined that there might be unknown risks from trusting a decryption key from criminals and that it might be illegal to pay a ransom for a key.
In summary, the discussion around Phase 3 reinforced several recovery fundamentals:
- Maintain a documented, business-approved list of recovery priorities
- Map application, identity, infrastructure, and third-party dependencies
- Define who is authorized to initiate recovery
- Verify that backups are clean, complete, and recoverable
- Validate systems before restoring them to production
- Test backup recovery to determine if RTO can be met under realistic conditions
A backup is only as valuable as your ability to restore it securely within the timeframe the business requires. And of course, a backup is not a true backup unless recovery has been tested.
Final Thoughts
During the first phase of the scenario, a theme emerged that carried through every discussion: every organization is different, and there is no one-size-fits-all approach to resilience. Some require immediate and total containment the moment compromise is confirmed because the data they hold could pose a threat to the public if it gets in the wrong hands. Other organizations deliver services such as healthcare and public utilities that need to stay operational while the threat is contained. The lesson is to determine in advance which approach your environment requires.
Strong resilience requires the alignment of people and processes with the proper technology. Before an attack, organizations need to test communication channels, identify critical applications, validate backups, and establish decision authority.
Tabletop exercises like these workshops create a safe environment to expose gaps before attackers do. Practice in advance elevates an incident response plan from a static document into a proven strategy.
Ready to evaluate and strengthen your own approach? Try attending a Veeam Security Workshop to work through realistic scenarios with peers and cybersecurity professionals.
You can also take the Veeam Data and AI Trust Quick Pulse for a fast assessment of your organization’s maturity, a benchmark against industry peers, and an initial view of opportunities to improve resilience and trust.