Hi There!
I've shared this story in my VUG Group, Spain,
But I wanted to translate it to English and share it with all of you,
It’s a Personal experience, the latest I’ve got in Prod before changing my role,
and now re-publishing it for the Sysadmin month.
Hi,
I want to share with you my latest experience as a IT Infrastructure responsible, just a week before moving into a new company, also in a new role / position.
It was August 10th 2022, enjoying my summer holidays before moving into a new company, celebrating my 37th Birthday at the Swimming pool area with my family, and surprise, my phone dings, and I just had a quick look and this came up on the screen:
“We’ve got a Virus in our Systems”
I felt goosebumps all over my body, I left my pint of beer (Yes, I love beer) on the table, and I addressed my wife and told he, “I gotta go” ransom!
I change my clothes, from my comfortable swimming suite to jeans and a t-shirt, went to the office to get a better understanding of what was going on, and execute the necessary actions to mitigate and solve the issue.
Into my head I had this crazy idea:
“It’s just a joke” my colleagues are doing this to scare me and it’s a surprise cause I’m leaving the company, and it’s a bye Joke… I was wrong.
We confirmed that the virus was a ransomware called <DonkeyF*cker> and hit our servers and spread himself like gunpowder into our vms.
After a huge effort, and coordinating all the IT resources at the office, we finally find the origin of the Infection, patient Zero! Of the encryption, we took it offline, formatted that PC with no regret, and then, Time to recover…
“Surprise!!”
Our Veeam Backup Server got hit and the Backup repository was not accessible!
Calm down, clam down, We recovered all our servers!!, Luckily, in our design, our two ESXi Hosts had replicated vms each other, like a “last resort” plan if we’ve lost production backups, replicas and even repos.
After a deep breath, we executed our Recovery plan recovering our environment from our replicas, and after a tense minutes, everything started to run as it was supposed to be, getting more confidence on each vm being recovered.
Such a great feeling!!
After less than 60 minutes, we were fully recovered, everyone working as it was supposed, and the Managers / C level where so pleased that we were able to protect and recover the business that quickly and smoothly.
I’m sharing this as a horror story, but with a positive ending, but when you are in the middle of the “situation”, it’s a Nightmare.
Always have an A, B, C and D plan, Test your backups, and test your DR plans!
Luis.
