31.3 is “Backup Day” Why?
Because next day is first of April? 😊 So, tidying up could resolve into unwanted deletion of your data, which could lead into surprise, after you find out that and this is not just as a joke😊It’s nightmare …
In nowadays we are trying to keep our platforms, environments most resilient as possible from ransomware attacks.
BUT
What is worse than ransomware attack?
Are you really prepared to cover all scenarios?
Sometimes it could be even bug of your nightmare in vendors software/firmware, which is not the thing that you never think about.
AND
Combination with ransomware attack? Admins have no clue what is correct.
What happened?
Customer found out that has infected VMs – all Windows and Linux/Unix Systems. Production was stopped. It did not take longer, and he discovered that ALL VMs are affected.
NOW was time to find out, when it happened.
They figured out that best date and time to restore them is 7 days back. It was disaster for customer to lose last seven days.
Backup environment was not affected. Multiple authentications to backup servers. Limited number of accounts and their access. Micro segmentation in network.
So backup is SAFE! That was good news.
We have calculated that with speed 300MBps per vmdk in 128 tasks it will be half day. So, let’s start!
We have restored VM and after 3 minutes it is gone!
What happened?
Is hypervisor affected? Are backups affected?
Trying to understand situation, vendor was involved.
Vendor of hypervisor discovered that there is bug on datastore.
Suddenly everybody stopped to be nervous. Ok, root cause is discovered.
First question: “When will be fix ready?”
“We can provide today as is known bug” was told from vendor.
Everybody was happy.
At the end of business day fix was applied and restore test was successful.
Right after that we started restoring.
In the morning, we provided to customer completed environment back. “So quick? It’s fantastic! We thought that it will take another week! You are our angles!”
All's well that ends well.
