I work already more than 6 years at my current company as a system consultant specialized in backup, DR, hypervisors and storage but at the first place being the Veeam Expert .
In those 6 years I have seen a lot, but one of the things I will never forget, is a story that I want to share with you .
When :
It happened more than 2 years ago
Who :
It happened to a customer of us, a secondary school
What happened :
They were using multiple standalone Hyper-V hosts with local storage.
They all were using RAID5 on the data-volume (where the .VHDX files = HyperV virtual hard-disks are located) of their hyper-v hosts
As you know : in a RAID5 setup, only 1 disk may fail...
On their most critical Hyper-V host 2 disks were stuck, so the volume was broken and all data was lost on that hyper-v server!
On that server a couple of virtual machines were running including one of their most important server : their big file-server!
That was not the only issue : the hyper-v host was not in warranty anymore , they decided 6 months ago not to extend this warranty anymore for this server...
Analysis :
Because of the urgency and criticality, the service desk called me to ask if I could look into this problem and find a quick solution for this problem.
I first checked the backups of the VMs that were running on that hyper-v host : no problem, luckily they ran successfully the evening before.
No problem I thought, I will do a full recovery for the less critical (bigger RTO) virtual machines on the storage of the other hyper-v hosts and perform an instant recovery (low RTO) for the most critical virtual machines
Problem :
The most critical virtual machine that had to be restored as quickly as possible was their big file-server (more than 6TB)…, the problem was that none of the other hyper-v servers was having enough storage to provide this file-server, so instant recovery is the solution, but with hyper-v (with that version of Veeam - version the whole storage must be provisioned on the hyper-v host (one of the biggest disadvantages against vSphere where you can run the instant recovery on your backup-server without provisioning on your vSphere ESX-host at that time) even if you are running the virtual disk on the backup-server...
No other storage was available
Solution :
Ok I discussed this with the customer and told them that I had an out-of-the-box (TEMPORARY) solution for this problem .
I asked them if they had a large USB-disk…
Yes they had
I told them to connect it on the hyper-v host and told them that their file-server will come online in less than 15 minutes, slow but it will work!
They were very surprised I could fix this
And it did !!!
After less than 15 minutes their file-server was online running on the backup-server with provisioning the storage on the USB-disk and performed in the meanwhile a sync in the background. After several hours all data was transferred to the USB-disk and the file-server was running on the USB-disk.
I told the customer that they had to order as quickly as possible a new server or new hard-disks for their server because that USB-disk won’t live long running like that, it was clearly a temporary solution
The answer of the customer : they will do that urgently and thanked me a lot and told that I was a wizard to bring their file-server back online in that short amount of time
After 1 week :
After a week my colleagues called me to ask if I would contact that customer again, the customer asked specifically for me
I called the customer back and do you want to know what they asked me ???
If I would do it again what I did before because the USB-disk was broken . They already bought a new USB-disk in the shop and connected it to the server and ordered that day a brand new server...
Conclusion :