SysAdmin Story : a Veeam feature saved a school


Userlevel 7
Badge +11

I work already more than 6 years at my current company as a system consultant specialized in backup, DR, hypervisors and storage but at the first place being the Veeam Expert 🤣.

In those 6 years I have seen a lot, but one of the things I will never forget, is a story that I want to share with you 🤗.

 

When : 

 

It happened more than 2 years ago

 

Who : 

 

It happened to a customer of us, a secondary school

 

What happened : 

 

They were using multiple standalone Hyper-V hosts with local storage.

They all were using RAID5 on the data-volume (where the .VHDX files = HyperV virtual hard-disks are located) of their hyper-v hosts

As you know : in a RAID5 setup, only 1 disk may fail...

On their most critical Hyper-V host 2 disks were stuck, so the volume was broken and all data was lost on that hyper-v server!

On that server a couple of virtual machines were running including one of their most important server : their big file-server!

That was not the only issue : the hyper-v host was not in warranty anymore :disappointed_relieved:, they decided 6 months ago not to extend this warranty anymore for this server...

 

 

Analysis : 

 

Because of the urgency and criticality, the service desk called me to ask if I could look into this problem and find a quick solution for this problem.

I first checked the backups of the VMs that were running on that hyper-v host : no problem, luckily they ran successfully the evening before.

No problem I thought, I will do a full recovery for the less critical (bigger RTO) virtual machines on the storage of the other hyper-v hosts and perform an instant recovery (low RTO) for the most critical virtual machines :wink:

 

Problem : 

 

The most critical virtual machine that had to be restored as quickly as possible was their big file-server (more than 6TB)…, the problem was that none of the other hyper-v servers was having enough storage to provide this file-server, so instant recovery is the solution, but with hyper-v (with that version of Veeam - version 8) the whole storage must be provisioned on the hyper-v host (one of the biggest disadvantages against vSphere where you can run the instant recovery on your backup-server without provisioning on your vSphere ESX-host at that time) even if you are running the virtual disk on the backup-server...

No other storage was available 🤔

 

Solution : 

 

Ok I discussed this with the customer and told them that I had an out-of-the-box (TEMPORARY) solution for this problem :wink::blush: .

I asked them if they had a large USB-disk…

Yes they had 🤞

I told them to connect it on the hyper-v host and told them that their file-server will come online in less than 15 minutes, slow but it will work!

They were very surprised I could fix this 😋

And it did !!!

After less than 15 minutes their file-server was online running on the backup-server with provisioning the storage on the USB-disk and performed in the meanwhile a sync in the background. After several hours all data was transferred to the USB-disk and the file-server was running on the USB-disk.

 

I told the customer that they had to order as quickly as possible a new server or new hard-disks for their server because that USB-disk won’t live long running like that, it was clearly a temporary solution :smile:

The answer of the customer : they will do that urgently and thanked me a lot and told that I was a wizard to bring their file-server back online in that short amount of time :blush:

 

After 1 week : 

 

After a week my colleagues called me to ask if I would contact that customer again, the customer asked specifically for me 😅

I called the customer back and do you want to know what they asked me ???

 

If I would do it again what I did before because the USB-disk was broken :thinking: . They already bought a new USB-disk in the shop and connected it to the server and ordered that day a brand new server...

 

Conclusion : 

 

Veeam Instant Recovery → it just works and can save a company !!!


7 comments

Userlevel 7
Badge +20

Great story Nico.  Too bad it is late for the voting.  I am going to post one just for the heck of it as I am late too.  😋

Userlevel 7
Badge +17

hahaha...well done @Nico Losschaert ...twice even! 🤣😂

Userlevel 7
Badge +17

Looking forward to your post @Chris.Childerhose 

Userlevel 7
Badge +11

Great story Nico.  Too bad it is late for the voting.  I am going to post one just for the heck of it as I am late too.  😋

Too late 😥??? SysAdmin is just Friday and hadn’t had time before, I was on vacation in Germany - not doing IT related stuff 😥

Userlevel 7
Badge +20

Great story Nico.  Too bad it is late for the voting.  I am going to post one just for the heck of it as I am late too.  😋

Too late 😥??? SysAdmin is just Friday and hadn’t had time before, I was on vacation in Germany - not doing IT related stuff 😥

Too late for just the voting/contest part.  Not for Sysadmin day in general.  😁

Userlevel 7
Badge +6

Great story...I love creative thinking like this and somehow seem to be subjected to such requirements on a regular basis.  I do love that they broke the disk and had to repeat again.  Imagine that.  Hopefully the got that all worked out before failing a third time.

Userlevel 7
Badge +8

Great story. Pretty cool how almost all of us have made some fantastic saves with Veeam. 


Not as much a story, but every day I have someone extremally thankful I was able to retrieve their files or something they deleted by accident.  I recently had a lady break down in tears and hug me she was so happy I saved 15 years of her email after her harddrive was refreshed. That one involved Veeam, VSS snapshots, and a few other data recovery tricks but seeing the look on her face after was totally worth it.

 

 

Comment