Servers & Drive Numbers - Cause Chaos

So, I thought for the blog of the month special edition I would go back to when I first started out in the IT world early in my career.  I found a job as a Systems Administrator at a small law office in downtown Toronto.  I was so excited to get my IT career going and this was my first job in the Sysadmin space.  The firm also had an external IT company as well under contract, but they felt the need to have someone onsite to provide easier assistance to users, manage servers, etc.

So, I was working there for a few months when I happened to one day do my daily routine check in the server room and noticed an amber light on one of the drives in the server.  Now keep in mind I was new to IT and did not understand the concept of the hard drives starting at Zero not One (you can see where I am going with this ).

So, to be diligent and ensure things were done right I contacted the person at the IT consulting company, and he told me that server drives are hot swappable.  I took their advice but to my own credit did not ask about drive numbers.  So, I proceeded to go into the server room and pull out the drive that I “thought” was the right one, but alas it was not.  Boom the entire file system on the File Server went down.  Little to say I was nervous and scared of what may come.

So, I contacted the company again and we walked through replacing the correct drive but then the recovery process started.  Back in those days I had not even heard of Veeam but had that been around restoring would have been so much quicker.

I started to work on the restore process of rebuilding the entire file server which was terabytes of data and actually slept in my clothes I wore Friday to the office all weekend in one of the lawyer’s offices that had a couch.  I was completely drained by the time Monday came and had the server pretty much back up, but things were still rebuilding with the RAID, etc.

I spoke with one of the partners and managed to be able to go home to get some much-needed rest, but when I came back the next day that is when my manager called me in and unfortunately, they removed my position.  My own fault but it was one of the best learning experiences I had regarding servers and backups!

I eventually found another job after this one and don’t regret my actions but learned from them to advance my career.  I did more reading on servers and disk numbering, etc. to ensure if I was put into that position again, I would be able to swap the correct drive.  

If I had of had Veeam around in those days' recovery would have been much quicker I am sure, but now I have been using it for many years and cannot live without having my backups.

TIP - ensure you pull the right drive and if not, be sure you have Veeam backups!