World Back Up Day Blog of the Month Special Edition - My Story


Userlevel 7
Badge +21

Servers & Drive Numbers - Cause Chaos

 

So, I thought for the blog of the month special edition I would go back to when I first started out in the IT world early in my career.  I found a job as a Systems Administrator at a small law office in downtown Toronto.  I was so excited to get my IT career going and this was my first job in the Sysadmin space.  The firm also had an external IT company as well under contract, but they felt the need to have someone onsite to provide easier assistance to users, manage servers, etc.

So, I was working there for a few months when I happened to one day do my daily routine check in the server room and noticed an amber light on one of the drives in the server.  Now keep in mind I was new to IT and did not understand the concept of the hard drives starting at Zero not One (you can see where I am going with this 😂).

So, to be diligent and ensure things were done right I contacted the person at the IT consulting company, and he told me that server drives are hot swappable.  I took their advice but to my own credit did not ask about drive numbers.  So, I proceeded to go into the server room and pull out the drive that I “thought” was the right one, but alas it was not.  Boom the entire file system on the File Server went down.  Little to say I was nervous and scared of what may come.

So, I contacted the company again and we walked through replacing the correct drive but then the recovery process started.  Back in those days I had not even heard of Veeam but had that been around restoring would have been so much quicker.

I started to work on the restore process of rebuilding the entire file server which was terabytes of data and actually slept in my clothes I wore Friday to the office all weekend in one of the lawyer’s offices that had a couch.  I was completely drained by the time Monday came and had the server pretty much back up, but things were still rebuilding with the RAID, etc.

I spoke with one of the partners and managed to be able to go home to get some much-needed rest, but when I came back the next day that is when my manager called me in and unfortunately, they removed my position.  My own fault but it was one of the best learning experiences I had regarding servers and backups!

I eventually found another job after this one and don’t regret my actions but learned from them to advance my career.  I did more reading on servers and disk numbering, etc. to ensure if I was put into that position again, I would be able to swap the correct drive.  😂

If I had of had Veeam around in those days' recovery would have been much quicker I am sure, but now I have been using it for many years and cannot live without having my backups.

TIP - ensure you pull the right drive and if not, be sure you have Veeam backups!


8 comments

Userlevel 7
Badge +17

Oh gosh @Chris.Childerhose !! 😁 Well...lesson learned. And, I think things have worked out real well for ya bud. Thanks for sharing your story! 👍🏻

Userlevel 7
Badge +21

Oh gosh @Chris.Childerhose !! 😁 Well...lesson learned. And, I think things have worked out real well for ya bud. Thanks for sharing your story! 👍🏻

Yes, it was definitely one of those aha moments in life, but I always say live and learn.  If you don’t make mistake you cannot learn from them which is how you get better especially in our industry.  😉

Yeah, I would say things worked out pretty well after that hahahaha

Userlevel 7
Badge +8

I triple or quadruple check things sometimes.  Even still, deleting a 30TB or 40TB volume is always a stressful event. 

It’s amazing you you can check something 3 times, have someone else verify, come back, click delete, and are unsure if you clicked the right thing even. In my case every time I do that an unrelated issue occurs to give me a scare. It never fails.

Or if I pull a cable in the datacenter, a manager will come flying threw the door making me think I took the site down, only to ask me a question as I nearly have a heart attack. lol

Userlevel 6
Badge +3

I remember those days of pulling out drives. I remember that some newer systems had a LED light you could turn on to indicate the drive you were working on. It made it easier but if you had a big storage farm with lights left on then you would have to start turning them all off and starting again to be safe 😂

Userlevel 7
Badge +21

I remember those days of pulling out drives. I remember that some newer systems had a LED light you could turn on to indicate the drive you were working on. It made it easier but if you had a big storage farm with lights left on then you would have to start turning them all off and starting again to be safe 😂

Yes, that definitely would have helped me for sure, but hindsight is 20/20.  😋

Userlevel 6
Badge +3

I remember those days of pulling out drives. I remember that some newer systems had a LED light you could turn on to indicate the drive you were working on. It made it easier but if you had a big storage farm with lights left on then you would have to start turning them all off and starting again to be safe 😂

Yes, that definitely would have helped me for sure, but hindsight is 20/20.  😋

@Chris.Childerhose yes. I made my share of Resume Generating Events (RJEs) lol 🤣

Userlevel 7
Badge +9

Good one @Chris.Childerhose! I love the fact that you learned a lot. This is all that matters. 

Userlevel 7
Badge +21

Good one @Chris.Childerhose! I love the fact that you learned a lot. This is all that matters. 

Yep always learning I say. 😁

Comment