What is your BEST SysAdmin Story - I will feature it Friday!

Definitely tuning on to this one. Will try to think of a story as well and post back.

I joined a company and saved them from an ongoing undetected ransomware attack on week one.

I’d joined as the sole senior IT person, and one of the first issues I was informed of was “issues opening attachments in CRM”. The suspicion was that the CRM had two data silos with separate encryption keys, and one of these silos had been decrypted and migrated to a new CRM, and they believed this had messed with the encryption process for the other data silo. (Dev was Prod of course!)

Nope, it was nothing to do with the CRM, it was ransomware! I discovered the incumbent MSP & junior IT hasn’t been deploying any AV to new PCs, so for the past two weeks a couple of unprotected endpoints were sat there quietly encrypting data.

The company was between 10-15 years old and this CRM was the one true copy of the business’ transactions & history, operating in a regulated space this HAD to be reliable.

I did some research into the data and discovered it had started approximately two weeks ago.

I started looking into backups to recover this data and noticed rotated USB drives were being used, the MSP informed me that the retention period wouldn’t cover how far back I needed to go, and unfortunately the dataset would remain incomplete, closed the case their side. However I noticed that we were actually one USB backup job retention point away from losing this data. I ordered some new USB drives to allow for new backups to continue and then pieced together all encrypted data from the USB backups over that two week period, using the tool that would soon become my favourite, Veeam!

So I saved the day, saved the company, highlighted a bunch of weaknesses that needed improving, and discovered what was going to need to be my first priorities at my new job! And never let it be said this work goes unrecognised as I was made the IT Manager six months later!

While working on a third party, we found that customer’s servers were affacted by ransomware (talking about while ago, when fortunately exfiltration wasn’t a thing). All data were lost. Servers, replica, backups. All. Lost. We saw people crying. Then we tried the impossible and, with absolutely luck too, deep cold analyzing unused disk sectors we rebuilt a full data backup file where no sector were overwritten. It was less than a week before, better than nothing. And win the game.
What can we learn from this story? That backups saved a company life, one more time.

Great @Rick Vanover! I have created an alert in order to be notified. Some of my SysAdmin stories are as follow.

I have got a thing for Cyber Security and will be transitioning at a later time. Interestingly, I have been involved in setting up core security systems such as Cisco ISE, MBAM, Microsoft Defender Manager (3rd party tool), setup a secondary ManageEngine Password manager to eliminate SPOF, and Pleasant Password Manager etc across various organizations I have worked thereby helping to protect the IT systems and also reduce cost.

Before I joined a specific company, they were found of installing updated without testing and many a times, this broke the instances and they would have to create a new server from the AMI. Came in suggested we implement Windows Server Update Services and from this time, there wasn’t a need to rescue service with AWS support or recreate the severs from pre-generated AMI.

Interesting to sharing was been able to decommission and replace SBS with WS2012 and Exchange 2010 with Exchange 2016 at the very start of my carrier. This was very fulfilling! This was a great win as at then… 

Signed up as well for the raffle :-) 

I think I told one last year but.. the good thing about growing old is you accumulate IT stories like debt or parking tickets 🙂. 

I was working in a company and they had just introduced instant messaging. The problem was that no proper naming conventions had been implemented and everyone in the IT department, being always in preventative mode, decided to stealth their usernames so that people would not be able to easily reach out to “support” i.e. my printer is not working, my laptop is making a funny noise. So we picked names that were similar to people in other departments. Departments that we figured on one would ever want to contact. This was a dilemma at first. Obviously Accounts was a big no no, since they go harassed constantly for reimbursements and salary questions. HR was a no go since they knew all the billions of corporate rules and we figured there might be one about attempted username assimilation or something like that. Luckily there was a person who worked in a position called something to the effect of “volunteer committee” which was intended for those career “go getters” types. His job was to recruit employees for Pro Bono (i.e. free) extra work or event organizing, in other words things that no one wanted to do. I made my username to have just one consonant and a vowel different. I can’t remember his name but if it was John Doe, I mine Jon Dooe. I figured that even if people had a suspicion that it might be me they would still not want to risk coming under the radar of the Volunteer collector. 

The first few days were great, no messages, so we just stuck to ignoring the phones. However, on the 3rd day I almost collapsed in terror when I saw 10 new messages blinking in the corner of my screen. With the utmost dread I double clicked the icon with remorse at the fact that my cover had been blown.

To my utmost surprise and complete bewilderment I read the following: “I don’t have the courage to say this in person but I think that I am in love with you” A. There followed more to that effect as well. I knew who A was and my first emotion was utter bliss. She had never even noticed me before and yet all this time…. then I remembered my stealth trick.. That whole morning was absolute torment, what should I do? There was a chance that the IT dept’s trick had been figured out and... or … but the Volunteer collector fellow???? In the end I decided that I could take in no longer and changed my username to my real name and waited… All afternoon I surfed and negotiated through complaints requests and demands that things get fixed asap.. alas I never got another message from A again.

However, the story had a happy ending as the IT Manager soon found out what had been going on and everyone in the dept got scolded for not being honest. I stood there proudly as I was used as an example of someone with true integrity and decency, I even won Employee of the Month :)