Hi Folks,
When I think about being a Systems/Backup Administrator all of these years I can’t single out one single story that fully represents the highs and lows of this profession. So instead I have decided to write a few short sound bytes, streams from moments of memory suffixed with rhymes like one of the those “_restored” attachment names we use in order not to overwrite production.
Every moment is also and emotion. We add a taint of emotional metadata to events and memories that we often don’t fully comprehend ourselves. Our work is matrix of puzzles and problems that we face everyday with our silicon little friends. Although sometimes it feels like they are our foes.
Gratitude
The call broke through the comforting calm silence of a Friday morning IT room like a waiter’s tray that gets turned over in a busy restaurant. An important file was missing. It had been erroneously deleted in one of those careless human moments of blissful neglect. The stakes were very high. First attempt.. no good, some unintelligible error. Point in time to the day before resulted in relief and thanks galore. “you folks are the best, thank you so much!!”
I saw the person the next day in the hallway and they forgot to say hello.
“Nervous voice asking for a restore
First try failed but luck struck
On the backup from the day before
The love and gratitude flowed
But as time went by those feelings slowed
Walked right by but you forgot to say Hi”
Mean
“I don’t need any crap, I need this fixed now!” The boss was angry as our system was down. Active Directory was doing strange things and some employees were not able to login or their profiles were not loading correctly. I was not the main windows admin in the office but that did not matter we were all at fault. “You need to hurry up, we don’t pay for down time” was the next shout. I guess the Manager thought we needed some motivation.
Instead our thoughts froze. Every sound or angry rant was a distraction. You found yourself having to go over the same steps again and again. We were making no progress. The manager was a corporate type. He knew who to praise and who it was worth being polite too. He portrayed an image of a no nonsense person who got results. If things went well he was always there front and center, when things did not his eyes would start a frantic search for the most gullible and vulnerable admin who could “be a team player, and take one for the department”.
This time however, things were not going well, the upgrade was supposed to be a simple one, so he had already taken full credit for its planning in front of the powers that be. The ship was sinking and no one new where the hole was in the boat.
In the end we had to call in the regional support team experts. They fixed the issue and found a major flaw in the planning. After an investigation the Manager was transferred to a different location and to a position with less responsibility.
“ Server’s LED eyes staring down
At the humans acting like clowns
Wondering why,
The person chosen to lead was this guy.
But just maybe what we dare
To take as an empty mechanical stare
In fact does really care
And for the Sysadmins’ sake
The software picked just the right moment to break
And sealed the mean Manager’s fate”
The Real Benefits
“Hi its…… we have an emergency, our screens are locked and a message stating that our files are encrypted has appeared. Please help”
Sometimes things get personal. The small business owner was a nice guy. He often asked how we were doing and our team would enquire about family and other things. It was one of those business relationships that had crossed over to business friendship. Disaster does not play fair, it can hit anyone at anytime. Sometimes the people we like the most can be hit the worst.
“They just told us that our local backup server is completely encrypted, I don’t know what I am going to do, I could lose my business”
Dealing with customers in crisis requires more that just technical skills. You need employ psychology as well. I knew we had ransomware protection in place and I trusted our Veeam backups but I also knew from years working in IT that things can go drastically wrong despite all of the precautions. I kept my voice calm and informed him of the steps that we were taking. The cloud backups were gone as the ransomware had compromised the backup server and most likely sent a delete command down the pipe (this was before immutable repositories) but we had insider protection and were leveraging GFS.
It took some time but we were able to restore all of the client’s servers and data. I went home feeling like I had just received a huge bonus. In fact I had, no amount of money could make me feel this good. I had played my little part on the side of good. In the daily stream of bad news and depression that we encounter on our little planet, I had helped make a small difference, a pin point of bright light in what sometimes can be a very dark picture.
“Bad folks don’t care
The empathy is just not there
They hit you wear it hurts the most
Pay up or your business is toast
And even then who knows
Maybe we take the money and
Leave you with no clothes
Its all a joke to us
as long as our bitcoin accounts add the plus
----
But bro you made a big mistake
Veeam backups will take away your cake
Your coffers are empty now
No more milking the cow
The nice folks are saved
The storm has been braved
Veeam was the tool
Making you ransomware crooks act the fool
----
But don’t let your guard down
You know they will come back around
Ready to slate
So keep your Veeam software up to date
and seal the enemy’s fate
Shout 3, 2, 1
Our Data protection work is never done
And win we must
to keep the customer’s trust
and hand cyber criminals a big bust!”
