Happy Friday everyone!
Today I want to trade ‘war stories’ of IT mishaps and experiences learned.
Have you ever felt yourself psychic, that you could see something going horribly wrong, and it DID go horribly wrong? Maybe you had the overwhelming urge to say I told you so? Maybe you did tell them so and got promoted or fired!
Lets hear the stories!
My story that inspired this Fun Friday happened a fair amount of time ago, so it feels okay to discuss now.
Customer was going through some financial difficulties, cash flow was reduced, survival mode engaged. Cut all the costs that you can find.
The support contract for the backup server was due for renewal, this server was already 5 years old and with their financial forecast, a replacement was not on the table. We tried to be price sensitive and offered third party Next Business Day support, was as cheap as we could get. Customer asked if there was anything cheaper, we told them there wasn’t. Customer declined support renewal. I made a personal appeal in the conversation that this server only has a RAID 5, is their ‘all-in-one’ backup server, if it stops, all backups stop.
Customer still did not renew.
Fast forward 6 months, it happened. A disk failed in the array. Customer needs a quote for a disk, we get the quote, but the supply chain says it’ll take 10 days to get there. Panic sets in.
Customer got a disk failing warning for a second disk, second disk ordered. Customer is stressing a lot “have you not got any spares???” Type of questioning of us. Drive turned up, failed disk was replaced, rebuild completed over the weekend, before they got anyone to site to replace the failing disk, it failed, cue more stress whilst they awaited the RAID rebuild.
Side note: The 3rd party NBD support contract was less money than the two disks..