Fun Friday #58: The onosecond/ohnosecond

First off...that’s quite the collection he has sitting on the background of this video.

Back to topic, I’ve had several onoseconds and they still occur, though more often than not they are false alarms.  Where I thought I did something wrong, went back and looked...no, I’m good...I did it right.  Possibly the worse though was when I helped manage a website for my former employer.

Sit down kids, it’s story time…

I worked for a real estate company here, and the website contained not only the back off utilities that our real estate agents used for managing and maintaining listings, submitting listings to local newspapers and such, but also for the public website that everyone used to view houses for sale, rent, etc.  We had a system in place where an agent could create a shortened name for a listing that would redirect to the actual listing.  For instance, a typical listing URL would be something like website.com/listings/listing.cfm?id=20201202091305 where the long, randomish number was the MLS (Multiple Listing Service) ID.  Obviously, not very user friendly, so we created a system where users could create an aliased url like website.com/listings/123parkwaydrive that would redirect to the long URL.  Unfortunately, we had not developed a system for users to delete this alias, nor have to automatically deleted with the listing ID was removed from the SQL database. 

So I built one.  And like all good systems, I did most of my editing in production.  I wasn’t really a Cold Fusion developer, but was learning.  In fact, none of us were, but we managed.  So early on I did my editing in VIM, not yet learning the way of NotePad++.  I eventually got Dreamweaver for this, but regardless, I’m not sure I would have avoided the mistake either way.  When I wrote my cleanup utility, I would perform a cfdirectory (that is, give me a list of all of the files and folders in a given directory).  When I look at the SQL database, if I see a matching folder path in the database that corresponds with an active listing, leave it alone.  If I don’t see a matching path, assume that the folder path is for an old listing that no longer exists, so issue a cfdelete command to delete the contents of the folder and the folder itself. 

I wrote my utility, and then ran it.  Except….it was taking a long time to respond.  Several long seconds later, in confusion, I reloaded the page to run it a second time.  After a longer period of waiting, I F5’d again, but this time was met with a “The page cannot be displayed” message.  “Oh No!”  View the website as a whole….same result, “The page cannot be displayed”.

Perhaps you might see where this is going.  As you may or may not know , mistakes seem to happen in two possible ways.  Either they fail really fast where you’re blindsided by the result very quickly left wondering what happened, or the fail really slowly, where the system grinds away for a while, not necessarily letting you know what is happening, making you wait and wonder, wonder and wait, and giving you ample time to worry. 

Mine….the latter, which in itself was a rather spectacular failure.  I made a critical mistake in my cfdirectory command.  Instead of telling it to go to the /listings directory, I accidentally went up one level too high, that is, the root of the website.  My query ran really fast actually...it looked at the root directory, didn’t see a corresponding entry in the database table, and dutifully executed the cfdelete command across all contents at that level, that is, delete the entirety of the root folder.  All of the files.  Web pages.  Folders.  Images.  Everything.

Well, not entirely true, the SQL database remained.  But that’s of no help without corresponding pages to reference it.  And of course, when did this happen.  During the lunch hour.  You know, one of the peak times when people sitting a their computers are likely browsing real estate listings, looking for their next home while consuming a sandwich.  Except they were going to have to go look somewhere else, because there was no website to browse.

As an upside, when we took over the website from the contractor we had, we realized it was very poorly put together with zero redundancy.  We added some redundancy.  That is, this was running on Windows Server, and we decided that creating a DFS replication link from the datacenter where the website ran to our own server room over the VPN was a good idea.  I’m not sure if it was so we could take backups in our server room, or something else.  Also, this was a really large website.  Thousands of real estate listings.  Upwards of 20 or 30 pictures per listing, but we kept three copies of each image - a small thumbnail, a regular sized image, and a full scale image to zoom into.  This meant we could have anywhere from 60 - 90 images per listing making for a lot of files.  And suddenly deleting the entire website, that’s a lot of data to sync via DFS.  We quickly were able to disable DFS replication.  Copy the website back from our main office back to the datacenter.  Well, copy the folder structure and files, sans images, back to the datacenter quickly.

Then rerun our download script.  See, the MLS hosts all of the listings.  It’s rather cryptic, but the point is that every few hours, we’d run a script to check for new listings from the MLS.  Any new listings it found would be downloaded in imported into our database.  Any images would be downloaded, copied, resized, copied, and resized for the three copies for each listing.  Restoring the site didn’t take a long time, so at least there was something responding to folks on the web.  And then we just wait for the script to run, to download, and update each listing with the proper images.

Total website downtime?  Well, about 45 minutes I suppose.  Remaining downtime for the images to show back up?  Probably a couple hours.

Now certainly this would be considered an RGE - Resume Generating Event.  Better go look for a new job, right?  Actually, one of the many things I leaned from my IT Director, the other guy maintaining the website, is mistakes happen.  The bigger picture is to learn from your mistakes.  And I did certainly learn.  And this is something that I’ve kept in mind for my more junior engineers and administrators.  Mistakes happen, let’s fix it, and learn from it. 

Oh...and make sure you have usable backups of course.

Mine was when I first started in IT and we all know disk drives start at zero not one but when a drive at a law firm went bad guess what?  I pulled the wrong drive and then spent the weekend in the office to restore it. 😂

Something that taught me a valuable lesson moving through my career.

Absolutely but this was a physical server which I learned from. 😂

This is more or less what I did in my missing post.  But it was for a website instead.

😎 it's a very common experience….

Thanks @safiya!  It was long...didn’t want to write it all out again.  :-)

Thanks @MicoolPaul for the menition!

I think my own personal one would be when I had to restore a server without NICs alongside the running virtual server. I, basically, needed some data from an Exchange server that was running on Small Business Server. We had recently performed a cloud migration and the only way to get the data we needed was to restore the server and use Outlook.

When I went to restore the server, I was focusing so hard on making sure the NICs weren’t connected, that I accidentally restored the server in place. The live server was instantly removed and the company lost all work since the last backup, which was about 17 hours earlier.

I said some unrepeatable words and considered a change of pants! The lesson, as always, is: double check things before clicking and get a second pair of eyes if you need to.

I 100% used Norton Ghost to image a drive on a professors PC once and while he was yelling at me before I got there he continued to rush me and was quite distracting. 

fast forward a few hours.  When you ghost an empty drive full of zeros over a drive with 0’s and 1’s. you’re going to have a bad time. 

A valuable lesson learnt that day. Not even on the technical part, but if an angry customer wants to vent, stop working and let them vent all day. Proceed to tell them as soon as I can get back to work the faster things can be fixed. They usually bugger off pretty quick.

Now that I’m a SAN, backup, Server guy though most of those days are long over :)