Fun Friday: What's the worst reason (or lie) for an outage you've ever had?


Userlevel 7
Badge +4

Happy Friday! It occurred to me I haven’t done one of these about data protection yet, so here we are.

 

Tell us a short story of the worst excuse given for an outage you’ve experienced, here’s mine:

 

Worst Excuse Given:

Earlier this year, users of Amazon’s EU-CENTRAL-1 region had issues within a particular availability zone. Amazon claimed they had an increased API error rate and latencies for EC2 APIs and connectivity issues due to “an increase in ambient temperature within a subsection of the affected Availability Zone”.

This is technically correct, as the datacentre was on fire!

Corey Quinn on Twitter: "BREAKING: There is currently a fire in a data center that's part of @awscloud's eu-central-1 region. Impact limited to a single AZ, obviously. All humans reportedly safe." / Twitter

 

Once again, it goes without saying, check your backups, adhere to the 3-2-1-1-0 rule, protect your data!


16 comments

Userlevel 7
Badge +3

@MicoolPaul, I just want to subscribe to this thread please, as I can’t wait to hear all stories :smile:

Userlevel 7
Badge +2

@MicoolPaul, I just want to subscribe to this thread please, as I can’t wait to hear all stories :smile:

Come on @Kseniya, please tell us your story :sunglasses:

Userlevel 7
Badge +4

@MicoolPaul, I just want to subscribe to this thread please, as I can’t wait to hear all stories :smile:

Come on @Kseniya, please tell us your story :sunglasses:

You too Mildur! You’ve got to share!

Userlevel 7
Badge +2

@MicoolPaul, I just want to subscribe to this thread please, as I can’t wait to hear all stories :smile:

Come on @Kseniya, please tell us your story :sunglasses:

You too Mildur! You’ve got to share!

I’l think of some story which I can tell :D Give me some hours.

Userlevel 7
Badge +3

My Worst Excuse Given:

 

One day I note in my job a trouble with Internet. So, I call to service provider to report my problem. They told me that they are having interconnection problems with their link providers and that they are working to fixing it as soon as possible.

After some minutes I notice that Internet start work properly again. 

After some days ago I discover that the problem was caused by some wrong command that was ran in a router of them by one of their analysts. 

 

The End

 

Userlevel 7
Badge +4

My Worst Excuse Given:

 

One day I note in my job a trouble with Internet. So, I call to service provider to report my problem. They told me that they are having interconnection problems with their link providers and that they are working to fixing it as soon as possible.

After some minutes I notice that Internet start work properly again. 

After some days ago I discover that the problem was caused by some wrong command that was ran in a router of them by one of their analysts. 

 

The End

 

That sucks! But I’m glad they fixed it!

Userlevel 7
Badge +3

My Worst Excuse Given:

 

One day I note in my job a trouble with Internet. So, I call to service provider to report my problem. They told me that they are having interconnection problems with their link providers and that they are working to fixing it as soon as possible.

After some minutes I notice that Internet start work properly again. 

After some days ago I discover that the problem was caused by some wrong command that was ran in a router of them by one of their analysts. 

 

The End

 

This is used to happened , I saw my MPLS service provider did that same thing where my connection is back to back connected with their router at DC side. 

Userlevel 7
Badge +4

I will try to tell the best excuse I have ever heard in english. Luckily it was not in a production environment but in an environment for testing and training.

Suddenly the whole environment was dead. I went to the server room and the power for this environment was off, after some looking I saw that the emergency switch-off was pressed.

I asked the apprentice who has worked with this environment and he said he does not understand this. He pressed the “NOT-AUS” (German for emergency Switch-off) and this button is named not Aus…
So, nothing should have been happened…. :scream::tired_face::confounded:

 

I hope I could translate the the funny part from German to English….

Userlevel 7
Badge +4

I will try to tell the best excuse I have ever heard in english. Luckily it was not in a production environment but in an environment for testing and training.

Suddenly the whole environment was dead. I went to the server room and the power for this environment was off, after some looking I saw that the emergency switch-off was pressed.

I asked the apprentice who has worked with this environment and he said he does not understand this. He pressed the “NOT-AUS” (German for emergency Switch-off) and this button is named not Aus…

So, nothing should have been happened…. :scream::tired_face::confounded:

 

I hope I could translate the the funny part from German to English….

Thanks for sharing 😆 brilliant!

Userlevel 7
Badge +6

oh boy.. now you are asking a rough question. I have heard so many. The best and most honest was way way back in simpler times (A neighborhood cable internet company and the service was down for like 14 hours)  “Our main network administrator is dead drunk and we don’t want to put him anywhere near the equipment”. That is not a joke. The dead drunk part was an “ambient” translation from the terms used in the language of origin. 

Userlevel 7
Badge +6

I will try to tell the best excuse I have ever heard in english. Luckily it was not in a production environment but in an environment for testing and training.

Suddenly the whole environment was dead. I went to the server room and the power for this environment was off, after some looking I saw that the emergency switch-off was pressed.

I asked the apprentice who has worked with this environment and he said he does not understand this. He pressed the “NOT-AUS” (German for emergency Switch-off) and this button is named not Aus…
So, nothing should have been happened…. :scream::tired_face::confounded:

 

I hope I could translate the the funny part from German to English….

Nice! He probably thought he would turn the lights on :joy:

Userlevel 7
Badge +4

I will try to tell the best excuse I have ever heard in english. Luckily it was not in a production environment but in an environment for testing and training.

Suddenly the whole environment was dead. I went to the server room and the power for this environment was off, after some looking I saw that the emergency switch-off was pressed.

I asked the apprentice who has worked with this environment and he said he does not understand this. He pressed the “NOT-AUS” (German for emergency Switch-off) and this button is named not Aus…
So, nothing should have been happened…. :scream::tired_face::confounded:

 

I hope I could translate the the funny part from German to English….

Nice! He probably thought he would turn the lights on :joy:


He didn’t share what he had thought. He was beginning to be ashamed, so we booked this as lessons learned. Afterwards he was careful before switching something on or off.

He became a very good technical consultant at the end :sunglasses:

Userlevel 7
Badge +6

A customer had two sites about 10km apart. the environment was well designed and fail-safe. One day the power in both locations failed at the same time. after the UPS was empty, all servers were down. Reason: the two locations were connected to a high voltage line. this was switched off by the energy service provider to prevent a suicide. It turned out that this line supplied both sites with power.

Userlevel 7
Badge +4

A customer had two sites about 10km apart. the environment was well designed and fail-safe. One day the power in both locations failed at the same time. after the UPS was empty, all servers were down. Reason: the two locations were connected to a high voltage line. this was switched off by the energy service provider to prevent a suicide. It turned out that this line supplied both sites with power.

:open_mouth: thanks for sharing! Lack of information on utility dependencies is definitely a problem here in the UK too for planning.

Cornwall had an outage a few years back for a day because it’s widely considered the “boot” of the UK, it just sticks out from the mainland but it’s so rural there’s very few options for fibre runs. A lot of our fibre gets run alongside or under roads in the UK but there were very few suitable roads for this.

Both main routes for connectivity (which are quite far apart!) suffered issues simultaneously, impacting all of the towns and villages downstream! Was a nightmare, most of the county was completely cut off!

Userlevel 7
Badge +6

A customer had two sites about 10km apart. the environment was well designed and fail-safe. One day the power in both locations failed at the same time. after the UPS was empty, all servers were down. Reason: the two locations were connected to a high voltage line. this was switched off by the energy service provider to prevent a suicide. It turned out that this line supplied both sites with power.

:open_mouth: thanks for sharing! Lack of information on utility dependencies is definitely a problem here in the UK too for planning.

Cornwall had an outage a few years back for a day because it’s widely considered the “boot” of the UK, it just sticks out from the mainland but it’s so rural there’s very few options for fibre runs. A lot of our fibre gets run alongside or under roads in the UK but there were very few suitable roads for this.

Both main routes for connectivity (which are quite far apart!) suffered issues simultaneously, impacting all of the towns and villages downstream! Was a nightmare, most of the county was completely cut off!

It is even hard to get this information!

Userlevel 7
Badge +4

A customer had two sites about 10km apart. the environment was well designed and fail-safe. One day the power in both locations failed at the same time. after the UPS was empty, all servers were down. Reason: the two locations were connected to a high voltage line. this was switched off by the energy service provider to prevent a suicide. It turned out that this line supplied both sites with power.

:open_mouth: thanks for sharing! Lack of information on utility dependencies is definitely a problem here in the UK too for planning.

Cornwall had an outage a few years back for a day because it’s widely considered the “boot” of the UK, it just sticks out from the mainland but it’s so rural there’s very few options for fibre runs. A lot of our fibre gets run alongside or under roads in the UK but there were very few suitable roads for this.

Both main routes for connectivity (which are quite far apart!) suffered issues simultaneously, impacting all of the towns and villages downstream! Was a nightmare, most of the county was completely cut off!


I have gotten this information too late for one customer once :sunglasses:

This DC was powered via one supplier only. Under normal circumstances we try to have two power suppliers for each DC we are using.

And so there was an power outage one day. Luckily there was a rather big Diesel engine to power the DC during the outage and we had no loss...

Comment