Skip to main content
  • EN

Happy Friday, Veeam Community!

 

Since @MicoolPaul is on Majorca enjoying sunshine and beach, I am on a mission to back him up with the regular Fun Friday discussion! :blush:

 

What's the worst reason for an outage you've ever had and how badly did it impact you?

 

Here’s story that @MicoolPaul shared with us:

“When I was internal IT, we had a summer where we kept suffering power outages at our main building, with people keep getting stuck in the lift! One day we had a bad outage, but it was only partially an outage, apparently two of our three phases were dead! So we had a few random plugs around the office that still worked. To get things back up and running we managed to ignore all health & safety and ran a massive extension lead from a working socket to power our UPS and our entire rack! (it stunk of hot metal and I wouldn't leave it unattended for even a second!).

Eventually we found out that the building management had replaced all of the fuses for the building with ones that were HUNDREDS of amps lower than they were supposed to be. And every time someone used the lift it killed the power!

Once everyone found out, the landlord replaced the building management company with a more competent group! :smile: ”

Unbelievable story from @MicoolPaul ! I have to think a while for a case :wink:

Unbelievable story from @MicoolPaul ! I have to think a while for a case :wink:

I bet all of us have some stories in the pocket! :smile:

Oh, oh, this is not long ago…

In July there was a massive flooding in Germany and one customer had his server room in the basement….

Do I have to tell you more? :sunglasses:

 

The whole room with all systems is completely destroyed. Luckily we have a backup stored outside of this location and were able to get the most important systems up and running in the cloud.

The customer is now in the process to build a new server room. We did not agree that it is enough the move the server room just in a higher floor. :tired_face::tired_face::tired_face:

I love this story, @JMeixner! :smile:

Why would anyone locate the server room in the basement? it’s so risky! 

It’s also usually damp in the basement :grimacing:

Mhh, many server rooms are in the basement…. in many buildings it is cooler than in the other floors.

But you are right, it is risky - especially if there is water in the surrounding...

 

when i was an intern during summer time, we had problem with the air conditioning fans in the server room, I spent two weeks frequently on the building roof watering them with a colleague during an heatwave :tired_face: . We did the jobs and no outage occured.

It was ok, they provide to us beach umbrella lounge chair and the panorama was eiffel tower and montmartre.

when i was an intern during summer time, we had problem with the air conditioning fans in the server room, I spent two weeks frequently on the building roof watering them with a colleague during an heatwave :tired_face: . We did the jobs and no outage occured.

It was ok, they provide to us beach umbrella lounge chair and the panorama was eiffel tower and montmartre.

Haha :thumbsup_tone3:

Hadn’t heard this from IT up to now. My father was chef in one of the big kitchens at at a huge chemistry company in Düsseldorf and they had to water the roof of the kitchen and staff canteen in summer with several sprinklers. Otherwise the air condition did not work properly.

And no, it was not an old building...

@Kseniya : What a unique story :grinning:

when i was an intern during summer time, we had problem with the air conditioning fans in the server room, I spent two weeks frequently on the building roof watering them with a colleague during an heatwave :tired_face: . We did the jobs and no outage occured.

It was ok, they provide to us beach umbrella lounge chair and the panorama was eiffel tower and montmartre.

haha.. 

Oh, oh, this is not long ago…

In July there was a massive flooding in Germany and one customer had his server room in the basement….

Do I have to tell you more? :sunglasses:

 

The whole room with all systems is completely destroyed. Luckily we have a backup stored outside of this location and were able to get the most important systems up and running in the cloud.

The customer is now in the process to build a new server room. We did not agree that it is enough the move the server room just in a higher floor. :tired_face::tired_face::tired_face:

Oh no .. that always a case for the basement 

Oh, oh, this is not long ago…

In July there was a massive flooding in Germany and one customer had his server room in the basement….

Do I have to tell you more? :sunglasses:

 

The whole room with all systems is completely destroyed. Luckily we have a backup stored outside of this location and were able to get the most important systems up and running in the cloud.

The customer is now in the process to build a new server room. We did not agree that it is enough the move the server room just in a higher floor. :tired_face::tired_face::tired_face:

I love this story! I have a customer who’s building is next to a cemetery and the comms room has a sealed off tunnel that goes UNDERNEATH the graveyard… they say whenever it rains heavily they worry that the seal is going to perish and who knows what will enter the room 😬

Oh, oh, this is not long ago…

In July there was a massive flooding in Germany and one customer had his server room in the basement….

Do I have to tell you more? :sunglasses:

 

The whole room with all systems is completely destroyed. Luckily we have a backup stored outside of this location and were able to get the most important systems up and running in the cloud.

The customer is now in the process to build a new server room. We did not agree that it is enough the move the server room just in a higher floor. :tired_face::tired_face::tired_face:

I love this story! I have a customer who’s building is next to a cemetery and the comms room has a sealed off tunnel that goes UNDERNEATH the graveyard… they say whenever it rains heavily they worry that the seal is going to perish and who knows what will enter the room 😬

:joy::joy::joy:

The walking dead?

I love this story! I have a customer who’s building is next to a cemetery and the comms room has a sealed off tunnel that goes UNDERNEATH the graveyard… they say whenever it rains heavily they worry that the seal is going to perish and who knows what will enter the room 😬

 

OMG!!! :smile: That’s crazy :smile:

:joy::joy::joy:

The walking dead?

 

:joy::joy: :joy:

Thanks for posting this whilst I’m away @Kseniya . Don’t worry I’m having a cocktail whilst you do all the hard work!

 

Happy Friday everyone!

Happy friday Michael and have a nice vacation.

Just don't overdo the partying 🤣😎👍🏼

:hugging:

When I used to be a System-admin (former Veeam-client) we were going to test our automated shutdown-procedure by cutting the power and have the UPS automatically (when batteries were nearly drained) shut down the VM’s+hosts.
My colleague at that time was so friendly to cut the OUTGOING UPS-power instead of the incoming. Resulting in a sudden death of the DC 🙂.
That day we discovered that an automated boot-sequence is pretty important too (in case an sql or app-server boots before AD for instance 😛 ). 

Since that day the in-and outgoing power-fuses were documented and labeled very well ;-)

This was one of the most stressy-maintenance weekends I ever had...

When I used to be a System-admin (former Veeam-client) we were going to test our automated shutdown-procedure by cutting the power and have the UPS automatically (when batteries were nearly drained) shut down the VM’s+hosts.
My colleague at that time was so friendly to cut the OUTGOING UPS-power instead of the incoming. Resulting in a sudden death of the DC 🙂.
That day we discovered that an automated boot-sequence is pretty important too (in case an sql or app-server boots before AD for instance 😛 ). 

Since that day the in-and outgoing power-fuses were documented and labeled very well ;-)

This was one of the most stressy-maintenance weekends I ever had...

Thanks for sharing! In the U.K. we have a saying, when you ASSUME you make an “ASS out of yoU and ME”

 

At least lessons were learned!

Comment


Terms & Conditions