Gone are the days of "If its not broken, don't fix it"


Userlevel 7
Badge +6

I’ve been in IT my entire adult life. There is generally two mantras to IT.

 

  • “If its not broken, don’t fix it.”
  • “Update often.”

 

“If its not broken, don’t fix it.”

You know the people I’m talking about. “My uptime on this system is 1287 days! It is rock solid!”


I get it. Its great to have a system you can rely on. Stability and system uptime is the holy grail. You want your systems to be 100% available because that means you can focus your time on more important tasks than break/fix. 

 

These people are also a security expert’s nightmare. Vulnerabilities get exposed and new exploits are created regularly. Its not enough to protect your perimeter anymore, you have to protect every aspect of your environment. 

 

This mantra has outlived its usefulness. Uptime of 1287 days on a system is no longer a badge of honour, its downright scary. We’re not talking highly available services here where you can take individual components down to patch them, we’re talking those individual components themselves.

 

“Update often.”

Update often is the only mantra that matters anymore. Whether its firmware, drivers, system patches, version upgrades, etc. All these need to be examined on a regular basis, or you flirt with disaster. 

 

My uptime is 1287 days

 

Many security updates are no longer clearly specified or able to be applied individually - they are simply bundled with the next release version. If you’re lucky, release notes contain notes about these vulnerabilities being patched, but not all do. If a vulnerability was disclosed privately to a organization, they may silently fix the offending code and no one would be the wiser.

Major companies do this regularly, and at times only disclose the vulnerability after its been patched and released for a long while. Don’t believe me? Pay attention to CVE releases and check to see which versions they were patched in, you may be surprised to see the patched version was released weeks/months back. 

 

Note I’m not advocating for bleeding edge all the time in every circumstance. You need a plan. You need to analyze your compatibility matrices. But you do need to upgrade eventually, that’s just life as we know it now in IT. The days of “If its not broken, don’t fix it” are over, because it is broken, you just haven’t realized it.

 

So check your OS’s, servers, switches, routers, storage arrays and if you have IOT devices, perhaps even your fridges. Check your services. Are you patched? 


4 comments

Userlevel 7
Badge +20

This is so very true especially if it ain’t broke don’t fix it.  But nowadays you need to stay more up to date with cyber criminals being more active, etc.  Also patching is a must nowadays it seems so good to say on top of it for sure.

Userlevel 7
Badge +6

This is so very true especially if it ain’t broke don’t fix it.  But nowadays you need to stay more up to date with cyber criminals being more active, etc.  Also patching is a must nowadays it seems so good to say on top of it for sure.

100%. I was reading through this post

 which is what prompted my post. I had mused to myself about how many people will probably ignore this with their patching. 

Userlevel 7
Badge +8

Security updates are always top priority where as functional updates I tend to lag behind a bit.

 

If it ain’t broken don't fix it is true when doing updates for updates sake, but looking at release notes and patches is critical in modern day infrastructure.

 

Also, if you fall too far behind, it gets VERY complex with compatibility matrix’s, and doing multi step upgrades in several hardware/software environments just to get a 10/10 CVE that is released.

 

There is a balance between updating something that came out today, or not updating something for years. A lot comes down to What you are updating, how isolated your network and environment is, how it’s accessed and other aspects of security. Our more critical things I tend to ask the stake holders what they would like and give them my advise. At the end of the day, it’s their choice, and sometimes security means taking the risk of patching sooner.  If you go the other way, and someone hacks you, you would have rather done the update and taken the risk. 

 

Userlevel 7
Badge +6

Security updates are always top priority where as functional updates I tend to lag behind a bit.

 

 

Agreed. Functional updates are usually a lower priority. That said, I’ve seen patches where it looks to be functional only, but adds hidden security updates. 

 

 

Also, if you fall too far behind, it gets VERY complex with compatibility matrix’s, and doing multi step upgrades in several hardware/software environments just to get a 10/10 CVE that is released.

 

 

This is one of my biggest reasons for staying up to date, or within a reasonably up to date release on functional updates. Especially when you’re in an environment where multiple vendors have a part to play. Catching up on technical debt becomes far more difficult to plan when you need to consider how to upgrade 10 different pieces of software and firmware that all need to play nice together. 

 

 

There is a balance between updating something that came out today, or not updating something for years. A lot comes down to What you are updating, how isolated your network and environment is, how it’s accessed and other aspects of security. Our more critical things I tend to ask the stake holders what they would like and give them my advise. At the end of the day, it’s their choice, and sometimes security means taking the risk of patching sooner.  If you go the other way, and someone hacks you, you would have rather done the update and taken the risk. 

 

Will upgrading X require new compatible hardware? Definitely something stakeholders need to handle and budget for, and I try my best to voice those concerns well in advance. If upgrading X causes business processes/automation, etc to be updated, then I need a timeline on when we can get that done. It tends to be all part of modern IT management and should be a regular discussion point. Once that ball starts rolling, it becomes easy. Getting that ball rolling on the other hand… that’s a different blog post. :D 

Comment