Latest Reminder that the safest service is a disabled service (SSH Vulnerability)


Userlevel 7
Badge +20

Morning,

 

Just taking a moment to remind everyone that at this point it feels like a certainty that every application you use has a vulnerability, it’s just whether that vulnerability has been discovered yet.

 

Take this latest research into a new SSH vulnerability: New SSH Vulnerability - Schneier on Security

In summary, naturally occurring issues such as a flipped bit in memory can be enough to expose private host keys in some SSH implementations. I’m not saying to go run around in circles and panic, but what I am suggesting is that you review your defence in depth posture.

A great mindset to be in around any inbound connection to a device should be:

  • Do I need this service to be listening? If not disable the service.
  • Okay, I need it to be listening, what ports do I expect it to listen on? Only permit connections to it on pre-approved ports.
  • Does it need to be listening to connections from anywhere? If not, utilise IP address whitelisting on a firewall

There are lots of other layers you can insert to enhance upon this basic steps such as placing an internal VPN between your network and an isolated, sensitive network segment, that you secure further with MFA, to permit access to sensitive services such as SSH, RDP, IPMI etc.

 

We’re in a constant battle between less convenient = more secure, and more convenient = less secure, but establishing a consistent security baseline improves an organisation’s cultural adoption to security and helps teams understand applications better by seeing how the various components communicate.


15 comments

Userlevel 7
Badge +20

Having less convenient and being more secure shouldn't be an issue as taking the extra needed steps for security is the better approach.  Hate seeing these vulnerabilities lately as we are always patching.

Userlevel 7
Badge +17

Oh..the ole less convenient/more secure & vice versa mantra. There is a fine line there. But in recent yrs I’ve certainly changed my mindset around this and just tend to ‘bite the bullet’ to make my environment the most secure I can...within reason of course.

Thanks for sharing Michael. 

Userlevel 7
Badge +8

Good post and reminder.

I’ve seen people leave SSH on for all of the ESXI hosts, and every other device, then use a multi tab putty or terminal program and have their passwords preconfigured. GROSS!

 

Unfortunately every port will have something come up at some point. Think of how Windows XP, Windows 7, older unsupported software and hardware for a second. They all brag about new security features when they are released. At the time it’s great and things are patched. 

 

Another thing to consider is sometimes an App, OS, or hardware device has to be kept around for legacy reasons or just falls out of scope/sight. Attackers only have to be right once is such a good thing to remember. 

 

Even if I can prevent most of this in firewalls, the extra layer is helpful. I often worry about how easy a user can click a bad link, install a file and have someone remote access them. 

 

MFA and locked down is going to be the only way soon, even if it isn’t convenient for admins. 

 

Userlevel 7
Badge +6

Good post and reminder.

I’ve seen people leave SSH on for all of the ESXI hosts, and every other device, then use a multi tab putty or terminal program and have their passwords preconfigured. GROSS!

 

Somewhat guilty of this.  I tend to leave SSH on for ESXI hosts because I’ve had multiple instances where the GUI has become unresponsive and SSH was the only access into host without physical access to the console so that I could restart services, etc. 

With that said, those days seem to be way less often with newer, more stable builds of ESXI.  And, if you’re patching regularly, you’re a lot less likely to run into this sort of issue.  Most of the instances where I saw such an issue were on ESXI 5.x and 6.x, and usually on hosts that people forgot about for WAY too long and had uptimes of 1-3 years which while impressive, also tells me something about your sense of security and regular maintenance.

Userlevel 7
Badge +8

Good post and reminder.

I’ve seen people leave SSH on for all of the ESXI hosts, and every other device, then use a multi tab putty or terminal program and have their passwords preconfigured. GROSS!

 

Somewhat guilty of this.  I tend to leave SSH on for ESXI hosts because I’ve had multiple instances where the GUI has become unresponsive and SSH was the only access into host without physical access to the console so that I could restart services, etc. 

With that said, those days seem to be way less often with newer, more stable builds of ESXI.  And, if you’re patching regularly, you’re a lot less likely to run into this sort of issue.  Most of the instances where I saw such an issue were on ESXI 5.x and 6.x, and usually on hosts that people forgot about for WAY too long and had uptimes of 1-3 years which while impressive, also tells me something about your sense of security and regular maintenance.

 

 

From our conversations I know you are a security aware guy and a great tech so don’t take this as a lecture 😉

 In my opinion, A VMware host should not lock out the GUI. If it does, that’s a VMware issue and a support ticket. Sure, things happen, but I’ll create an outrage to fix it and it isn’t my fault as it’s not part of the designed intent. If I enable SSH and we get hacked that becomes my fault for leaving it.  This becomes multiple benefits as you are more secure, and can pass the blame to C.Y.A.   I have backups of my storage incase it was to break, but don’t walk around thinking my array is going to fail. I have backups of my VM’s, but don’t expect VMware to crash either.  


When I started at this place, 2000 days uptime was common. Impressive, but scary.  Now, I won’t release a server to a team, or for a project until a reboot and update schedule is in place. People get frustrated but I can now say ALL of the servers now have regular updates and patching. (Pats self on back😀) 

 

You are 100% about keeping things patched and updated creating less issues too.   When Everything is updated, upgrades are easy, when everything is down-level those product matrix’s get tough.

When you have to check the storage, HBA’s, Server hardware, ESXi, vCenter, OS, Tools, SRM, Horizon, Applications, etc. having something several years old can block everything.

 

Another “when I started here” rant. NOTHING had been updated as it was running just fine.  I had pages and pages of spreadsheets and drawings laying out the firmware/software and compatibility as some items needed 4+ updates to get to the latest version. All it takes is a single system that is no longer getting an update to stop the whole train and require hardware replacements / capital budget to move forward. (I won’t get into the hardware lifecycle and budgeting at that time 😅)  Regular patching would have caught that system going end of life. 

Userlevel 7
Badge +8

Love it!

I always have the same discussion (positive and funny)

what is the best firewall or security policy?

Turn off the Router, when not needed, and you should be good to go!

and if its a server, unplug the ethernet cable!

🤣

For security and simplicity, I totally agree with your entry.

cheers.

Userlevel 7
Badge +20

Thanks everyone for the comments. Just to hone in on a particular point you make @Scott regarding patching schedules & configuration. I was the lead IT person at an organisation for 5 years, and within 9 months of my departure, they got ransomware’d. The reason? They stopped patching the month after I walked out the door! Cultural adoption is the key to continued success. (Btw I’m not preaching at you Scott, just sharing a personal experience!).

 

I got contracted back in to help them recover as my replacement didn’t understand the architecture (I was contracted part time after I left for 3 months to have a weekly catch up with my replacement to allow them to ask any questions that might be business process vs technology or not in the documentation) but as they didn’t take notes and make the most of this time, they set themselves up for failure.

Userlevel 7
Badge +8

Thanks everyone for the comments. Just to hone in on a particular point you make @Scott regarding patching schedules & configuration. I was the lead IT person at an organisation for 5 years, and within 9 months of my departure, they got ransomware’d. The reason? They stopped patching the month after I walked out the door! Cultural adoption is the key to continued success. (Btw I’m not preaching at you Scott, just sharing a personal experience!).

 

I got contracted back in to help them recover as my replacement didn’t understand the architecture (I was contracted part time after I left for 3 months to have a weekly catch up with my replacement to allow them to ask any questions that might be business process vs technology or not in the documentation) but as they didn’t take notes and make the most of this time, they set themselves up for failure.

That is frustrating that they didn’t take notes.  We now have a dedicated security guy which takes some of my load off, but I’m usually the one insisting on the patching as if/when it does happen, it becomes the server/backup guys problem. With our volume of data I don’t want to be restoring that much data and getting things back to a workable state.

 

Uptime is the LAST thing I want to see in this day and age :) 

 

I think the cultural adoption at my place of work has changed drastically since I arrived and now people give me a maintenance window most of the time when they ask for a server as they know I’ll be asking them anyways. Education to the end users about security has been beneficial as well.

Userlevel 7
Badge +6

That is the single thing that I find the most irritating.  I love teaching people...part of my role is training my engineers (although I’m lacking in that department because of time constraints).  But if someone ask’s me a question, and then they ask again, by about the 3rd time I’m pretty irritated because if they aren’t retaining this knowledge in their head, they’d damned well better be writing it down to reference, and at that point if you’re not, you’re wasting both my time and your own.  If you care enough to ask the question, care enough to note the answer.  Show some initiative and grow your knowledge. 

As someone who loves to be a sponge for knowledge, I at times have a hard time understanding folks that don’t want to gain knowledge and simply want the answer enough to do the job once and be done with it.  That’s on of the key indicators for me of Administrators/Engineers that I want to invest time into developing.  It goes back to the 80/20 rule for me...put 80% of your effort and resources into the top 20% of your people.  And that top 20% of your people will give you 80% of your production.

Userlevel 7
Badge +8

That is the single thing that I find the most irritating.  I love teaching people...part of my role is training my engineers (although I’m lacking in that department because of time constraints).  But if someone ask’s me a question, and then they ask again, by about the 3rd time I’m pretty irritated because if they aren’t retaining this knowledge in their head, they’d damned well better be writing it down to reference, and at that point if you’re not, you’re wasting both my time and your own.  If you care enough to ask the question, care enough to note the answer.  Show some initiative and grow your knowledge. 

As someone who loves to be a sponge for knowledge, I at times have a hard time understanding folks that don’t want to gain knowledge and simply want the answer enough to do the job once and be done with it.  That’s on of the key indicators for me of Administrators/Engineers that I want to invest time into developing.  It goes back to the 80/20 rule for me...put 80% of your effort and resources into the top 20% of your people.  And that top 20% of your people will give you 80% of your production.

 

 

I’m working at keeping my stress levels down as sometimes my place of work can be VERY stressful. I always like to turn things into a positive, or learning experience these days.  I feel the exact same way as I take notes, as I personally don’t want to have to ask someone the same thing twice, but you can think of it like this.

 

I’d rather someone ask me something 5 times, than NOT ask me and take something down, delete something critical (like a 30TB volume of data), or do something to cause me extra work.  What doesn’t fly well with me, is if you ask me something, I give you a specific answer, and you go do something else anyways. Especially when the outcome is bad.  I had a coworker at a previous job who used to call every other tech for advice, and use the answer that provided him the least amount of effort. It didn’t take long for us all to find out what he was up to.  I understand if the advice you get needs confirmation, but when 4 guys tell you something, just go do it as if the 5th says something else, we are all going to ask questions haha. 

Userlevel 7
Badge +9

Thanks everyone for the comments. Just to hone in on a particular point you make @Scott regarding patching schedules & configuration. I was the lead IT person at an organisation for 5 years, and within 9 months of my departure, they got ransomware’d. The reason? They stopped patching the month after I walked out the door! Cultural adoption is the key to continued success. (Btw I’m not preaching at you Scott, just sharing a personal experience!).



> but as they didn’t take notes and make the most of this time, they set themselves up for failure.

BTW, they can easily forget any knowledge transfer without taking notes. Not sure what organisation this is as there are regulations that mandate proper documentation of processes, security measures, knowledge transfer etc. In the event of an issue such as the ransomeware attack, how can they demonstrate that they took reasonable steps to prevent issues.

Userlevel 7
Badge +9

Thanks everyone for the comments. Just to hone in on a particular point you make @Scott regarding patching schedules & configuration. I was the lead IT person at an organisation for 5 years, and within 9 months of my departure, they got ransomware’d. The reason? They stopped patching the month after I walked out the door! Cultural adoption is the key to continued success. (Btw I’m not preaching at you Scott, just sharing a personal experience!).

 

I got contracted back in to help them recover as my replacement didn’t understand the architecture (I was contracted part time after I left for 3 months to have a weekly catch up with my replacement to allow them to ask any questions that might be business process vs technology or not in the documentation) but as they didn’t take notes and make the most of this time, they set themselves up for failure.

That is frustrating that they didn’t take notes.  We now have a dedicated security guy which takes some of my load off, but I’m usually the one insisting on the patching as if/when it does happen, it becomes the server/backup guys problem. With our volume of data I don’t want to be restoring that much data and getting things back to a workable state.

 

Uptime is the LAST thing I want to see in this day and age :) 

 

I think the cultural adoption at my place of work has changed drastically since I arrived and now people give me a maintenance window most of the time when they ask for a server as they know I’ll be asking them anyways. Education to the end users about security has been beneficial as well.

I love how you fought for this adoption. If there were resistance by some users, how did you deal with it?

Userlevel 7
Badge +8

Thanks everyone for the comments. Just to hone in on a particular point you make @Scott regarding patching schedules & configuration. I was the lead IT person at an organisation for 5 years, and within 9 months of my departure, they got ransomware’d. The reason? They stopped patching the month after I walked out the door! Cultural adoption is the key to continued success. (Btw I’m not preaching at you Scott, just sharing a personal experience!).

 

I got contracted back in to help them recover as my replacement didn’t understand the architecture (I was contracted part time after I left for 3 months to have a weekly catch up with my replacement to allow them to ask any questions that might be business process vs technology or not in the documentation) but as they didn’t take notes and make the most of this time, they set themselves up for failure.

That is frustrating that they didn’t take notes.  We now have a dedicated security guy which takes some of my load off, but I’m usually the one insisting on the patching as if/when it does happen, it becomes the server/backup guys problem. With our volume of data I don’t want to be restoring that much data and getting things back to a workable state.

 

Uptime is the LAST thing I want to see in this day and age :) 

 

I think the cultural adoption at my place of work has changed drastically since I arrived and now people give me a maintenance window most of the time when they ask for a server as they know I’ll be asking them anyways. Education to the end users about security has been beneficial as well.

I love how you fought for this adoption. If there were resistance by some users, how did you deal with it?

Refused to move forward with the project, or server creation until I had a time and date for a maintenance window for patching and security updates. I had support from my supervisor as everyone agreed in MY area, but no one had taken such a hard stance ever.

 

If I have learnt 1 thing in IT, it’s once it’s in production, it’s much harder to change something. “We’ll sort that out after” means it will never get done. I have been burnt by that which caused me to draw a line in the sand.

 

Now if they want a server, it’s actually part of the request form I created.  There is a submit button at the bottom that emails a PDF to the service desk and forwards me a copy to keep records. It has things like what the server is for, who needs what access, CPU, Memory, storage requirements. Networking info etc. Once I get it I have the ability to add and change things too base off the ACTUAL vendor documentation of what’s needed in the VM :).  

 

The sneaky part is I designed it the submit button doesn’t work until you select a date, time, and timeframe in the maintenance window section that states update/reboot.  When I get a request now I just say “Sure fill out the form”  There are a few other fields, like backup frequency and retention I have people put in. If they choose no backup, or not enough, that is now logged in the intake request. Often I’ll discuss with them if they are asking for too much CPU/Memory, crazy RTO/RPO, SRM when it isn’t needed, or NVMe disk in the wrong spots, but so far it’s been amazing. 

 

 

 

 

Userlevel 7
Badge +9

Thanks everyone for the comments. Just to hone in on a particular point you make @Scott regarding patching schedules & configuration. I was the lead IT person at an organisation for 5 years, and within 9 months of my departure, they got ransomware’d. The reason? They stopped patching the month after I walked out the door! Cultural adoption is the key to continued success. (Btw I’m not preaching at you Scott, just sharing a personal experience!).

 

I got contracted back in to help them recover as my replacement didn’t understand the architecture (I was contracted part time after I left for 3 months to have a weekly catch up with my replacement to allow them to ask any questions that might be business process vs technology or not in the documentation) but as they didn’t take notes and make the most of this time, they set themselves up for failure.

That is frustrating that they didn’t take notes.  We now have a dedicated security guy which takes some of my load off, but I’m usually the one insisting on the patching as if/when it does happen, it becomes the server/backup guys problem. With our volume of data I don’t want to be restoring that much data and getting things back to a workable state.

 

Uptime is the LAST thing I want to see in this day and age :) 

 

I think the cultural adoption at my place of work has changed drastically since I arrived and now people give me a maintenance window most of the time when they ask for a server as they know I’ll be asking them anyways. Education to the end users about security has been beneficial as well.

I love how you fought for this adoption. If there were resistance by some users, how did you deal with it?

Refused to move forward with the project, or server creation until I had a time and date for a maintenance window for patching and security updates. I had support from my supervisor as everyone agreed in MY area, but no one had taken such a hard stance ever.

 

If I have learnt 1 thing in IT, it’s once it’s in production, it’s much harder to change something. “We’ll sort that out after” means it will never get done. I have been burnt by that which caused me to draw a line in the sand.

 

Now if they want a server, it’s actually part of the request form I created.  There is a submit button at the bottom that emails a PDF to the service desk and forwards me a copy to keep records. It has things like what the server is for, who needs what access, CPU, Memory, storage requirements. Networking info etc. Once I get it I have the ability to add and change things too base off the ACTUAL vendor documentation of what’s needed in the VM :).  

 

The sneaky part is I designed it the submit button doesn’t work until you select a date, time, and timeframe in the maintenance window section that states update/reboot.  When I get a request now I just say “Sure fill out the form”  There are a few other fields, like backup frequency and retention I have people put in. If they choose no backup, or not enough, that is now logged in the intake request. Often I’ll discuss with them if they are asking for too much CPU/Memory, crazy RTO/RPO, SRM when it isn’t needed, or NVMe disk in the wrong spots, but so far it’s been amazing. 

 

Great efforts @Scott! The take away here is this “I had support from my supervisor“. 

Userlevel 7
Badge +6

If I have learnt 1 thing in IT, it’s once it’s in production, it’s much harder to change something. “We’ll sort that out after” means it will never get done. I have been burnt by that which caused me to draw a line in the sand.

 

 

The one phrase I’ve learned, and it somewhat applies here, is nothing is more permanent that a temporary solution that works.  Once something is in production, it’s pretty hard to roll it back out with an actual permanent solution. 

 

Now if they want a server, it’s actually part of the request form I created.  There is a submit button at the bottom that emails a PDF to the service desk and forwards me a copy to keep records. It has things like what the server is for, who needs what access, CPU, Memory, storage requirements. Networking info etc. Once I get it I have the ability to add and change things too base off the ACTUAL vendor documentation of what’s needed in the VM :).  

 

The sneaky part is I designed it the submit button doesn’t work until you select a date, time, and timeframe in the maintenance window section that states update/reboot.  When I get a request now I just say “Sure fill out the form”  There are a few other fields, like backup frequency and retention I have people put in. If they choose no backup, or not enough, that is now logged in the intake request. Often I’ll discuss with them if they are asking for too much CPU/Memory, crazy RTO/RPO, SRM when it isn’t needed, or NVMe disk in the wrong spots, but so far it’s been amazing. 

 

This sounds like a pretty great workflow to be honest.  I think we’ve had a similar form in the past at a previous job, but it was some sort of webform or something like that...wasn’t a PDF.  But yeah, having those required fields are key.  

Also, I’m not sure that uptime is the LAST thing I’d want, but your point is very much made.  Reliability is required, but we sure don’t want to see months or years of uptime anymore unless your patching can always be done as a hotfix.  I love that you’ve been able to change the culture at your workplace to one of sanity,  Seems like a lot of people can’t get the full buy-in on these sorts of things and have to struggle with those pains of pushback, so I can sure appreciate that you’ve got a system that works and people are following it.  That’s pretty rare really.

Comment