Root Causes 224: The Five Problems with SSH Keys

Hosted by

Tim Callan

Chief Compliance Officer

Jason Soroko

Fellow

Original broadcast date

May 17, 2022

In this follow-on to our earlier episode explaining SSH keys, we discuss the five problems SSH keys present to organizations using them. And we give a peek at how to solve these problems.

Podcast Transcript

Lightly edited for flow and brevity.

Tim CallanThis is the second in our series of podcasts on SSH keys and SSH certificates. And in this case, we want to discuss the five problems with SSH keys.

Jason SorokoI think that we're going to be going through some what I could consider categories and under each of these categories, there's a ton of detail. We could literally do podcasts about bullet points under these categories, because - -

Tim CallanAnd we may later.

Jason SorokoAnd we may. It’s a big, big topic, but this is, this is sort of a capstone idea that everything underneath it kind of just flows out of these different categories. I think that you can even consider that there are other categories. We are sticking to the problems that are specifically related to the key pairs with regards to SSH, not all of the other issues that might be related things like networking, etc.

Tim CallanFair enough. And let me just real quick, say, we covered the basics of SSH key pairs in our Episode 221. So if you would like that background, by all means, go listen to that either before you listen to this or afterward. That's a good one. If you want a little more on SSH, that's a good one to listen to. So ok, with that, shall we? Shall we roll into the five problems?

Jason SorokoSure, Tim. I'm going to introduce the first one and make the big caveat for anybody technical who is listening right now. The biggest problem being unsafe user behavior. So I'm talking about specifically, a user of SSH being somebody who is using OpenSSH, as an example, as a client to some remote server. And those of you who hear that unsafe user behavior as name of a category might say to me, hey, I know how to use SSH, and I'm not using it wrong. Give me a break, right. What I'm going to tell you is anybody who would react that way, you had better show me your homework that you've done, because most SSH usage I've ever seen, and just about all SSH usage that I've ever done myself personally, makes a huge, huge assumption that you know exactly what remote server that you have accessed. So on your on your first usage - in other words, the idea of the trust on first usage, part of what is important about SSH is when that client connects for the first time, you don't have the public key of that other server and part of what the client does is to actually insert a note into your domains files that are, what are the domains that I'm actually trusting here. And so you had better be awfully darn sure that there's no man-in-the-middle attack going on. There's no other shenanigans that are going on to make you think, yes, this is the server that I want to trust, and that it may not be.

Tim CallanAnd after that, you’re just gonna keep right on trusting it.

Jason SorokoAnd in fact, Tim, this is what's interesting to me. The people who design OpenSSH know this and so there's a great big disclaimer warning. It’s actually just a straight up warning. But to me, it's almost like a disclaimer warning in the sense that, hey, you're about to take a big chance here by accepting this first communication because there's consequences. You’re accepting a public key, you're handing out a public key, and you're also making file change to your trusted domains list. And that big warning message most people just enter in yes, please proceed. And how do you know? And I think a lot of people who might be listening to this would say to me, Jay, I don't, but I gotta get on with my life. My response to that is, what if, what if, I were to tell you that there's a better technology, in terms of the key pair itself, because the problem is you and the destination server, the remote server itself, you don't know each other. You don't know each other yet. All you have is this domain, this domain information that you're trying to connect to? And that's all you've got. You have no other assurance. There's no central authority that has say issued a certificate with a policy saying, hey, I vetted out this and you guys can go and make a connection because, I know you guys are both trusted and you're part of the known infrastructure of this organization. There's none of that. And so I just wanted to point that out and say, what you, what all of y'all who do SSH might can be considered just normal behavior of well I just trust it, because I trust it, that's really bad practice. It's a bad user behavior. It's so bad that I think people just type in yes on the trust of first use warning message, because it's just, it's just what you do. It's just, there's nothing else to do. And yes, there are better technologies. But in terms of a traditional SSH problem summary, that's category one - unsafe user behaviors.

Tim CallanI think you did that really well. The other thing I would add is, even hypothetically, if somebody could make the case that they themselves, were not engaged in unsafe behaviors, are you the only person in your organization who is using SSH keys? Can you be confident that nobody else has any unsafe user behaviors? Can you be confident you even know exactly who is using these keys? I'm not even sure that we could get away with that.

Jason SorokoNo. This is the big problem, Tim, is that when you're using raw key pairs, you - which is so common. It's just so darn common. I think people have forgotten that, that's just the way it was set up. But it's not the best way of doing business. And in fact, I find it ironic that anytime there is PKI in an enterprise for authentication purposes, oh my goodness, there's just this enormous infrastructure, and there's all kinds of governance and there's a risk officer, and there's people who obsess over making sure the certs are just so just right, and they are handed out to the right places that are trusted. We could go on and on. In fact, many of our previous podcasts are about obsessing over how to do it right.

And all the mistakes that can be made. The pitfalls of PKI, etc. And yet, I would say OpenSSH, whereas PKI is a pretty specialist type of thing, OpenSSH is almost literally everywhere. I'm not joking when I say that. And yet, none of those scrutinies that we obsess over with PKI are there. And yet, to me, it's just the Wild West. And if you wonder why it's a golden age to be a bad guy right now, this is one of them.

Tim CallanVery good. Second one on the list, costly key tracking.

Jason SorokoSo Tim, you know a lot about tracking certificates. And we talk a lot about certificate lifecycle management on this podcast, because the visibility to your certificates, the visibility to the keys to the kingdom are something that is incredibly important to an enterprise. Hey, where are all my credentials that have a form factor that can be sitting as a digital file, and that can open a door? Well, if you think about an SSH key pair, as I said, it is so utterly, utterly ubiquitous, almost every enterprise has this and you just brought up the point of, do you even know who is using SSH in your enterprise?

Well, if you wanted to get into the business of having visibility to your keys, and you don't have the equivalent of a certificate lifecycle management for these raw crypto key pairs, how in the world are you tracking them? Well, the simple answer is most organizations are not tracking them at all.

And for those that sometimes do think that they're tracking them, Tim, this is one I've read countless times, and it makes me cringe bad guys or pen testers more often - hopefully, more often - will go into an organization and see almost the equivalent of a digital key rack right at the door.

Tim CallanIs this a spreadsheet that has them all sitting in cells? Is that what we're talking about?

Jason SorokoIt's very, very similar. It's very similar to that. And I think that it's just incredible to me that you can think that, hey, I'm gonna put all my private key files in this one central place, and we're good to go.

Tim CallanBecause these are raw keys, because there's no certificate enveloping it, you don't have any opportunity for metadata. So you have to know what the key is just by matching it to some other source that identifies any other information. This is when it was created. This is who it was issued to. Here is who is supposed to be using it, etc. And so this is what it's for. And so fundamentally, that means you have to have a repository that has that information.

Jason SorokoTim, it's incredible that a certificate, key pair wrapped within policy can be tied to a device, could be tied to a person. Almost by definition, it has these properties and through certificate lifecycle management, all the metadata, all the context, all the usage is noted and you have full visibility to it and you can automate all of that. With these key pairs, you have nothing and you're inventing key management and unfortunately, some of the inventions that I've seen to do basic key management are of advantage to the bad guy. And therefore, it's the worst case scenario. It's just, I gotta say it, I gotta say it. OpenSSH, and SSH itself been around since the mid to late 1990s, it's utterly ubiquitous. It's used everywhere. And it is just a bunch of doors to very important locks that are just lying around. It's incredible.

Tim CallanSo you touched on our third one, just a little, a second ago and that is - does not scale.

Jason SorokoSo if you take this problem of a lot of keys sitting around and you don't know where, what they're for, you don't know where they are, you have no visibility, well, the problem then is, if you are, Tim, using something like a spreadsheet to keep all the keys in one place, and to store some of the metadata, well, you and I both know that as soon as systems get large, keeping track of things by spreadsheet, or whatever method you're using, it simply will not scale to the size, to the amount of keys that you're using. I would say even medium-sized organizations, medium-sized companies are going to have so many of these keys used for various things that it's just it's not going to scale and for large organizations forget about it. It would be insane.

Tim CallanOn the bright side, I guess you don't have to worry about unexpected expirations but we'll come back to that.

So number four - loss of control.

Jason SorokoI'm going to use a real world example to talk about what we mean by loss of control and you could probably stuff this under unsafe user behaviors. But it's so, so bad that it requires its own category.

And that is, Tim, if you want to be authenticating into systems, what happens if it's not just a person authenticating? What happens if you're using an automated system to authenticate into remote servers? Well, I'll tell you that Secure Shell SSH is phenomenal. Just fantastic to be able to do things securely through that encrypted pipe. Just so great. It's the reason why SSH is used everywhere. Well, unfortunately, it's so good that if you if you are creating some sort of automated system that needs to have that authenticated SSH session, you might actually say, I'm going to take my private key, and I'm just going to embed it in the code so that it can go off and authenticate all by itself. And you might laugh and say, oh, nobody would ever do that. Well, go to your favorite web browser search engine and look that up and you will see tons of examples of this being found in the public domain. In other words, GitHub repositories, just, loaded with SSH private keys. And so thankfully, thankfully, there is a tool that's available to do a best practice of where to put your keys, where to put your secrets, and where to hide them in a secure manner, and use them in an automated way and those things are called credential vaults. This is a place to put key material and usernames and passwords. So it's not just being a password engine, it is a place to put a lot of your secrets. You can call it by an API. You can call it by a number of different methods and so that when you're automating authentication to various remote sites with your SSH keys, you can then use that to be able to retain control over those keys. But for now, because of this ubiquitous nature, there's still a lot of, well, I'll call it what it is. It's unsafe user behavior that’s so egregious that I think this category of loss of control needs to be called out.

Tim CallanAnd of course, because they're just keys, they don't have any controls built into them. There are no policies. There's nothing you can do to prevent this key from being used this way, because it just does what it's supposed to do cryptographically. That's all that it is.

Jason SorokoThat's it, Tim. And I think that leads to our fifth and final category, which is really all about, it's all about our favorite policy.

Tim CallanWhich is - never expires.

Jason SorokoIn other words, SSH key pairs don't expire. Once that key pair has been generated, and you're using it to establish nice authenticated, secure sessions between your client and whatever remote server you want to be able to type on the command line securely, that key pair will live forever. Whether you know it or not, whether you intend it to or not, it doesn't matter. Once that has been created, if the bad guy gets a hold of the necessary key material, they can then use that to authenticate all they like as long as they like, and there isn't even a mechanism for you to say, hey, I need to get rid of that.

Tim CallanThere’s no revocation. Yep, absolutely.

Jason SorokoSo, obviously, those of you who are technical know, from the server side, I could do certain things to not have that key basically be used. But who does that? This is this is the problem is that it's so - -

Tim CallanWell, and how do you know there. I mean, it goes back to the same thing, just like if say, well, all of my user behavior is perfect. Well, how do you know that other people aren't using keys? How do you know that that private key wasn't shall I say, borrowed for some other purpose that wasn't its original intended purpose and wasn't the one you know about?

Jason SorokoI love that you brought up that idea of metadata, Tim. You don't know any of the context of the keys that are listed.

And because of that, doing what we consider revocation in the world of PKI and certificates, you just don't have that in SSH raw key pairs.

Tim CallanYou don’t have it. It’s just math. The math doesn't stop working. And so the public key and private key are always going to do their math correctly as long as that private key is in place somewhere then that public key can be used against it.

Jason SorokoThe best physical analogy you can come up with is literally a key that is cut for a door. And then how do you control the fact that that key isn't cut multiple times? And then handed out to all kinds of people you don't want to have holding those keys? Well, how do you deal with that? Well, you go and physically change the locks so that all the keys are no longer used.

Tim CallanBut even then, in this scenario, it could be that somebody else out of convenience sake, or for whatever reason, decided to use that exact same key for a different lock. And you wouldn't know. You can’t know.

Jason SorokoPrecisely. This is where the analogy breaks down because it's far more complicated than just changing the lock itself. First of all, it's very, very rarely done in the real world in IT. And secondly, as you say, Tim, all of a sudden, you don't, you don't even know what the locks are and what the keys are, you just don't have visibility to any of the context of the usage, and telling me that you could tell the difference between malicious and legitimate keys and usage – you simply can't.

Tim CallanSo doesn't expire, number one, and expiration forces all this stuff to get cycled out. The other one is expiration forces crypto agility. So even if nobody did something bad with my private key, if I stuck a key on some system that I created in the 1990s, and it's still there, then your key length is probably woefully inadequate to today's computing power anyway.

Jason SorokoFun fact, Tim. I believe that OpenSSH, just, just, just, I shouldn't say just, just but it within the reasonable amount of time in the past, I think within a year or two, SHA-1 was finally deprecated as an option for when you're generating these SSH keys. So think about that. Even after SHA-1 was long deprecated, because of the fact of the need for backwards compatibility for the enormous number of long lived key pairs, they had to maintain this backwards compatibility long, long after the deprecation.

Tim CallanAlright. So those are the five - unsafe user behaviors, costly key tracking, does not scale, loss of control, and never expires. Once again, check out our Episode 221 if you want a little more background on this, and stay tuned for a podcast to be created in the future, where we will talk about SSH certificates and how they address these concerns.

Jason SorokoThat's the important point. Tim and I always like to bring up all these problems and then we break down, hey, how can we solve these problems? What is the bright light? And I do believe that the bright light is coming as a trend for SSH, and that is to wrap these key pairs into policies and let's call that wrapped key pair a certificate and boy, does it ever make a difference in solving a lot of these problems?

Tim CallanExcellent. And we will get into all of that in that future podcast.

Stay informed with expert insights

Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!