Root Causes 221: What Are SSH Keys?

Hosted by

Tim Callan

Chief Compliance Officer

Jason Soroko

Fellow

Original broadcast date

May 5, 2022

SSH (Secure Shell) keys are ubiquitous for authenticated access to Linux systems. In this first of three episodes we explain what these keys are and how they're used.

Podcast Transcript

Lightly edited for flow and brevity.

Tim CallanUltimately, what we want to talk about is what are SSH certificates? This is a new certificate type, newish, but really garnering a lot of attention right now, and I think maybe we want to walk through and just make people understand what they are.

Jason SorokoThat’s right, Tim. I think it was your proposal to break this into three podcasts, which I think this first one today, right now, we’re going to just get right in to what is Secure Shell, what is SSH for those of you who haven’t heard of it. Because I’ll tell you, I don’t know too many enterprises who don’t use it somewhere today. Especially if you’re using any kind of cloud service, if Linux exists anywhere in your enterprise, you probably are using this somewhere. You have some administrator who is using this. May not talk about it might not be at top of mind in terms of you running your business, but it’s there, and I think you need to know about it, and I think you need to know about the innovative changes for how to do SSH, which are going to be a lot more manageable and look a lot more like your typical PKI governance system, Tim. That’s where we’re going to get to by the time we get to the end of the third podcast.

Tim CallanI love it, and just to – this maybe – I may be going out of order on this, but I’ll just say this. You sort of touched on the idea that the person who is listening to this podcast who oftentimes is somebody who’s really in charge of the traditional certificates and the certificate lifecycle management CLM in their company, this may not be on their radar, but if this isn’t on your radar, you need to get it on your radar because this is a basic Authentication tool, it is something that people are using for very critical systems, and if it’s kind of going on Skunk Works, Shadow IT inside the development department, then that is definitely a risk. And after we walk through it, I think you’ll see why that’s the case.

Jason SorokoSo, where we’re going to start here is fairly higher level, it’s the what it is, and so therefore, if you are a Linux administrator who uses OpenSSH, uses SSH in general, I would say, click off now, save your time, and tune into Episode 2 because that’s where we’re really going to get into what are the problems with traditional SSH, which is so, so prevalent and ubiquitous, Tim. I cannot state how much it just is used out there. And then by Episode 3, we’re really going to double down on SSH certificates specifically and why they’re just so great.

Tim CallanAgreed. So, for starters, just for everybody’s background, set us all at a level, set of knowledge here. SSH. Definition of SSH.

Jason SorokoSSH stands for Secure Shell and, I could say something real blunt like it’s for remote access to Linux systems, especially in the cloud. But what does that really mean? The term “Secure Shell” should really tell you what it is. I think it’s the word “shell” that hangs up non-hardcore technical people. If you’ve ever watched a computer geek wizard type guy, woman, whoever it is, and they are typing on a computer furiously. Think about the movies. Quite often, they are typing into a shell, if they’re typing computer commands at all. And so it’s making computers do things, and quite often, and this is why it’s very important, Tim. Typically when you’re at the command line, when you’re at the shell of a computer system, you’re doing really critical things. You’re configuring that computer, you’re configuring a system, you could be doing something like, heck - how many times, Tim, have you and I talked about installing a certificate into a web server? We’ve talked about that an innumerable number of times in this podcast. That is done at the shell. That is done at the command line of typically some form of Linux system, if you’re running your web server on Linux, which is very, very common. And so think about that. That web server is probably not going to be your primary workstation computer. You might have a Windows machine or a Mac or something else. It’s typically not going to be a Linux server running a web server. Or at least, best practice is to say you really shouldn’t. So that web server is probably going to be remote. Even if it’s just in the next room, it’s still remote, and therefore, needs to be accessed over some form of network. But I would say in a modern age, it’s very common for that web server example, that Linux web server, to be off in the cloud somewhere.

So how do you get to the shell? How do you type in commands from your workstation computer to that other computer that’s remote to you? Well, you need to get to that command line. You need to be able to enter the commands but typing it from your own computer. And that’s what we mean by remote access. And what are you remotely accessing? You’re remotely accessing the shell. Now over the years, Tim, there’s been a lot of ways to do that, as you know. And back, I think it was 1995 when the idea of SSH first came about and was designed and eventually became what most people use today, which is OpenSSH, which is part of most Linux distributions, which essentially uses a key pair to be able to do this remote access. So think about the use case you and I talk about probably most commonly, which is SSL, SSL using TLS certificates, which essentially is putting – it’s using a key pair to be able to encrypt traffic between a web browser and a web server. In the case of Secure Shell, this is a protocol that is very specific that enables a client computer, your workstation, to securely communicate to the remote Linux server. So again, it’s a client server type of concept and similarly to the way SSL works, with an SSL certificate which includes a key pair, in the case of traditional SSH, it’s using only the key pair typically, it’s not using certificates. That’s something we’re going to get into by Episode 3 of this particular series of podcasts. We’re going to be talking about the problems of just using the public and private key pair, but I think that what’s most important to note is, we’re going to go right through why that has been problematic from Day 1. Even though this is so ubiquitous, it works very well, the problem is, I think it’s a golden age to be a bad guy, and I also think it’s because the reason why it’s just been typically raw key pairs for years and years with Secure Shell, it really comes down to the fact that managing the lifecycle of these identities that are created, this particular credential form factor of key pairs, the technology to automate that just hasn’t been there, from Day 1.

Tim CallanSo, not to put too fine a point on it, this is very similar to the key exchange we all understand from TLS. There is a server. The server has a public key and a private key. The server gives out the public key, and then based on that, the server can decrypt the messages using the private key and what that does, that establishes identity, and everybody knows who everybody is.

Jason SorokoExactly right, Tim. So, think about any system that does not have a certificate and does not have centralization. This is a raw key pair. Like, for example, Tim, I always use this analogy. If you and I wanted to do business together, one of the things that we could do is just agree upon a shared secret. We could say – come over to my house tonight, knock on my door three times, and I’ll know that’s the secret for – that’s you and I’ll open the door. Well, of course, we want to get away from shared secrets. We want to use asymmetric secrets. So we take the secret, we break it in half, we have one half of the secret that we keep very tightly to our chest, and we don’t ever give that away. What we do give away is the public key. So that public key exchange is very important, and it occurs in SSH upon the trust on first use. In other words, the very first time that I go to authenticate to a server with my client, there will be an exchange of public keys. Therefore, the next time we communicate with each other – terrific, we can make that a communication because when I knock on the door of that server, that server will challenge me, and because of the fact that we both have our public keys, we both are able to sign and encrypt each other’s communication between each other. It’s terrific. I think that’s probably the most important thing to note about where we’re going talking about SSH.

Tim CallanThis a venerable practice. This is something people have been doing for more than 20 years.

Jason SorokoOh, since 1999. Open SSH. This is more than 20 years, and it is, I think I’ve said it several times already. It is utterly, utterly ubiquitous. Trust me when I say, if your company runs a web server, if your company runs anything in the public cloud, if your company – like there’s very few enterprises or governments today that are so purely a Microsoft stack or so purely some other stack of technologies that you don’t have some distribution of Linux in there somewhere, it’s almost, it’s almost hard to imagine. I’m not going to say it doesn’t exist. I’m just saying, anybody who has a client server architecture of any kind, including – and I use web servers because it’s such an easy example to remember and know – that you probably have employees that you have hired that are using OpenSSH as part of their Linux distribution in order to be able to remotely access the command line of other computers in your enterprise. They are using this. Therefore, these key pairs are all over the place. That’s an important point, Tim.

Tim CallanThen the other point - and you’ve said this, but let’s just make sure we’re not being confusing on this in anyway - is, we talk a lot about key exchange and stuff, and we tend to sort of almost as short-hand, talk about the key exchange in a certificate scenario, like, let’s say TLS. In this case, all that’s still going on but it’s real important to understand there is a no certificate here. It is just raw keys. That is essentially important to this conversation.

Jason SorokoWe’re only going to touch on this now, because I don’t want to steal the thunder of the second podcast, but let’s call it out, Tim. Key pairs can be compromised. If somebody gets a hold of your private key, my goodness, that’s rough. Because all of a sudden the private key that might exist traditionally on your workstation, if that is compromised, the bad guy will be able to authenticate as you to the remote server with the same level of privileges that you’ve assigned to that credential.

That means that – and you could have the most terrific monitoring tools. You could have all kinds of security practices in place, but if that private key is compromised, you’re in a lot of trouble. Don’t forget, there’s a lot of ways that that key can be compromised. It’s not just somebody compromising an endpoint. You might have the human error factor, which is, alright I authenticate a lot. I need to build automation software. I’m going to go and make an ansible script and put in my private key.

As well, Tim, you’ve said a couple times now, let’s keep in mind this is not certificates-based. Certificates are synonymous with policy. A raw key pair has no policy by definition.

Tim CallanKeys have no policies. They have no rules governing them. They have no memory. They offer no visibility on how they’re used. Yes. And I too am trying so hard not to steal the thunder from the next episode, but all of these factors are going to be essentially important as we go through the trouble with SSH and introduce the concept of an SSH certificate.

Jason SorokoSo think about anything that is not securely managed that is essentially the keys to your kingdom. The things that have infinite lifespans, no expiry dates, no policies, no visibility. Tim, you and I have talked so much about the advantages to Certificate Lifecycle Management Certificates and all of that. You get none of that with traditional SSH.

Tim CallanAlright. So, that’s your teaser. We’re going to roll into the next podcast if we don’t stop ourselves.

Jason SorokoNo, I know. So, Tim, this is just so important because you’re talking about an utterly ubiquitous technology that unfortunately – the really unfortunate thing about these really good ubiquitous technologies that are proven from left to right, is that you're using them and they just make you feel secure. Unfortunately, there’s some missing steps to truly making them secure, and SSH is one of these candidate technologies. VPN - we had a whole podcast series on the problem with VPN because VPN makes people feel secure, and yet, there are problems with that thinking. SSH is another really truly classic security technology that has security issues that you need to close the gap on.

Tim CallanPerfect. And in our next episode we are going to do that. Maybe not our next episode on the podcast but our next episode in this series we are going to do that. So, please return to us. Stay tuned, and we’re going to talk about SSH certificates, the problems with SSH and how SSH certificates are emerging as a solution for that.

Stay informed with expert insights

Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!