Root Causes 297: Certificate Expiration Creates Starlink Outage

Hosted by

Tim Callan

Chief Compliance Officer

Jason Soroko

Fellow

Original broadcast date

April 26, 2023

A recent outage in the Starlink internet service was caused by an unexpected certificate expiration. We discuss this ongoing problem and how 90-day maximum certificate term will exacerbate it.

Podcast Transcript

Lightly edited for flow and brevity.

Tim CallanAs so often occurs in the world these days, this story starts with a tweet by Elon Musk.

Jason SorokoTweets from Elon Musk. Well, they are always interesting to read.

Tim CallanYes. There’s always a lot.

Jason SorokoThat’s where I get all my investing advice and etc.

Tim CallanRight. A lot can come out of a tweet by Elon Musk. This one is from April 7, 2023 and the first tweet reads, “Sorry, slight glitch with the SpaceX Starlink. Coming back online now.” And then there was a follow up shortly thereafter that reads – also by Elon – “Caused by expired ground station cert. We are scrubbing the system for other single-point vulnerabilities.” So, of course, this is something that you and I always notice when these kinds of outages occur, but in this case, Starlink is taken down by an expired certificate in the ground station. And, Jason, I think you felt this one personally. Did you not?

Jason SorokoI did. I did. And in fact, I didn’t even see Elon Musk’s tweet, mostly because of the fact that I didn’t have access to the internet. I am a customer of Starlink and I have one of his satellite dishes at a rural property that I happen to have and I depend on that because of the fact that there really is no other kind of internet service out there and I enjoy the service. So there it is. The problem is, of course, when it goes down, it goes down. A lot of times it’s due to who knows, maybe there’s wind that’s shaking my satellite dish or some other crazy thing. But in this case, the outage was a lot longer than what was normal and myself and a few family members were kind of looking at each other going, geez, I hope this thing comes back. And sure enough, it did. And then it wasn’t that much long after, Tim, when you shared that tweet with me and I was like, my goodness, I know that outage. It affected me very personally and what was it, Tim? What was it? An expired certificate.

Tim CallanAn expired certificate. In April of 2023, an expired certificate. That was probably an SSL certificate. I guess we can’t know for sure, but an expired ground station cert, you would imagine that’s what it is. Connecting to something else sitting on a server, connecting up to the satellite itself and then, of course, this ties directly back to a topic that I think is just gonna come up over and over and over again, is this is still in the era where people can get a 398-day cert. What’s gonna happen to situations like this once we are down to 90-day certificates?

Jason SorokoYeah. You are exactly right, Tim. Some people could say, hey, who knows? Maybe it was a private TLS certificate we are talking about here, but it doesn’t matter. That had some kind of an expiry date that was not captured. The certificate was not renewed on time and guess what? It affected people. It even affected me. A lot of times when we talk about these things, Tim, you know, sometimes it’s European outages or people are in gaming systems. I don’t have time to game, so it doesn’t affect me. This one affected me and so to me, I don’t care. I don’t care what kind of cert it was. It expired and it wasn’t handled. And isn’t it amazing, Tim, Elon Musk, with all his technical resources, all the people who work for him, you know, his SpaceX, very tech leaning company and…

Tim CallanYes. SpaceX is not a technically naïve company. This isn’t your local law firm that has some IT infrastructure because they just have no choice. This is a company that is born in technology. It is about advanced technology and it is led and founded by somebody who is all about advanced technology, and even here we have this kind of problem.

Jason SorokoThis is a company and a person who has the capabilities of regularly launching hundreds of satellites into space and then having the spaceship come back to Earth and land on its feet. But they cannot renew their certificates. Wow!

Tim CallanSigh. So, how long are we gonna keep telling this story, Jason?

Jason SorokoWe talked about this just before we came onto this podcast and said, ah, just another one of these, you know, expired cert stories but the difference being, of course, it affected me and I hate to say it but here we are – April 2023. You and I looked into our crystal ball and predicted we’d see more of these in the future. Here it is. We are seeing these again.

Tim CallanHere it is.

Jason SorokoAnd, Tim, I’m gonna throw it in just because I think this is the call to arms and the big red light warning. If you haven’t heard it on previous podcasts, Google is probably going to at some point, either passed through CA/Browser Forum or force it themselves, maximum of 90-day certificate lifespans for SSL certificates, for publicly trusted certificates. Which means this kind of site outage due to an expired cert is now gonna be running the risk not just once every 398 days but every 90 days. I can’t imagine.

Tim CallanAlright. So, Jason, I’m gonna take a little side jog here and you and I are gonna both make a prediction right now. I’m gonna put you on the spot. And for what it’s worth, I haven’t thought about this at all. So, we are both in the same boat because I thought of this right this second. It is the middle of April. We are going to do almost two podcasts a week for the rest of the year. So, let’s say that is roughly 65 podcasts. That’s probably the ballpark. What’s your over/under? Think of a number. I’ll think of a number and we’ll both say on how many podcasts we will mention 90-day certs between now and the end of the year.

Jason SorokoI’d be surprised if it wasn’t at least -

Tim CallanOk. I got a number

Jason SorokoSure. I’d be surprised if it wasn’t at least three. It’s gotta be at least three in that many.

Tim CallanAt least three podcasts? Between now and the end of the year?

Jason SorokoThat’s right.

Tim CallanMy number is 45. I think we’re gonna mention it almost every episode. That’s my prediction. I think we are gonna mention it two-thirds of episodes for one reason or another. My number is 45. So, we’ll see. Now of course, this is bias now because I can just make sure I mention it 45 times. It’s unfair.

Jason SorokoMy problem is that I would like to maintain our listenership who doesn’t want to just tune in and listen to the latest outage. But the reason why we wanted to bring up this one - -

Tim CallanNo. Not outages. 90-day certs.

Jason SorokoOh. How many are you gonna hear about 90-day certs?

Tim CallanI predict we will mention it 45 times.

Jason SorokoOh. My apologies. I heard you wrong. Yeah. So, I would say at least three major outages by the end of the year.

Tim CallanSure. I would agree with that.

Jason SorokoI would say though that, geez, how many times are we gonna mention 90-day? I’m gonna say - - I’ll tell you what. I’ll be really generous to the audience here and say I bet you it’s 35 to 40. Let’s give it 40.

Tim CallanAlright. So, we are predicting between 30 and 45. I think that’s a reasonable range because this is the story. And the point I’m going to with this – not to belabor it, this is the story. Like, this came out of nowhere. It is the story for the rest of this year. It is definitely the story in the next year for some period of time until it becomes reality, at which point it continues to be the story. So, this is gonna be the story for years to come.

Jason SorokoIt is going to be. Yeah.

Tim CallanAnd we are seeing it just showing up over and over again.

Jason SorokoYou are not wrong at all, Tim. I completely agree. And, in fact, for those of you who hear us on the next 30, 40 episodes and mention it, it’s gonna be on purpose. It is absolutely on purpose. You need to hear more about this than you realize and until it totally sinks in to everyone, we are going to continue to talk about this because we do not want to enter a time in the deeper future, 2024, right, next year – we would love for people to have a soft landing with 90-day certificates and if we don’t push it hard now, it’s not gonna be a soft landing.

Tim CallanAnd I think there’s that point about educating. I think that’s a very valid point, Jason. The other thing about you and me, we kind of like to analyze and explore. I do not believe that the industry or the IT community actually has wrapped its head around the full set of implications of this change.

Jason SorokoCorrect.

Tim CallanAnd I think that one of the reasons you and I are gonna discuss it a lot this year is we are gonna take different angles on it. Think of all the different angles we’ve had on Post-Quantum Cryptography. Right? Say, well, what about the Z-date and what does harvest and decrypt really mean and is China gonna outspend the U.S. and all these different kind of sub-angles that we took. I think we are gonna see the same thing with 90 days. We are gonna keep thinking of new implications and new reasons why it’s important and as we do, we are gonna want to explore them and that means we are gonna be returning this topic over and over again.

Jason SorokoWe absolutely will. So, stay tuned, folks. The 90-day announcement from Google, go back to a couple of previous podcasts that we have published on this – you are gonna hear a lot more.

Tim CallanYep. So, this was a certificate outage reported by Elon, impacted Jason who is among the most technically astute people I know, so if you can be taken down by a cert expiration, Jay, I think anybody can.

Stay informed with expert insights

Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!