Podcast
Root Causes 297: Certificate Expiration Creates Starlink Outage
Hosted by
Jason Soroko
Fellow
Original broadcast date
April 26, 2023
A recent outage in the Starlink internet service was caused by an unexpected certificate expiration. We discuss this ongoing problem and how 90-day maximum certificate term will exacerbate it.
Podcast transcript
Lightly edited for flow and brevity.
TCTim CallanAs so often occurs in the world these days, this story starts with a tweet by Elon Musk.
TCTim CallanYes. There’s always a lot.
TCTim CallanRight. A lot can come out of a tweet by Elon Musk. This one is from April 7, 2023 and the first tweet reads, “Sorry, slight glitch with the SpaceX Starlink. Coming back online now.” And then there was a follow up shortly thereafter that reads – also by Elon – “Caused by expired ground station cert. We are scrubbing the system for other single-point vulnerabilities.” So, of course, this is something that you and I always notice when these kinds of outages occur, but in this case, Starlink is taken down by an expired certificate in the ground station. And, Jason, I think you felt this one personally. Did you not?
TCTim CallanAn expired certificate. In April of 2023, an expired certificate. That was probably an SSL certificate. I guess we can’t know for sure, but an expired ground station cert, you would imagine that’s what it is. Connecting to something else sitting on a server, connecting up to the satellite itself and then, of course, this ties directly back to a topic that I think is just gonna come up over and over and over again, is this is still in the era where people can get a 398-day cert. What’s gonna happen to situations like this once we are down to 90-day certificates?
TCTim CallanYes. SpaceX is not a technically naïve company. This isn’t your local law firm that has some IT infrastructure because they just have no choice. This is a company that is born in technology. It is about advanced technology and it is led and founded by somebody who is all about advanced technology, and even here we have this kind of problem.
TCTim CallanSigh. So, how long are we gonna keep telling this story, Jason?
TCTim CallanHere it is.
TCTim CallanAlright. So, Jason, I’m gonna take a little side jog here and you and I are gonna both make a prediction right now. I’m gonna put you on the spot. And for what it’s worth, I haven’t thought about this at all. So, we are both in the same boat because I thought of this right this second. It is the middle of April. We are going to do almost two podcasts a week for the rest of the year. So, let’s say that is roughly 65 podcasts. That’s probably the ballpark. What’s your over/under? Think of a number. I’ll think of a number and we’ll both say on how many podcasts we will mention 90-day certs between now and the end of the year.
TCTim CallanOk. I got a number
TCTim CallanAt least three podcasts? Between now and the end of the year?
TCTim CallanMy number is 45. I think we’re gonna mention it almost every episode. That’s my prediction. I think we are gonna mention it two-thirds of episodes for one reason or another. My number is 45. So, we’ll see. Now of course, this is bias now because I can just make sure I mention it 45 times. It’s unfair.
TCTim CallanNo. Not outages. 90-day certs.
TCTim CallanI predict we will mention it 45 times.
TCTim CallanSure. I would agree with that.
TCTim CallanAlright. So, we are predicting between 30 and 45. I think that’s a reasonable range because this is the story. And the point I’m going to with this – not to belabor it, this is the story. Like, this came out of nowhere. It is the story for the rest of this year. It is definitely the story in the next year for some period of time until it becomes reality, at which point it continues to be the story. So, this is gonna be the story for years to come.
TCTim CallanAnd we are seeing it just showing up over and over again.
TCTim CallanAnd I think there’s that point about educating. I think that’s a very valid point, Jason. The other thing about you and me, we kind of like to analyze and explore. I do not believe that the industry or the IT community actually has wrapped its head around the full set of implications of this change.
TCTim CallanAnd I think that one of the reasons you and I are gonna discuss it a lot this year is we are gonna take different angles on it. Think of all the different angles we’ve had on Post-Quantum Cryptography. Right? Say, well, what about the Z-date and what does harvest and decrypt really mean and is China gonna outspend the U.S. and all these different kind of sub-angles that we took. I think we are gonna see the same thing with 90 days. We are gonna keep thinking of new implications and new reasons why it’s important and as we do, we are gonna want to explore them and that means we are gonna be returning this topic over and over again.
TCTim CallanYep. So, this was a certificate outage reported by Elon, impacted Jason who is among the most technically astute people I know, so if you can be taken down by a cert expiration, Jay, I think anybody can.