Podcast
Root Causes 429: ServiceNow Outage Due to Expired Root Certificate
Hosted by
Jason Soroko
Fellow
Original broadcast date
October 8, 2024
A ServiceNow private CA root expired, creating outages across hundreds of enterprises. We explain what appears to have gone on.
Podcast transcript
TCTim CallanSo we just a couple episodes ago, talked about a major outage due to an unrenewed certificate expiration. That was Bank of England. And we're going to talk about another one today.
TCTim CallanService Someone. Someone Now. Who is that? ServiceNow. I mean, what a giant, ubiquitous IT platforms that’s all over this globe. They had an outage. I read an article that said that something like 600 enterprise customers were affected. And how did this happen, Jay? What was the problem?
TCTim CallanSo it was their root that they had set up in their PKI sometime in the past, and they were using this PKI for putting certs on internal apps and roots are certificates. They're root certificates, and every certificate expires, as we've discussed. So the expiration date for this root rolled around, and I guess nobody knew?
TCTim CallanSo this is a point you make a lot, Jason, which is that PKI is hard, and PKI experts are experts, and everybody who is a competent IT professional is not a PKI expert. And if you get it wrong, things like this happen. The stakes are very high.
TCTim CallanIt really is. And, it takes a long time and a lot of work to become a PKI expert. It's not a thing you just wake up tomorrow and decide you're going to be.
TCTim CallanAn organization that organizationally definitely has a whole lot of demonstrated technical competence. It’s ServiceNow. And yet this happens to them. And, one of the things we've mentioned in the past, part of the reason that we point out when these high profile certificate error based outages occur to these companies, is that they are the best of the best. And if this can happen to ServiceNow, or Bank of England, or, going back and some of the other people we've talked about recently, Microsoft. If this can happen to them, then surely it can happen to any of us.
TCTim CallanIt's an interesting angle here. Because you and I talk about this, and when we talk about this, I think there is kind of an assumption that we're talking about a leaf cert that's sitting on a server somewhere or a device somewhere, and that leaf CERT is supposed to get renewed, whether it's public root or private root and that doesn't occur, and stuff starts to crash. That's the scenario we all kind of have in our mind’s eye. But this is interesting in that it's a little bit of an object lesson in the perils of your own internal CA and not operating the internal CA correctly. And of course, the problem there is you don't lose one leaf certificate in one system. You lose all the certificates that are on this root and all the systems associated therewith. And that could be a massive blackout very easily.
TCTim CallanAnd CLM, obviously is the obvious one for your leaf certificate problem. For your PKI, for your internal CA problem, like we see here, getting an expert partner in to help you with that and deal with those details for you is also a best practice. And you really want to look at both of those technology categories moving forward if you want to protect yourself from problems like this.
TCTim CallanAll right. Thank you, Jay.
TCTim CallanThis has been Root Causes.