Podcast
Root Causes 325: Certificate Error Causes Sharepoint Outage
Hosted by
Jason Soroko
Fellow
Original broadcast date
August 11, 2023
A recent outage in Microsoft Sharepoint was caused by an error in certificate installation. We explain what happened and the lessons to be learned.
Podcast transcript
Lightly edited for flow and brevity.
TCTim CallanI am looking at an article. This was written by Lawrence Abrams. It's in Bleeping Computer. The date is July 24, 2023 and the headline says it all - Microsoft SharePoint Outage Caused by Use of Wrong TLS Certificate. So what happened here, Jay?
TCTim CallanMicrosoft claims 10 minutes. Other people think it was longer, but that might have had to do with systems catching up and propagating. So go ahead.
TCTim CallanYep. Just plain put the wrong certificate on the wrong machine.
TCTim CallanThis is TLS working correctly, right?
TCTim CallanThe certificate installed on the wrong machine is exactly the same from a TLS perspective as the person impersonating you using the wrong machine. It's the exact same thing and under those circumstances, this is exactly what TLS is supposed to do.
TCTim CallanHow many human beings are using SharePoint at any given time. Exactly.
And, we're talking about just human error here, meaning automation could have helped you, especially if that automation configuration went through some sort of double check procedure. But, certainly, this looks to be human error without automation.
TCTim CallanAnd we don't know for sure, and we'll never know, but why don't we make that the working assumption for the rest of this episode? Because that seems very likely, like, that's the explanation.
So then you kind of come to well, like, this is just a very tidy example of all the stuff we're always talking about. Like, if you want to talk about a company that knows how to get IT, you can't do much better than Microsoft. And yet, here we are.
So then you kind of come to well, like, this is just a very tidy example of all the stuff we're always talking about. Like, if you want to talk about a company that knows how to get IT, you can't do much better than Microsoft. And yet, here we are.
TCTim CallanRight and to pick up something you just threw down. Not only just that this is Microsoft, but this is Microsoft on an insanely important part of the business. If this was some little obscure thing off in a corner somewhere that nobody really looks at, then you might rationalize that. I mean it still is embarrassing, but you might say, well, the risk/reward math is different. But what's the consequence of a 10 minute outage on SharePoint and OneDrive? Oh my God. And so in that regard, the stakes are extremely high, and this was an extremely high stakes portion of the business where this occurred.
TCTim CallanI think this is precisely right. And the other thing, again, that you just said that I just want to make sure we put a little bit of spotlight on is, this is a mistake that anyone can make. So it's not even like yeah, somebody made an error, but we can't even turn around and attribute this to incompetence. What you attribute this to is, it's a task that is fundamentally error prone, that must be repeated a whole lot and where any error, even a tiny error, has vast consequences. That's basically the situation with certificates.
TCTim CallanAbsolutely. I'm pretty sure that at are 2022 lookback, you and I predicted that we're going to have more stories about high visibility outages due to certificate errors. I'm also pretty sure that when we first introduced the idea of Google 90 day certs, we also predicted that even prior to 90 day certs, we were going to be having more stories about errors due to certificate, or outages due the certificate errors and indeed, both of those predictions have proven true.
TCTim CallanWhat did they know about IT? Anyway, there we go. And I think we've promised the listeners that when these things happen in high visibility circumstances with companies that definitely have the technical chops to get this right, that we're going to bring it to your attention, not to shame those companies, but to point out how hard it is. It's hard. It's hard. And here we've done it again.