Podcast
Root Causes 287: GoDaddy Private Key Breach
Hosted by
Jason Soroko
Fellow
Original broadcast date
March 21, 2023
A GoDaddy breach exposed customer private keys. We explain expectations surrounding private key exposure and get into the interesting question of when an incident is or is not part of a large company's CA business.
Podcast transcript
Lightly edited for flow and brevity.
TCTim CallanAt the top of the show, I always say that we talk about all matters digital certificates and PKI and in this case, I think we are talking about a matter of digital certificates. So, I’m looking at a headline. This is from February 20, 2023 from InfoSecurity. Author is Alessandro Mascellino and here’s the headline:
“GoDaddy Announces Source Code Stolen and Malware Installed in Breach.”
So what happened here and what does this have to do with certificates?
“GoDaddy Announces Source Code Stolen and Malware Installed in Breach.”
So what happened here and what does this have to do with certificates?
And we want to bring out the root cause of the issue. Basically – and I’ll read right out of the text. It says, “however, in some cases, customer’s SSL private keys were exposed and if abused, this key could allow an attacker to impersonate a customer’s website or other services. While GoDaddy has reset customer WordPress passwords and private keys, it is currently in the process of issuing them new SSL certificates,” Tim.
TCTim CallanOk. So, if there is a private key exposure for a public SSL cert, which is what we are talking about here in this scenario, right, because they are for hosted websites, then there is a 24-hour revocation required on any exposed private key. That means that in the event that GoDaddy has become aware of the private key, that they actually have 24 hours to revoke that cert. So, you know, this might be just a matter of how things got reported to the journalist but the process of is a little surprising to me unless this is being reported in real time. Because that’s the kind of thing that needs to be dealt with very, very quickly or you are out of compliance with the CA/Browser Forum rules and the Root Program rules.
TCTim CallanOk.
TCTim CallanYeah. That’s interesting. And it does need to be done in 24 hours. So, I guess the other thing that’s not clear to me, and I don’t know if you have a viewpoint on this or not, is if this was a November incident, why are we learning about it now? Did GoDaddy just reveal that this occurred or is there new information or what happened with that?
TCTim CallanAh! They filed their 10-K. That’s what happened. And I guess this was covered in their 10-K. So, presumably all of those certificates were dealt with long ago and there’s probably a tense problem in the way we are discussing it because you can’t let exposed private keys sit around for months. That’s just utterly unacceptable.
TCTim CallanYeah. It was in November 2022. There is a quote here, “In early December 2022 we started receiving a small number of customer complaints.” So, when this incident occurred and this actually was monitored and covered in the public community and the CA/Browser Forum and that occurred at that time. I remember that. There was no real conversation at that time about them not dealing with any exposed private keys that they had. I recall that there wasn’t a lot of clarity in the beginning about whether or not private keys were affected because they had this major breach and people started asking questions to say, look, were private keys stolen, and there was a period of time where I don’t know if people at GoDaddy knew the answer, but the community at large did not know the answer. It was definitely a big question mark that we had. So, at this point, we can say that according to this journalism at least that private keys definitely were and let’s hope that all the certs were already replaced
TCTim CallanWhat’s interesting in this case was there’s different rules—and this is probably an interesting thing to highlight. They are very clear rules about transparency when it comes to public certificates. We’ve talked about this ad nauseum in the past. The enforcement mechanism is that there is a small number of browsers that really control how public certs are treated in the world. We say browsers, but it's operating systems and software. We call them browsers but they are mostly not browsers anymore. And as a consequence of that, there’s a great deal of strength of enforcement for public CAs to be very, very transparent, and public CAs have to go through a great deal of exposure of any incident that occurs by the rules, and if they don’t do it, that can be a very serious problem. In some of the distrust events that have occurred in past, at least one of the contributing factors was lack of transparency. Like in every case I can think of at least one of the contributing factors was lack of transparency. There were always others as well, but that was one of them. And so, the CAs have this very high bar that they are held to get over. Now that doesn’t mean every CA always does but, in general, you see a large quantity of compliance. Now if you turn around and you look at another industry space, there is seldom that level of forced transparency on just about any other industry space. Certainly in the hosting space. Like there’s no equivalent of the CA/Browser Forum and there’s no equivalent of a browser root store program for web hosting services. So hosting services are policed by what are vaguer, less attentive and laxer requirements like, you know, laws around things like fraud that are just much more general and less focused. As a result, you see the difference, which is there is a breach in GoDaddy and they’ve got some breach notification laws they have to follow and I’m sure that a bunch of people got a letter and maybe some people got some kind of make-good service that they had to offer but it was nowhere near the level of codified prescripted reporting and scrutiny and obligation to answer questions from any member of the public who knows how to use a web browser. Like, all of that that exists for public CAs just isn’t there in the hosting space. So, you know, it’s interesting to see, to observe the difference between a breach that occurs in a hosting platform that affects a very large number of people and once that same breach may have taken their private keys. Right? And all of the sudden, a whole different set of rules kicks in. And, you know, this is, I think, illustrative of the very high degree of transparency that a public CA is held to as opposed to almost any other technology space in the industry. I won’t say any other, but almost.
TCTim CallanI’m glad that you brought that up because the other thing that came up in the public discussion around this is, well, what’s the definition of a CA? It is the same company and if it’s different people at different desks, can they argue that they are not a CA? Because, well, these people at these desks that are dealing with it don’t work for those people at those desks who are the people who are the CA. And how do you draw those lines and where do you draw those lines? You know, a lot of companies are in a lot of spaces, right? GoDaddy is a CA. They are also a large hosting provider. Microsoft is a CA. They are also a huge operating system provider. Google is a CA. They are also an operating system and web services provider, etc. Apple is a CA. And so you get into these weird, nebulous questions about well, is this, now is this particular thing, is this particular issue or vulnerability or failure part of the CA’s obligation to report and remediate or not? Because if you are one of these providers and you can find a way to cleverly and creatively construe that this is not occurring in your CA operation, then all of the sudden the requirements in what you do drop dramatically. So you get into that whole interesting conversation that does not have any kind of clear answer and doesn’t even have a clear referee.
TCTim CallanThey have to.
Your existence depends on it and for everybody else who is acting as a CA because they’ve got root certificates in the Trust store of your browser, ok, to me that’s a CA. It’s a short conversation. You got that, you are a CA. And therefore, GoDaddy is a CA and so, therefore, if you are procuring SSL certificates from that vendor and they are not going to be super transparent about these things, well, you gotta think twice about where you are getting your SSL certificates from.
TCTim CallanI think that’s a valid point, Jay, which is to some degree the market does have an opportunity to have an enforcing function here which is to say, look, you might come back and you might argue this is here or this there but at the end of the day, if I can’t trust your overall company to, you know, maintain extremely high standards then how can I trust you as a CA? And that absolutely is a valid approach and I think there is some strength to that. I think it’s less strength than if Mozilla says you need to straighten up or we are gonna cut you out. So, that’s kind of the challenge. But, you know, to your point, maybe to some degree that is what we have to work with and so maybe we just need to use that as our lever, as our tool.
TCTim CallanOk. So there you have it. A simple story but it turns out there is a decent amount there when we unpack it.