Root Causes 631: Did the Bugzilla Bloodbath Change Anything?

Hosted by

Tim Callan

Chief Compliance Officer

Jason Soroko

Fellow

Original broadcast date

June 19, 2026

2024 saw a flurry of high profile incidents for public CA, which we named the Bugzilla Bloodbath. We look back to see how the WebPKI has changed as a consequence.

Podcast Transcript

Tim CallanSo, Jason, in the summer to autumn of 2024, we coined a term on this podcast - which became a word that other people used out in the world - which is the Bugzilla bloodbath.

Jason SorokoTim, to me it was some of the most thoughtful reporting on the most exciting period - and by exciting I mean "whoa," period - in WebPKI.

Tim CallanWith the possible exception of now. But yes, yes, agreed. We talked about it a lot at the time when it was happening. Now that original kind of flurry of incidents and problems has run its course - though the amount of incidents and bugs that's being reported continues to be very high compared to historical numbers. I thought now, a year to a year and a half later, depending on when you declare it to have stopped, would be a good time to say: did this matter? Is anything different now?

Jason SorokoAnd I have a viewpoint, but go ahead.

Tim CallanI'm going to give this a real simple answer. Yes, I agree. I'd say there's some cosmetic ways that it's different, that it mattered, and then there's some deeper ways that it mattered. What I'll call the cosmetic ways that it mattered - although it certainly isn't cosmetic to those who are directly affected - is distrusts. We went from a pace of maybe having a distrust every other year to we've had four in the last two years. You and I predicted in our 2026 predictions episode - and I feel good about this prediction - that we won't make the year without more. So let's say there's two more, hypothetically. If you got six in three years, that's just a very different pace than what we were on before. So that's an obvious consequence. And connected to that, the consequence that we're now all - any thoughtful person who's following this space - is aware of the fact that root store distrust is a legitimate tool in the root store's toolkit that can, and when necessary will, be used. So that's a consequence. Now, with something in the ballpark of 80 roots in the popular root stores, you might argue that taking four out isn't that material. That's a legitimate way of looking at it, even though one of them was pretty popular. But I think the deeper impact is coming out of the Bugzilla bloodbath, and as a direct consequence, it tuned up the community of people who demand quality and precision and transparency from public CAs. This is more than browser root stores, though it certainly includes browser root stores. It is academics, it is concerned citizens, it is white hats, it is auditors, it is certain members of the CA community. It got that group activated in a way they hadn't been previously. It got that group understanding, with clarity, a more effective way to demand quality and accuracy from the CAs. The Bugzilla bloodbath directly led to changes in language and the form of root store policies - to clarify expectations, to close loopholes, to demand more, again, more precision and more consistency and accuracy.

Tim CallanThe Bugzilla bloodbath ultimately led to some of the initiatives and some of the work that we're talking about now. So you and I have talked lately about the increased scrutiny on CPSs and what they say - and whether CAs are behaving accurately to their CPS. You and I have talked about how, for a lot of years, that was all real sludgy. It's still sludgier than many people would prefer, but it's crystallizing at a rapid rate. And ultimately, I think there has been a philosophy shift that should not be underrepresented, in how we demand consistent compliance, and correct compliance, and transparency, and action about noncompliance from our public CAs. I think that's top of the list.

Tim CallanAnd the action about noncompliance - this is another big one. According to policy, you must have a clear action item, one or more clear action items, for every root cause of every incident. You have to be able to map them back. When CAs don't do that, they get called out, and there is a demand that they do. If I'm going to say that one of the root causes was that people make mistakes, then I need an action plan for that. So that's a new way of thinking. And that level of rigor that has been added to the process, and that is demanded, is definitely new. It's post-Bugzilla bloodbath, and it's a direct consequence.

Jason SorokoI think that's a great articulation, and it's a great reflection on that time and where we are now.

Tim CallanThen, of course, obviously, there's a list of very specific loopholes and dodges and bad things. Some of this is bad reporting, bad transparency, bad revocation, where the level of tolerance for this has just - where you used to be able to get away with it, getting away with it's gotten really hard. That's part and parcel of the same thing as well. So I think that's the big outcome: this sort of fundamental attitudinal shift about what's expected of a CA. In my opinion, I think this is very healthy.

Jason SorokoI'm quoted as saying on this podcast before, as we were going through this, that a healthy self-policing system is one in which it actually has teeth, actually acts, actually makes real improvements through time, does the hard things. A self-filtering system is going to be a better one. I think now even what constitutes the filtering rules are a lot more clear. I think that those of us who are left standing through that bloodbath are -

Tim Callan- there might even be more comfort right now. I think if you're buttoned up, if you're a CA that feels like you have a handle on your rules and your processes, I think there may be more comfort. I think there are a lot of CAs that either are deeply uncomfortable, or should be and aren't - which is absolutely worse for their prospects - who don't understand, at some level, to the point where it translates to consistent action, that expectations are not what they were two years ago. They're still operating like it was two years ago. Or there are CAs where maybe the people who operate the CA on the day-to-day get it, but the people who are ultimately responsible for certain decisions - who have to approve or greenlight certain choices, or have the ability to block certain actions - don't get it. That's part of the reason that I forecast additional distrust, because I think, unfortunately, there appear to be some slow learners in the group.

Tim CallanWell, then the filter will work. You hope, you imagine, that what'll happen is somewhere along the line, everybody's going to get the memo. We're not wishing for any more distrusts, but if those distrusts do occur, you hope the consequence is that that next slowest antelope realizes it needs to run faster -

Jason Soroko- before it gets eaten too. Sometimes stress can be a motivator. I think that's what you're saying. And some of these CAs might improve to be above the threshold.

Tim CallanExactly. All of this doesn't come from a perspective of cruelty or sadism - we don't want to make the CAs unhappy. It comes from a perspective of enforcing a certain level of quality that's deemed to be necessary for the system that fundamentally serves the public trust of nearly 6 billion humans. And when you look at it that way, then a distrust is viewed as an unfortunate but necessary consequence of inability to meet expectations - or unwillingness, perhaps

Jason SorokoExactly. Eventually you get to the point of equilibrium.

Tim CallanAnd maybe the more of these things happen, maybe the message does sink in in places where it hasn't sunk in yet. So I think ultimately - and the reason I wanted to say something about this, and we can leave this here, is we've stopped talking about it, and that actual kind of period is gone. But I think we're in a post-Bugzilla bloodbath era that, in important ways, is different from what the WebPKI looked like two years ago.

Jason SorokoYes. And to your opening question - there absolutely was an impact. Are we in a better place?

Tim CallanI think so.

Jason SorokoThanks, Tim. All right.

Stay informed with expert insights

Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!