Root Causes 388: What Is the WebPKI?

Tim Callan

So, Jay, we've been talking a lot about a lot of the stuff that's been going on in Bugzilla, and public CAs and, you know, there was an episode called the Bugzilla bloodbath, and a number of them after that, and one before And in that timeframe, there's been a word that we don't necessarily use a lot that we have been using a lot, which is the WebPKI. And your suggestion, which I think is an excellent one is, do the listeners know what we mean when we say the WebPKI and maybe we should tell them? So I think that's what we're gonna do today.

Jason Soroko

That sounds great. Yeah, yeah. What kind of tweaked me was, you and I say the word casually and that's because we are like, so deep in the bench of inside baseball it's - -

Tim Callan

This is our industry, man. This is what we do.

Jason Soroko

What we do. And you know, we have too many listeners now, to assume that everybody can just snap their fingers and know exactly what that is and what the implications of some of them are subtle points of WebPKI. What are its components? What's the framework of it? How does it work? How has it been working for the past couple decades, Tim? I don't think they could get anybody better to listen to than you to help to talk about what it really is.

Tim Callan

So shall we start with a high level definition?

Jason Soroko

Let’s do it.

Tim Callan

Okay, now, I'm going to say, the high level definition today, my high level definition today, is not what it was ten years ago. And this is an important point.

So what I mean today, and in a way, the WebPKI is a bit of a misnomer now. And we'll get to that. What I mean today, when I say or - - let’s do it the other way. What would I have meant ten years ago? Ten years ago, if you said to me, what is the WebPKI, I would have said, that's real simple. That is the full body and corpus of the public root SSL certificates that are issued in the world, and the surrounding infrastructure, procedures, rules and ecosystems that make them possible and reliable. That would have been my definition a decade ago, I just made that up. But I think it's pretty good. I can stick with that. And that's still true today. But the WebPKI, in my definition today has expanded and it includes, first of all, a broader set of supporting structures, for instance, certificate transparency, and certificate transparency tools, and things that just didn't exist then. But in addition to that, it really has expanded its footprint to many more types of certificates, really to damn near any certificate that sits on a public root. So nowadays, we would consider an S/MIME cert or a client certificate or a code signing certificate or a document signing certificate. As long as it was sitting on a public root that was used in the general public root stores, I would lump that into the WebPKI.

Jason Soroko

Yes, yes. It all belongs. It's all part of that ecosystem that exists. And I think that is, Tim, what really might confuse some people in that how did all this come together?

Tim Callan

Yeah. Cause that name, right? Right? That name. That's where the name is a misnomer. How did it all start? How it all started was we had a thing called the World Wide Web and we needed to do certain things on the World Wide Web. We needed to authenticate identity and we needed to encrypt data across the DMZ and the best solution for that was PKI. Right? Very well established network, point to point communication and it worked, and it was reliable and robust, mathematically proven and all that wonderful stuff. And so as a consequence, it's kind of obvious to say this is PKI for the web, because at the time, first of all, the web in most people's mind and in most of real human experience was human being sitting in a browser goes to a website and gets HTML back.

Which is a small minority of how SSL certs are used today. And the second component of it or element of it was that, you know, it was only SSL, right? Like there was no sense of saying, well, we need to consider code signing, or S/MIME as part of this single system with similar expectations and similar rules and similar processes. None of that was there at the time. The maturation just wasn't there yet.

Jason Soroko

So Tim, what you're saying is, and you know, I lived through this. I'm old as well.

Tim Callan

You and me both brother.

Jason Soroko

What came first, the chicken or the egg? You know, what came first? It was the browser, right? So in other words, what came first was the internet infrastructure. That was even before the browser. I was on the internet, doing internety things before a browser. But then the browser showed up and it was - - Tim, you remember the earliest days. It does not resemble at all what is happening today through a browser. But soon after that is exactly the ecosystem you're talking about, that early ecosystem, which is and what it belonged to, was this. There were people who were registering domains. So interestingly enough, right, the domain registrar system is, you know, a very close cousin to a lot of what's going on here.

Because you can't get a cert really without that domain. Now there's, many examples now where you can but back in the day, that's how it started. A cert and a domain were kind of like peanut butter and jelly. They went together. But what you needed as part of that infrastructure was you needed the browser, and you needed a browser that trusted a Certificate Authority and then you needed subscribers. Right? Who were actually going off and getting those certificates, Tim, and then things started to get more complicated from there.

And then, of course, we had the CA/Browser Forum, right?

Tim Callan

Right.

Jason Soroko

2012. And BR Rules, which is a big part of WebPKI. Right? All the different rules surrounding it.

Tim Callan

And remember that it had its roots. So CA/Browser Forum was founded in 2006.

Jason Soroko

Right.

Tim Callan

And EVGs were released in 2007. And that's the real - - like, this is one of the things that a lot of people forget or don't realize is EV whether or not you're a fan or not - EV is a very divisive subject. Some people love it. Some people hate it. Whether a fan or not, EV was the kernel of the whole thing. EV was that original acorn that eventually grew into the oak tree that we think of as CA/Browser Forum, and the Baseline Requirements, and this whole cloud of surrounding systems. And all of that started with EV.

Jason Soroko

Yes. Isn't that interesting for everybody? Right? And isn't interesting how late in the game. Like if you were to draw a timeline, you would have expected that to come at the beginning and you and I had a podcast on this and the surprising note of, geez, this came much later in the game. But it came at a point of usage of the web where it absolutely had to happen.

Tim Callan

Yeah. I mean, if you think back in those days, so at that point, we were a little over ten years in and, you know, it was very wild and wooly in the beginning. Nobody knew. People trying to figure it out, and things moved very fast. It's very reminiscent of how AI is now, which is it's changing so quickly that nobody really knows what its gonna look like in a year. And that was true in 1995, 1996, 1997. Nobody really knew what it was gonna look like in a year or two or five. And, they were right, it didn't. And so, as a consequence, everybody at the time was just kind of keeping their feet moving as fast as they could just to keep up, right. Just to keep their feet under them. And, then as things got more solidified and established, there started to be a concern, which is look, we need some level of consistently high quality, trustworthy high quality here. We need codification. We need rules that are understood and universal, and that you can rely on the fact that somebody is following them. We need transparency. We need auditability. We need the public to be able to shine a light on a CA. And all of those things were absent, and they were vulnerabilities, and you know, honestly, like, a lot of stuff could have gone worse than it did, considering how un - - I don't even want to say unregulated but just inconsistent and not even entirely understood this whole system was.

Jason Soroko

Yeah. The whole internet was the Wild West, and this was not a lot different before the rules started coming in.

Tim Callan

Yeah. And, you know, so you know, that's a lot of what I think the desire there was. Right? And again, this goes all the way back to EV to create something that was universally understood and followed a set of rules that whether or not they were perfect, we could be confident that they were pretty good. And then all those things evolved and changed over time and that's fine. It's supposed to. But at least you were starting from a baseline that was known to be pretty good and you knew that with an EV cert and you didn't really know that with a non-EV cert. It wasn't until the BRs that we started to raise the level for everybody and that's where all certs started to suddenly have to live up to certain standards that were originally required by EV.

Jason Soroko

Tim, so let's talk to the two very important clusters of concepts within WebPKI. The CAs cluster of concepts because, you know, CAs also work sometimes with relying parties, which is something you've defined on previous podcasts.

And also around clustering around the CAs is also the concept of validation. Right? And also, revocation technologies, CRLs, OCSP, etc. So those are all things that are clustered around a CA. Some things that are around - -

Tim Callan

And, Jason, let me add one more. CT reporting.

Jason Soroko

CT reporting.

Tim Callan

Certificate Transparency reporting is a critical thing that CAs must do without which the whole ecosystem doesn't work correctly.

Jason Soroko

Validation, revocation, transparency.

Tim Callan

Yes. I think that’s a good way to do it. Yep.

Jason Soroko

I would say those are three big words in the cluster of the Certificate Authorities. I would say the browsers have their own cluster, Tim. The browsers have the certificate repositories, the trust stores.

Tim Callan

Trust stores. Absolutely.

Jason Soroko

So you know, the number one question I get, Tim, maybe tell me what you think. The number one question I get is, how do you get to become a CA?

Tim Callan

So there's a process. There's a process and what you do is, there are publicly available rules that anyone can read, that are about what CAs need to be able to follow. And they're all captured in the Baseline Requirements. So you basically go, and you look at the Baseline Requirements, and it's a lot of documentation and this is very specifically what you're expected to do. And what you have to do is you get your systems and everything in place to the point where you can credibly contend to a trained auditor that you're able to do this correctly. And then you get a pre-issuance audit. You get a readiness audit. And it’s basically where a certified web trust auditor comes and looks at your practices and your systems. And this is technology, and this is policies, and this is procedure, and this is a bunch of things. Because remember, you don't have any existing performance in the world at this time. So a backward looking audit just isn't possible. So instead you get a readiness audit, and they look at your systems and processes everything and say, okay, I accept that this organization can do this correctly. And then at that point, you apply. And you apply to the root stores. And every root store has its own process, but there's basically four root stores that if you don't make it into those root stores, you're not really going to be any good as a CA, right? So the four that you have to be included in, you have to pass, are Chromium, Apple, Mozilla, Microsoft. There's other root stores in the world. It's great to get them but get those four, you're able to really be a viable commercial public CA and without it, you're kind of not. And then they all have their own processes, and they want to get those audits and they want to look at things. They may very well follow up. There is an element called CCADB, and CCADB - Centralized CA Database is used by several but not all of the root stores and so for those guys, what you do is use CCADB. So, for instance, you're gonna get Apple that way right and you do your whole process through CCADB. And that has advantages. And we've discussed this in a previous episode, but that has advantages because it gives us sort of normalized, expected behavior with the CA, right. We're expecting things to be a certain way. And you go apply the CCADB, and then you go through an application process with these root stores. And then they normally have a cadence. They don't just stick in whenever they want. They normally have a cadence where they do this like a few times a year. It's pretty typical. Three times a year, maybe, they'll allow new roots into their stores, and you get in in the next window, or you don't.

There also have been people who've been rejected very, very publicly. And we discussed this on our podcast years ago, there was a Saudi company called Dark Matter that tried to become a public CA and basically, the public sentiment was that we do not feel from an integrity perspective, like this is a trustworthy organization. And at the end of the day, Dark Matter was not allowed to become a public CA. And so that's an element of this too, right. And that's part of the discussion as well. Especially in Mozilla, because Mozilla is so transparent and so public.

And then at the end, if you can get over the hoops for all four of these, these trust stores, then you'll be able to issue a certificate that will actually work inside of their devices, their browsers and operating systems. And then you go into the normal cadence of CA. You're expected to do the things CAs are expected to do. You get your annual audits, you report your bugs, you follow the rules of the root store programs. You do all of that stuff at that point.

Jason Soroko

Tim, thanks for that. Geez, it is not easy to become a CA. It’s been proven that you've got to work really hard to maintain being a CA, and I tell you, there's a long tail of CAs out there. Once you get past the first, you know, four or five on the list of, you know, rated by issuance, total issuance let's say, you start to get into a long tail of CAs that are very, very specialized and I can see how in the future that tail might shorten a little.

I think for people who are curious about WebPKI, I don't think that the list of CAs is going to grow a lot in the future. I think it might tighten up a little more than it more than it’s probable of growing.

Tim Callan

It's a little surprise - - , like you do kind of routinely see requests for new CAs to, for instance, join see CA/Browser Forum. And you know, presumably these people are out getting into root stores. So you know there are new CAs being established. At the same time, old CAs also sometimes go away either on their own. Right? They decide it's not that important to them. It's kind of a pain in the butt. They don't want to do it and they stop, and they phase that out. Or sometimes every once in a while there's a very visible distrust event, right? But there is an important point here to understand about the WebPKI and you talk about the longtail CAs and this is another thing that probably it's good to be clear on is I think sometimes there's an assumption that is a false assumption that a CA that is small or niche, or pretty much unknown, somehow represents little or at least less risk to the security of the overall WebPKI than a CA that is large and established. And that isn't the case. Like the way to think about it is every CA - - think of it as a security metaphor. Let's pretend it's a castle. And there's walls around the castle and we want to keep the bad guys out and the good guys in, right? Every CA is a door in that castle. And the fact that one CA is well-known and popular, and another CA is niche and specialized doesn’t matter. It's just another door. And if the bad guy wants to spoof a site, or a brand, an enterprise that is using the most popular CA in the world, right? They're using Let's Encrypt, let's say or let's say they're using Sectigo or DigiCert or one of the big ones. They don't have to defeat Let's Encrypt or Sectigo or DigiCert. They can defeat Ma & Pa’s Corner CA and still get certificates against that big brand.

Jason Soroko

If it's in the root program.

Tim Callan

If it’s in the root program, it's in the root program. It's a door to the castle.

Jason Soroko

That’s right. And so Tim, all it takes is one misissued cert for the purposes of man in the middle attack and decrypting communication. That is all it takes.

Tim Callan

Right. And so that's a very important thing to understand and I see people sort of unthinkingly make this assumption. And I get why you would make that assumption if you didn't know the detail of why that's not an accurate assumption. But it isn't an accurate assumption because every one of them, every trusted root, at least in today's architecture - maybe this won't always be the case - but in today's architecture, every trusted root gives you the same privileges inside of the relying software.

Jason Soroko

Awesome, Tim. Hey, you know what, while we're talking now, I'm going to offer three more components in WebPKI because I know we're long in the tooth in this podcast, but it's a rich subject.

Tim Callan

It’s okay. It’s okay. Yeah.

Jason Soroko

So I'm going to offer up technologies surrounding key generation.

Tim Callan

Okay. Sure.

Jason Soroko

I'm gonna offer up provisioning technologies. ACME, right?

Tim Callan

Yep. Absolutely.

Jason Soroko

Being a big one.

Tim Callan

ACME being the most obvious one. Yep. But not the only one.

Jason Soroko

It's not the only one for sure. So I'm gonna then offer up one more, which is all right, key management technologies, we've got provisioning technologies. I'm going to call it Certificate Lifecycle Management Technology. CLM. I think, Tim, tell me what you think. I think it belongs in the category of a vital component of WebPKI.

Tim Callan

Yeah. And this is also an evolving area, where what's happening is a few things are changing. One is that just the number and variety of certs you have is going up. The duration of those certs is going down. The need for agility and management of the certs is going up and the stakes are going up. And so maybe we break all those down individually.

The first one makes perfect sense. There's just more and more of them. And there’s more than there’s ever been and there's going to be more tomorrow and more of the day after that. It's just a very clear trendline that's been true for decades and isn't going to change.

The variety of them is going up. There’s certs everywhere and like we talked about at the top of this podcast, you know, these sort of obscure certs that aren't just SSL are gaining prominence or that aren't sort of a traditional SSL. Like you think stick something on your server on the webpage, and it runs for years, right? Those have been gaining a lot of prominence as well.

The third component is that the complexity of the whole thing is getting harder, because you have all these different certs. They're serving different roles. They're sitting in different places. That makes the whole management task that much harder.

But then lastly, the stakes, right? Everything - not only is everything digital these days and if you go way, way, way back in time to our Tim's Digital Haircut episode, you'll get my thesis on this - but not only are the most offline things in the world, like hairdressers, and restaurants, 100% digitized, but in addition to that, it's all interconnected. So we are at the point now where if you pull down one tent pole, the whole tent falls, because everything is connected to anything else and one failure anywhere in the system brings down the whole system. And so there's this expectation that everything is digitized. There's this expectation of availability and always on. Like, can you remember, Jay, back in the 90s, where you'd have a favorite website, you'd go there to do whatever you're going to do and once in a while it wasn't running, and you go, eh, website is not running. Okay. And you go check back in an hour. And maybe it was running and if it wasn't, you’d go, okay, I guess I'll look tomorrow. And we thought that was normal. Right? Can you imagine what would happen if that were normal today?

Jason Soroko

Tim, there's businesses right now where they talk about millions of dollars per second. That’s just unbelievable.

Tim Callan

Right. And so the stakes are just insanely high. And so when you put all of this together, there is a need for a reliable, error proof, always on process. That just wasn't the case even a decade ago. Even five, seven years ago, not like it is now. And that is a real big part of the whole thing.

Jason Soroko

I would say Tim in, you know, the COVID era, my goodness, right? I think that was the absolute death nail for anybody thinking you could be offline for any period of time, because now it has to be there. It just has to be there.

Tim Callan

So, you know, I think those four things together really kind of make that just into a must do at this era in history.

Jason Soroko

That's it, Tim and WebPKI has so much to do with all of it, Tim. It used to be something that was a curiosity back in my, you know, earlier university days, and now it's, it's just everywhere. It's so ubiquitous, most people don't even think of it at all. And that may be one of its greatest achievements. But let's not make a mistake - There's so much going on underneath.

Tim Callan

Absolutely. And so yeah, that's good. It's good to have a survey of WebPKI. And we tend to dive deep on a lot of these different aspects. And we tend to dive in and just talk about that, which is great. I think we should. I think it's a good way for this podcast to serve our audience in general. But, just sort of walking everybody through what do we mean when we say these words? It's a valuable thing. I'm glad we did it.

Jason Soroko

Thanks a lot, Tim.

Tim Callan

All right. Thank you, Jason.

Jason Soroko

Take care.

Tim Callan

This has been Root Causes.