Podcast
Root Causes 331: Microsoft Restores Trust to VeriSign Code Signing Root
Hosted by
Tim Callan
Chief Compliance Officer
Jason Soroko
Fellow
Original broadcast date
September 13, 2023
Recent erroneous behavior for certain applications on Windows has drawn attention to the Microsoft trusted root store. It turns out that Microsoft removed - and then re-added - a legacy VeriSign root in its trusted roots list. We give you the details of what went on and why.
Podcast Transcript
Lightly edited for flow and brevity.
Tim CallanSo we have a crazy one today. I'm looking at an article from Ars Technica, written by Dan Goodin, 8/25/2023. So August 25 of this year, and the headline reads, Renegades Certificate Removed From Windows. Then it Returns. Microsoft Stays Silent. So I normally say what went on here, Jay. Should I go over the basics of what went on here? What went on here?
Jason SorokoWhat went on here?Tim CallanSo, for several days starting maybe around August 20 or so - I don't have that exact date - but that's the ballpark, people started to see unexpected and strange behaviors in a variety of Windows applications that they were using. It included versions of QuickBooks, among other things, and, um, it was all very kind of enigmatic and cryptic. And what it turned out is at the end of the day, there was a bad certificate. Like a certificate error. Certificates were showing up as looking like they were revoked on certain aspects of Windows, which was weird, and it didn't make sense and there was just this certificate that came out of the clear blue. And then in several days they started working again. This left a lot of people confused for a little while and it turns out the gist of it is that an old VeriSign root certificate that was still sitting in the Windows root store had been distrusted and these problems occurred and then Microsoft readded it to the root store.Jason SorokoReadded it to the root store, Tim?Tim CallanReadded it to the root store. So I've seen a lot of confusion online, where people are saying, I don't understand. You can't revoke a certificate. How did they revoke the certificates and then they're freaking out and going, it was not a matter of revoke certificates being unrevoked. What happened was every browser has their operating system - we call them browsers in shorthand - has control over what's in the root store and anytime they are capable of updating their root store, which is a lot, they are capable of adding and removing roots. They could put in new roots tomorrow, they could take out roots that they already have, and they can change the status of roots. So, in this case, the root was always there, but it was about the status. A change in the status. And they can do that as often as they can push an auto update. You might see things on Twitter or around or people say, Microsoft unrevoked certs. They did not. You can't unrevoke a cert. But what they did was they moved a root out of trusted status, stuff broke, presumably that they didn't expect to break, and they moved it back.
Jason SorokoThat makes sense. So in other words, we talked about root store programs, the typical one for most people is, of course, you go to a website, that website happens to have an SSL certificate that was signed by a CA. If that certificate from that web server chains up to the root store public key that's within the root store of the browser, all of a sudden, hey, you're off to the races and you have yourself an encrypted session and that's the way most of your web browsing works. You can absolutely remove a certificate public key from a root store and not revoke it. That word is distrust in a way. That's typically the way that you define that and that seems to be what's happened here, Tim.Tim CallanRight. And they have control over their own trust settings that they're going to have for each individual root. So if we go into a little more detail on that, so this was one of the old VeriSign roots. So if we will recall - a little bit of history lesson - there were a series of VeriSign roots. Those were taken over by Symantec when Symantec took over that business from VeriSign. VeriSign sold that business, got out of the certificate business and gave those roots to Symantec as part of that deal. Symantec operated those roots and was operating those roots in 2017 when Google executed their distrust of the Symantec roots, which was something that had been going on for like a year. And when that occurred, then once the Google distrust had occurred, basically, these certificates were useless and people couldn't use them practically in the real world, and they weren't. And then other browsers and operating systems kind of followed suit because these certs were no good, nobody was issuing on them, and they went ahead and in their own timelines, and however they were, they went through a process of deprecating these things. And all that makes perfect sense because I couldn't get an SSL certificate with one of these roots today. If I had an SSL certificate that had one of these roots back in the day, it would be long expired. You would think there'd be no problem, right? You’d say eh, no problem. It's all solved. But there's code signing. And the thing about code signing is, it's a different paradigm. While an SSL certificate is supposed to be working right now, in real time, affecting things right now only because it's transactional, right? It's a communications tunnel. Code signing is about signing a piece of code and then at a date in the future, having confidence that who created it, who signed it and that code hasn't changed. And as such, code signing is fundamentally a future looking thing, which means that there is still code out there that was signed on those VeriSign roots that is still being used on systems.Jason SorokoRight. I remember back in the day, Tim, where there were other use cases outside of SSL, such as, hey, what about IoT devices that might have received some of these certs, right?And here's a good example, code signing being such a long term practice, software lasts a long time, and if there's a check on root certificates, then this would be effective. And so, I can definitely see how this can happen. I can also see how it would have been very easy to take it away thinking that was the right thing to do and then having to bring it back.
It’s amazing that this has happened, but, not shocking. But here it is. I guess it’s happening.
Tim CallanA little more nuance on this on how Microsoft does it because I think it's an interesting peek into how these things work - is Microsoft actually had it, basically with a not before flag. I might be getting some of the nomenclature here a little bit wrong, but this is the right idea. There's a concept of a not before, right, it's basically the date the certificate is issued, and that's attached to the certificate. It means that you can't use it before that date. That prevents people from doing weird things like changing system clocks and getting credit for a certificate before it was issued. So they're using the not before, basically they had a not before flag so that if the code signing cert was at or earlier than a certain date, then it would be allowed and otherwise it would not. Right? So they wanted to prevent these roots from being used for signing after a certain date but stuff that had been signed in good faith prior to that date was still allowed, right? And you see why that makes sense. Right? What it does is it prevents you from using these certs now today in the future, but you can still use the code that you signed with those certs back when those certs were considered to be trusted.And what happened was they changed the status of that. This is part of how people manage their certs over time. And I'm looking for the specifics here, but they basically changed it from a this not before status to what is essentially a distrusted status, where they just said, ok, we're not going to trust this anymore. And you can imagine this too - you can imagine somebody who is running a root store, they want to clean things up, they want to get rid of the old stuff and so that's part of what they do. And they do this all the time. Old roots that are in the newest anymore with no active leaf certificates, they will go ahead and distrust them and knock them out. And there may have been an opinion that someone came to that it was the right time in the lifecycle to do this with that root. And so they went ahead and executed that. They may even have talked to the owner. I guess that would have been DigiCert. I don’t think it would be DigiCert. Well, that's kind of the problem. There isn't an owner there. But normally they would talk to the owner, and they would say is it okay for us to deprecate this root? Right? They might have contacted VeriSign for all I know, and VeriSign might have said, hey, no skin off my nose.
So then they move ahead, and all of a sudden, their tech support lines are ringing, and somebody realizes, gosh, this broke stuff and I want to unbreak it so they go in and they change the status back to what it was. And that's fine. Like on the one hand, it is okay to clean up your root store. Good idea. On the other hand, in this particular one, it doesn't seem to me like there's any particular deadline or any particular hurry. There's no reason why it has to happen now, as opposed to in a year or two or whatever. And so I have to suspect - I don't have any inside information here - that they all thought it was fine. They thought it wasn't in use. This was a really old thing. And it’s like, look, it's really old and it’s unsupported software. It's unsupported operating systems. It's time we can shut that off. And then they did and then people got upset, and they decided, oops, okay, maybe not. Let's let it keep running. And it seems to my eyes, that's what went on here.
Jason SorokoSeems to be Tim. This brings up a larger problem of usage of certificates for long term periods, use cases that last a long time. There are some natural things, in IoT, etc. very, very long spanned trust, but it really goes to show you the difficulty in running a root store program. The difficulty in the fact that a lot of these things are how are these certificates used? It's just not well documented. Even I'm sure a company as well managed as Microsoft, right? They obviously have a lot of people working there keeping track of things and even that, right, just didn't - -Tim CallanIt's an interesting point you bring up Jay, which is that, and we see this taking a lot of forms in a lot of ways, but in a lot of ways, crypto agility and backward compatibility are natural enemies in the wild, right? Like crypto, you fundamentally want a crypto agile system, where you can update your cryptography really, really, really fast and we've done whole episodes on the concept of crypto agility in the past. So if you want to dive deep on that, go look one of those up and read them but at the same time, you have this backward compatibility problem, where you want to keep the old stuff around and one of the challenges that you often run into is how do you do those two things together?The other thing that happens is that, in a lot of ways, security and backward compatibility are natural enemies in the wild, because you frequently see the argument where someone says, well, we want to be able to support things going back to the following systems and you're talking about versions of Android from 2004. And then somebody says, well, wait a minute, those versions of Android from 2004 are chocked full of known vulnerabilities. And are you doing somebody a favor by allowing them to use that old operating system? Or are they better off if you say, no, I'm not going to work with that operating system, get on something that isn't complete and utter Swiss cheese? And so that's a problem too. And where do you draw those lines? And, it's easy to have conversations that are all theoretical and on paper, and everybody is in agreement and Kumbaya and then you get down to brass tacks and it's messy. And I think this is an example of seeing that kind of messiness and that push/pull in action in the real world.
Jason SorokoYou got it. And because of the fact that these things live for so long, this may not be the last time we see this but it’s unusual. That’s an unusual story.Tim CallanIt's unusual. And these are old. I mean these are old roots. And maybe last point there – not to say something that’s not too painfully obvious, is stuff lasts a long time. And stuff lasts longer than you and I figure. You and I aren't - I'm not. I'm sure you're not. - running any hardware from 2015. But clearly people are. And that's another thing that you see going on here. And so all of those stories that we see played out over and over and over again in our industry - Here's another chapter in that book.Stay informed with expert insights
Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!
