Root Causes 534: Signing the Machines That Think

Hosted by

Tim Callan

Chief Compliance Officer

Jason Soroko

Fellow

Original broadcast date

October 10, 2025

Imagine what happens if you use the wrong LLM, including a malicious model placed there to create mischief or crime. How do you know? Jason proposes that, the same way we sign our code, we should be signing our AI models as well.

Podcast Transcript

Tim CallanJason, should we be signing the machines that think?

Jason SorokoSo, what the heck does that mean, first?

Tim CallanWhat does that mean?

Jason SorokoTim! Code signing. Something we just do. Does code think? No. That’s not what we are talking about. We're talking about something different than that. Unfortunately, a lot of code is run without signing it. Within live systems. But bad practice. Bad practice. A lot of people really should be using code that is signed. So you know what is intention to run. And there's just so many reasons for code signing. Let's just leave that there. So Tim, we were talking before about - a previous podcast to this – AI. The iceberg of what's above the waterline and below the waterline? Well, one of the things that's above the waterline that's really, really becoming important right now. In fact, I believe it was - it might have been Nvidia who said it - but small language models as an example are really probably going to end up becoming the majority of what's below the waterline. Interestingly enough. And it’s because of the efficiency of small language models. Like the reason I call out small language models is because most people, the vast majority of people who have ever played with AI, a large language model, that's exactly what they know about. They're dealing with something that was trained on enormous amounts of data and has a very general purpose. But realize that there are such things as small language models that are very, very, very specific and extremely efficient and sometimes act even more powerfully because of the way they're trained and the way that they're used compared to large language models. Those things might be running at the edge, might be running inside your smartphone. In fact, there are small language models right now being put out by very large organizations meant to be running completely offline inside of mobile devices, and probably IoT devices coming up. Do you hear me say anything about those things being signed? Do you know that you're using the small language model that you intended? Sign the machines that think. It's no longer about code signing. It's about model signing.

Tim CallanModel signing. So, you imagine a scenario where the model is wrong.

Jason SorokoMaliciously changed.

Tim CallanCould be malicious. Could be accidental. And under those circumstances, you get results that are suboptimal, possibly even malicious.

Jason SorokoI guarantee a very large chunk of what's going to be below the water line, according to your definition, Tim, is going to be like IoT devices running small language models offline, and those small language models were embedded at the point of manufacture, just like code. I bet you that IoT device has a code signed firmware. Might not have a code signed small language model.

Tim CallanSo this is bad, because PKI has a very checkered track record in the IoT world.

Jason SorokoThat’s even being generous.

Tim CallanNo. It’s not checkered. It's just all blank. It's just a big field of white. I mean, that's bad.

Jason SorokoTherefore, it's bad enough when you have extremely procedural code that's not signed, and if you flip a few things in there maliciously, like introduce a bad firmware, you can cause really bad things. What happens when you have something that's only barely deterministic by definition? Tim, think about it.

Tim CallanFirst of all, how do you detect it?

Jason SorokoSo I hope anybody listening to this right now is going, holy crap, I can see you're doing the arithmetic right now in your head, and you're like, this is crazy. When you add up all the risk. I'm just saying I haven't heard anybody even talking about it yet. I wanted to be one of the first.

Tim CallanSo, like, signing a model, like, conceptually, I understand it. In terms of what that really looks like, that's a little bit of a brain burner.

Jason SorokoDon’t forget though, the model will contain weights that will be static. Therefore, there is a static model that will be embedded in the edge. Let's call it edge devices. Could be IoT, could be mobile devices could be whatever. And these are static. It's not code. It's a model. And it has to do with statistical weights that can be signed. So the signing of it is not really delivered in some kind of - -

Tim CallanThere'd have to be some kind of mechanism for delivering it and checking it and all that.

Jason SorokoAnd that's where I'm stopping and I'm saying, look, I'll let the really, really, really, really smart people who will analyze how to do it properly. This is not what this podcast is about. This podcast is backing up 10 steps and saying, you should be signing that small language model because shenanigans are gonna happen, folks.

Tim CallanShenanigans are gonna happen. I agree with that. And the more pervasive this becomes, the more we take it for granted, the more actuated it becomes, the greater the risk.

Jason SorokoBecause I think what a lot of people who are listening to this podcast might not realize is we have large language models, we have small long language models. A lot of you who might have downloaded Ollama or Misty when I was talking about it back in the day, you might have had a lot of fun with those small language models. But what about really small language models, or nano language models? I think, Tim, that's going to be the majority of what's under the water. And the shenanigans on that, because they can be manipulated, because they're not even signed, nobody's even talking about it yet.

Tim CallanI mean, there's no body, there's no consortium, there's no rules. There’s no infrastructure.

Jason SorokoIt is the wildest of Wild West.

Tim CallanIt is the wildest of Wild West. There's not even a vocabulary for it.

Jason SorokoAnd that's why, when we're doing this is the sixth series of the Toronto sessions podcast.

Tim CallanHoly Moses.

Jason SorokoThe sixth.

Tim CallanWow.

Jason SorokoThe six times we've been on these Earth chairs.

Tim CallanOr Earth chairs. Earth couches.

Jason SorokoI wanted to point that out, because every time, Tim and I, every time you and I come here and sit on these couches, we prepare a list of what we think we want to talk about. And today I was trying to really go off the wall. You saw by some of the terminology I was using in some of my titles, and it's like, wow, you gotta sign the machines that think.

Tim CallanGotta sign the machines that think. Wow. I agree with you. Now that you've explained it. In fact, as soon as you said the title, I thought I knew where you're going, and I agreed with you, but I agree with you 100%. I don't even know where we're going to begin.

Jason SorokoThis is not the last podcast in this topic.

Tim CallanThis is not the last podcast on this topic.

Jason SorokoBut we had to have the first one.

Tim CallanAnd the other thing I just want to say is way to bring together the AI and PKI topics.

Jason SorokoI knew it.

Tim CallanGood job. I knew it was coming. I knew it was coming. You brought them together. This is a to be continued, for sure.

Jason SorokoIt is. Thanks, Tim.

Tim CallanThanks, Jay.

Stay informed with expert insights

Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!