Root Causes 87: Zoom's (Not) End-to-End Encryption

Tim Callan

So, today we're going to talk about end-to-end encryption, and video conferencing, particularly Zoom. So, Zoom, obviously, has been huge in the headlines of late due to the COVID-19 pandemic, sheltering in place, working from home, massive adoption of video conferencing software, including Zoom. Zoom stock made a run. And one of the things that came out that made big headlines was that despite literature available on Zoom’s headline, and in Zoom’s collateral that states that your Zoom call is end-to-end encrypted, it turns out it most definitely is not. So, I thought, well, let's explore what that means, explain what's going on, why it makes a difference, and what the implications are. So, first of all, how about a one-two cents definition of end-to-end encryption?

Jason Soroko

Sure, Tim. Whenever I'm using something that claims to be end-to-end encrypted - - regardless of the technicals, let's get into what the outcome of end-to-end encryption is and the outcome of that should be that the service that I'm using should not be able to perhaps store a key and listen into part of my conversation, perhaps for quality control checks, you know, for the least bad, you know, bad example, malicious example, to the most malicious example, which is - -

Tim Callan

Spying.

Jason Soroko

- - they've been subpoenaed by a government in some part of the world that we don't like, and they want to listen in to your corporate secrets, etc.

Tim Callan

Right. In a real layman's definition, when I'm sitting on my computer, and I'm the originator of content, that content is encrypted and that content remains encrypted, using secure cryptographic protocols that aren't breakable using methods in computing that we're aware of today, until it gets to the intended recipient no matter where they are in the world and only then is it decrypted to the point where it's intelligible. Like that's what people mean when they say end-to-end encryption.

Jason Soroko

Yeah. And it's not easy.

Tim Callan

Sure.

Jason Soroko

It's not easy especially if you want to include all kinds of really great fancy, easy to use, functionality that most teleconferencing software has, and that includes Zoom, and that includes all of Zoom's competitors as well.

Tim Callan

So, like, yeah, oh, sure. I'm sure this is very common. I mean, maybe not every system in the world, but for everything to remain end-to-end encrypted, basically, they’d all have to be point to point communications, right? So, if I get on a Zoom call, and I don't know that much about Zoom’s architecture, but I got some guesses. If I get on a Zoom call, and I'm talking to five other people, I think what happens is I stream up to some servers that Zoom owns, and then Zoom streams me down to the people who are on the call with me and then Zoom takes the streams of the people who are on the calls with me, and it streams them down to me.

Jason Soroko

Yeah, Tim. So, let's, let's get right into that. That's a perfect, perfect segue to exactly what the problem was and still is. So, on April 3, it was research that was published from Citizen Lab, which is actually part of the University of Toronto. So, shout out to my fellow Canadians there for publishing that and they're the ones who actually blew the horn on the end-to-end encryption claim. So, exactly what you just said, Tim. You get an invite to a Zoom meeting, and then you click the button and your client opens up and you get onto the meeting. It's awesome. Right?

Tim Callan

Yep.

Jason Soroko

Such great functionality. Behind the scenes, there's actually a TLS certificate that's been issued and actually makes its way to you. So, in other words, the certificate itself originates somewhere and one of the things that was discovered by these researchers was that there were 73 potential points where that certificate could be could be issued from and five of them, five of them were out of China.

Tim Callan

Okay.

Jason Soroko

Which is very problematic, right?

Tim Callan

Sure.

Jason Soroko

And additionally, there's another problem with this, which is the certificates themselves. I mean, you and I have talked a lot about problems of certificate definition, certificate policies and cryptographic algorithms. Zoom had actually claimed also not only end-to-end encryption, but they claimed to have AES 256, when, in fact, it was only at AES 128-bit, but even worse than that was that the encryption method of AES 128-bit was ECB, which is electronic codebook. Very problematic because it does not fully obfuscate the underlying artifacts, and leaves a lot of patterns. So, if, you know, if you wanted to decrypt something, and you were recording that stream, you know, let's just say that there's a number of attacks against that particular, that particular methodology. It definitely was encrypted and it's, you know, was not easily gotten to, but much, much more easily decrypted, or at least artifacts to be found within it, than compared to other forms of AES, and we can get into what Zoom has actually announced more recently.

Tim Callan

Yeah. So, let's do that but before we get there, I think this is - - you hit a bunch of good things for us to unpack there. One of which is, this isn't just an academic conversation. So, you know, end-to-end encryption means something real specific. Zoom isn't doing it. If you know kind of some fundamentals of how video conferencing works, you can probably guess that it was highly unlikely that Zoom was doing it, but then you could go to the next question, which is the - so what? Is this fundamentally just a semantic academic argument and does it not really matter? But what you've said is, no, actually, it also looks like it matters?

Jason Soroko

Yeah. I would say the problem of end-to-end encryption is less problematic to me than the fact that they had - - I would say the number one problem they had was the fact that they had key generation in jurisdictions that might not be friendly, number one, and number two, the other problem, of course, is that they were using an encryption algorithm that wasn't particularly strong. I wouldn't say that that was the worst problem in the world, but it's something that they did, in fact, address recently. So, it was problematic enough where they even they felt they had to step up.

Tim Callan

Sure. And you can see where changing the encryption of - - going from a AES 128, AES 256, if that's what they did, you can see where that might be a hassle, but also how companies go and do this, right? Because that sort of thing is done all the time. But fundamentally changing their basic architecture at its core, from not being end-to-end encrypted to being end-to-end encrypted, that's an order that's tantamount to creating all of the software from the beginning from the ground up.

Jason Soroko

Yeah. And they're not going to do that.

Tim Callan

So, let's just let's presume that's not happening. Exactly. Um, so what was their response?

Jason Soroko

Yeah. The response was they’re going to give administrators, whoever it is that hold the administrative keys to Zoom and your organization will be able to select which regions your data is actually going through, i.e., you can actually avoid China. That's the presumption. So, that's a good thing.

Number two is that they're actually changing their cryptographic algorithm from AES 128-bit ECB, to AES 256 GCM or Galois/Counter Mode, which is one of the stronger encryption algorithms available as part of AES. So, that's, that's actually a really good response from them.

Tim Callan

So, why would they have been on the weaker one to begin with? Is it better performing? Does it work better over light bandwidth environments? Does it save them money on servers?

Jason Soroko

Two-part answer, Tim.

Tim Callan

Okay.

Jason Soroko

The first part is, presuming they knew what they were doing, it might have been because of performance, but that doesn't really pan out as an argument anymore, because of the power of modern computers. But number two, I hate to say it, I just don't think -- I think they might have just thrown a dart.

Tim Callan

So, it might have just been an error. Pure and simple.

Jason Soroko

Yeah.

Tim Callan

Yeah. And I wonder about the same thing. And, again, if we go back to something real fundamental, like, you know, in the security section of your website publishing that you have end-to-end encryption when you just don't, when it's a pretty well understood term, and it's pretty unambiguous that you just plain don't, and nobody notices it because nobody cares that much and then all of a sudden, something happens and people start caring a great deal. Like if you take a charitable viewpoint, and you say, okay, maybe that's just some sloppy copywriting. Right? Maybe somebody on the web team who was tasked with writing some copy, just didn't do a very good job. Like, shouldn't these things be checked?

Jason Soroko

Isn't it interesting, Tim?

Tim Callan

Yeah.

Jason Soroko

What it took for it to be checked was the fact that Zoom became a verb in people's lexicon because of COVID-19 and because of the ease of use of Zoom, and hey, kudos to them for building software that works and works well, right? That's why, you know, everything from Jimmy Fallon to, you know, probably your family, now that we're all locked indoors is to use some sort of teleconferencing software to be able to talk to each other. Zoom seem to be quite popular and so therefore, really hard-hitting researchers really started to take a look at this and pick it apart. and that's when it was like, hey, this is problematic. And so, Zoom went from a security is the least of our problems, we're just going to throw a dart and who cares, to now being, I mean, Zoom themselves, the CEO, has stated, security will be foremost as part of their thinking.

So, you know, I hate to say it, if you're, if you're a software company, and you're doing any kind of key management at all, talk to an expert, because if your company is lucky enough to become a unicorn, and your company name becomes a publicly spoken verb, you're going to get scrutinized and that scrutiny is going to lead to really good researchers picking you apart. So, talk to experts first before you start choosing. You know, Tim and I are always talking about cryptographic algorithms. You might wonder why. This is why.

Tim Callan

Well, and then also, the other thing is, I know it's hard, like, it's a very jargony field and, you know, it's really easy to get confused about the different flavors of algorithms and, you know, here's this. These are the kinds of encryption and these are the kinds of hashing and these are - then do I really care, blah, blah, blah, but ultimately, you can't afford to just let your eyes glaze over because these words do have meanings. They have clear meanings. There's not ambiguity there. They're not opinions. They're meanings and if you use the wrong words, you're making false statements, even if you're doing it without malice and those false statements can be problematic. And that's what we saw here.

Jason Soroko

Yeah, Tim, and I'm sure you're very sensitive to that because of the fact you know, you, you work quite hand-in-hand with marketing. You're one of the voices of a company that has a lot to do with trust worldwide.

Tim Callan

Oh, yeah.

Jason Soroko

And you have to be incredibly careful with the words you use.

Tim Callan

We do our damnedest to get every little word, right. And I can remember, you know, plenty of occasions in my past, where I just like misspeaking, I just put the wrong noun in without really even realizing I did it, and getting roasted by people for it. So, it certainly is important to, to - - especially something like copy that's been written, that's been double checked, that's being published, like a technical resource needs to be reviewing this stuff.

Jason Soroko

Yeah. You're completely right. Get ahold of the experts. There's people who know about this stuff. There's people who can help you to choose the right algorithms and so, therefore, when researchers start knocking at, you know, knocking at your software, you won’t become the poster child for how to do it wrong. But Tim, you know, before we end this podcast, can we can we address one of the elephants in the room here?

Tim Callan

What's that?

Jason Soroko

Zoom bombing,

Tim Callan

Zoom bombing. By all means. A separate, right, independent issue, but definitely also a big topic right now.

Jason Soroko

I just wanted to address it because I think Zoom in their version five of their software has done a lot to address a lot of the issues. So, I'm not going to get into, you know, top 10 things you should do to avoid Zoom bombing, except to say that it got so bad that the FBI had to release a warning to everybody. You could do some internet searches on this problem and there's definitely some good advice out there in terms of how to avoid it. In fact, really, really good advice. Some of the, Tim, the reason I wanted to switch it in this direction was just to, to finish off on your thought about writing and being careful about what you write. There were articles out there written about, you know, Zoom is malware. And Kennedy and some of the other really good security researchers out there responded to that saying Zoom is not malware, in the sense that - - the one takeaway I wanted to give folks on this podcast that I think the research community is saying is look, we caught Zoom red-handed saying some wrong things, they've responded correctly. In fact, they've responded correctly very quickly. There's ways of avoiding some of these problems, especially things like Zoom bombing, but keep in mind, like any other tool, it's how you implement it and how you use it. So, there's best practices for using this tool and take that advice. Ed Giaquinto, our CIO, has actually published a blog on this and did a really great job talking about what you can do. I think Zoom, as the months tick by, is going to continue to make this you know, just better and better. They've got a really good track record of making things easy to use. So, Tim, I think the outcome of this is just going to be better software for everybody and better security.

Tim Callan

Yeah. And we've had this conversation with other people on other topics around the pandemic and lockdowns, which is the pandemic and lockdowns has forced a lot of people to do things that they probably should have done anyway, and tightening up security, if you get on a Zoom call, if we go back to time to say it's, you know, January 1, and you get on a Zoom call with the other key people in your organization, and you're sharing all the deepest secrets, and you're putting up the numbers and you know, you're a publicly traded company - gosh, somebody stealing that information would be incredibly damaging, right? You could trade on stocks, you could steal industrial secrets, you could do all kinds of really important things. So, this needed to be addressed and it's just because of the attention that it's gotten that it's being addressed now, but even when we're all not quarantining, right, even when we're all going back to offices and standing around the watercooler once again, this is still something that had to be there for us to really have the robust security that we thought we had.

Jason Soroko

That’s it, Tim, and you know what, I am going to propose right now, Tim, that in the very near future, we issue a podcast that's going to talk about using teleconferencing software with client certificates that are PKI issued.

Tim Callan

I love it. That’s a great topic.

Jason Soroko

Yeah. And so that's gonna include things like for you guys out there who are using Microsoft Teams, Cisco WebEx, and even Zoom and other teleconferencing software as well, the ways that you can really limit who is getting on your meetings, I'm going to go even beyond what you're going to read and headlines and tell you how to do it really securely.

Tim Callan

I love it. So that's a good teaser, Jay. Thank you for that. We will definitely want to record that podcast pretty soon. So, Listeners, keep your eyes peeled for that one. And in the meantime, if there are important developments and you know this conversation or other things around lockdowns, COVID-19, and how all of this is changing our security architecture landscape, we will definitely stay on top of it. Jay, as always, it's a pleasure to talk to you.

Jason Soroko

As always, Tim, thank you.

Tim Callan

Alright. Thanks, Listeners. This has been Root Causes.