Podcast
Root Causes 116: Ripple20 Exposes TCP/IP Vulnerabilities for IoT
Hosted by
Original broadcast date
August 31, 2020
Ripple20 is a recently announced set of documented vulnerabilities in the early Treck TCP/IP stack, a popular choice for early IoT devices. Our hosts are joined by guest Alan Grau, who explains the significance of these vulnerabilities, the difficulties in dealing with them, and how we can improve to avoid these problems in the future.
Podcast Transcript
Lightly edited for flow and brevity.
Tim CallanWe are fortunate once again to have our guest Alan Grau, VP of IoT here at Sectigo. Alan, we want to talk today about the Ripple20 I guess I would call it a vulnerability set. Right? Ripple20 is the name ascribed to a set of 19 vulnerabilities that were revealed for the Treck TCIP stack and tell us a little bit about this. This was from July I believe and what happened here?
Alan GrauYeah. We are actually joking ahead of time, it should probably be called Ripple19 since it was 19 vulnerabilities, but Treck is a software company that has been around for many, many years. One of the products that they developed, and sell is a lightweight TCIP stack that is used in a large number of embedded and IoT devices. So many small lightweight devices utilize this stack.Tim CallanAnd they are constrained and so they like it cause it’s small and it lets them use their very limited computing power in a better way, right?Alan GrauYep. Exactly.Tim CallanOk.Alan GrauSo, I was actually familiar with this company back in the late ‘90s. We were doing a lot of work in the embedded space clear back then and they were one of the people provided a TCIP stack and so in some of the early versions of their TCIP stack, going back it looks like as early as 1997 researchers starting, well they announced here just a couple of months ago, a few months ago, some vulnerabilities that they found in this stack. So, they were going back 20+ years and looking at a version of software that was developed at that point in time and just found that there were some vulnerabilities in the TCIP stack that are exploitable.Tim CallanSo, if you are going back 20 years to look at a TCIP stack, is that meaningful? Like if I went back to a 20-year-old version of Windows I could show you all kinds of vulns but who cares? So why is this important?Alan GrauYeah. This is where the IoT world is challenging because there are a lot of devices that were built 10, 20 years ago that are still out in the world doing things and we are talking critical infrastructure, medical devices, all sorts of manufacturing electronic devices really across all the various vertical markets where you would see embedded devices. So, yeah, again, the advantage of the Windows world is any things, you know, just fall off the usage quickly.Tim CallanRight. Everybody throws out his PC after five years anyway. Right. So, it only goes back so far before it doesn’t matter. But are there a lot of 20-year-old devices that are still in use?Alan GrauWell according to this article, I mean they are saying there are potentially hundreds of millions of devices that are being used that could potentially contain some of the software. So, it’s a staggeringly high number.Tim CallanAnd those possibly are rally important things too, right? This is one of the things we’ve talked about in earlier episodes is often that really old stuff is the real foundational stuff, right?Alan GrauYeah. I mean they are talking printers – well, that’s not quite as critical, but things, you know, commercial aircraft devices, transportation systems, industrial equipment, healthcare, you know, we’ve got really, really, critical stuff.Tim CallanSo, if these are 20-year-old devices that are out there somewhere in the world with presumably 20-year-old ecosystem support around them, are they updatable? Like what do we do?Alan GrauYeah. So, if you really dig deeply into this the vulnerabilities were in some of the early versions of the software stack from this company. Their newer versions of the software stack they’d solve the problems, the vulnerabilities are no longer there. I don’t have details on the exact date when those solutions were, you know, when those problems were resolved, and solutions became available to address that but even that was still a long time ago right? We are talking certainly 10+ years ago where the fixes were available. I don’t know exactly, again. But the problem is of the devices out there very few 20-year-old devices had remote firmware update capability.Tim CallanRight.Alan GrauSo, chances are that if you are using one of those old devices there isn’t a way to update it or even if there is, you know, is the printer company that made my printer 20 years ago or the medical device that was built 20 years ago, is the company still in business? Are they still providing firmware updates for 20-year-old systems?Tim CallanSure. Is this unsupportable? Could they even do an update? Could they do a build if they wanted to? Absolutely.Alan GrauWell, maybe. Right? I mean because you are talking about you need to go get a compiler that will work on a chip that was 20 years old and that’s probably on that laptop that got thrown away 10 years ago.Tim CallanYeah, exactly.Alan GrauSo, it’s a, you know, this is what one of the big challenges with security in the IoT world is these devices live a long time and early on security wasn’t a huge concern and certainly being able to remotely patch the device was not commonplace.Tim CallanSo, are we going to have this same problem? Are we all going to be sitting when we are on episode 1000 of this podcast 20 years from now, are we going to be having the same conversation about things that people are making today?Alan GrauYes and no. I am sure that there is software being released today and developed today that have vulnerabilities that have not yet been discovered. Some of the things that are changing though is security has become a front of mind, is huge for developers as has been a strong recognition that you need to be able to patch these devices.Tim CallanYeah. That’s a big difference. Right? Is again, we always come back to this concept of agility. Crypto agility, certificate agility. Where the agility is just so much higher today than it was in the year 2000 or even for that matter the year 2010.Alan GrauYes. Very true. And we are seeing regulation and standards requiring software update and patching capability for a variety of embedded devices.Tim CallanSo fair enough and it’s hard and they are out there, and they are being used in large numbers still and there are probably a lot of them are foundational important, maybe some of them are only printers but maybe some of them are keeping someone alive. At the end of the day though, I feel like from I read it seems like some of these are very big vulnerabilities. Like are pretty important. So, something has to be done?Alan GrauYeah. And it is a challenge, right? I think the good news is people are aware of it. So, if you are a company that’s operating systems and devices that are fairly old there is at least some traceability to go back and say hey, you know, what’s the legacy of this device. Where did it come from and some ability to determine whether it is potentially vulnerable and then you are left with a couple of choices. Either you can try to make sure that that is on an isolated network and provide protection for the device by isolating it through using kind of traditional IT tools, but as we know that’s fallible or the other alternative is, you know, a rip and replace, which is expensive.Tim CallanYeah. And may be difficult. And maybe not all that easy. Maybe the device that you were going to must rip there isn’t something obvious to replace it with.Alan GrauWell, I’ve worked on software that’s orbiting our planet right now. Right? They’re in satellites. It’s kind of hard to rip and replace those.Tim CallanYou can shoot them down, I guess.Alan GrauYeah.Tim CallanThat’s what it would take. Yeah. Or I suppose you brick them and put a new one but that’s gotta be, that’s horribly expensive and difficult and has a space junk consequence and all kinds of other things. So, wow. Where does that leave us?Alan GrauWell, I think there’s two main points. One is where you can take some action to address the problem, you need to do so and if it’s a matter of changing some firewall rules or putting some network segmentation or ensuring that some of these devices are run in isolated networks then that’s critical. If you can update the device. Right? Maybe you can’t update it today but as you are looking at plans for new capital expenditures and network equipment that suddenly must be a much higher priority. So, there’s the mitigation factor which is critical and then there’s the, you know, what are the lessons learned and how do we not make these same mistakes.Tim CallanRight. Very important.
Jason SorokoOn that topic, Alan, what do you think are some of the major trends to avoid problems? There’s a lot more code analysis going on than there was 15, 20 years ago. We also have things up now that are automatically built with micro segmentation in mind. Languages such as Rust which inherently do not have things like buffer overflow problems. I don’t think it’s impossible, but it definitely is reduced because of the way that the language is actually designed. What do you think are the main trends towards avoiding these problems in the future?Alan GrauYeah. I mean there are many, as you said, and the things that you touched on are certainly important parts of that and in the embedded world, you know, starting to have security as a front of mind issue and not an afterthought is critical and is starting to really change. How that plays out is in many ways code analysis tools are an important part of it, you know, security best practices. Ensuring your device has Secure Boot built into it. Ensuring that your device is using strong authentication methods is an important part of it. Ensuring that you’ve got an ability to update the firmware on the device is a very, very critical piece of it.Jason SorokoAnd don’t roll your own TCP stacks.Alan GrauYeah. I think that’s one that we’ve gotten away from. I mean, you know, we are talking 1997, right, when TCIP was fairly new and the idea of a lightweight TCIP stack was something that wasn’t readily available, and customers were starting to get away from it and probably everybody’s first version of TCIP stack had vulnerabilities. I mean we certainly know that Windows did and when people first started building these stacks, they hadn’t been battle-hardened and that’s true of most any software. Right? The first version of anything is likely to have some vulnerabilities. It takes time. It takes security testing. It takes penetration testing to work through those issues and the key is that that first version or second or third version that hasn’t yet been battle-hardened isn’t in a piece of critical infrastructure equipment that is still operating 20 years later.Jason SorokoYeah. Satellites and pacemakers come to mind for sure.Alan GrauExactly. I mean and this article lists satellite communication equipment as one of the devices or device types.Tim CallanYeah. It’s very different than servers and desktop machines and mobile phones. All of which have a much shorter lifespan and are much easier to deal with.All right. Good to know. This is a fairly recent news item so I’m glad we covered it and is there anything we want to add to this topic, gentlemen?
Alan GrauNo. I think we’ve covered it. I mean the only thing I would say is the story when you first read it sounds bad and it is, but it is to recognize that there are companies that use this software that upgraded their equipment to newer versions. The company did address these vulnerabilities a decade plus ago. So, it wasn’t as flagrant as the article may at first sound but it was the long-lived nature of these devices that really made this a problem.Tim CallanYeah. The point is more the inflexibility of this kind of device rather than the individual decisions that were made by companies or developers at any stage in history. Right?Alan GrauExactly.Tim CallanRight. All right.Jason SorokoYeah. So, Tim, taking inventory of the devices you are using within critical infrastructure is just so important. I think that there are movements in the industry to do that, especially now because of things zero trust architecture principles where we are just not trusting what’s behind the firewall and part of not trusting is not just addressing everything with a secure identity, it’s also taking inventory of what certain kinds of devices are capable of doing, how they are privileged within your network, etc. because the fact of the matter is we could probably be reporting on bugs like this if not on a weekly basis then fairly often at the very least and so it’s probably reasonable to assume that most critical environments are gonna have some kinds of devices where there is known vulnerabilities of some level and you are just gonna have to live with it because there’s no way to swap it out so how do you deal with that. That’s perhaps a whole podcast unto itself down the road, Tim.Tim CallanGood. Good one. I love when we come up with new podcasts while we are doing existing podcasts. That’s always a great development. Alan, thank you for joining us and walking us through Ripple20 and what it is and what it means.
Alan GrauThanks, Tim.Tim CallanJason, always a pleasure to talk to you, of course.Jason SorokoAlways, Tim.Tim CallanAnd Listeners, this has been Root Causes.Stay informed with expert insights
Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!
