Root Causes 254: Toyota Symmetric Key Exposed on GitHub

Hosted by

Fellow

Original broadcast date

November 8, 2022

In a recently exposed error, key material for a popular automobile manufacturer's PKI has been discovered on GitHub, resulting in exposure of sensitive information. In this episode we explain the dual errors that led to this breach.

Podcast transcript

Lightly edited for flow and brevity.

TCTim CallanToday, we are talking about a recent news story and there are probably multiple sources for this. The one I’m looking at is from the GitGuardian website and the headline reads, “Toyota Suffered a Data Breach by Accidentally Exposing a Secret Key Publicly on GitHub.” That was from October 11 by Dwayne McDaniel. So, Jason, what went on here?

Jason SorokoThis is something like we’ve reported in the past. A few times in fact. Which basically is for those of you who don’t work in software or work around developers, there’s a concept of putting source code into a repository.

And that’s typically done when you are working in a team so you can imagine how difficult source code control is when you are dealing with something as sophisticated as computer code.

And you can also then imagine that within computer code you quite often are logging into systems – databases, other servers, etc. That’s what computer code is often doing. It’s computers logging into things, getting information, doing something with it and then proceeding. The problem is this. You and I on this podcast, we almost entirely dedicate the podcast to talking about credentials and how to keep them safe and how to generate them safely and in this case, it looks like a contractor who was working with Toyota had uploaded a piece of source code – not an entire system of a car, let’s say, but a piece of basically to make a car do certain kinds of functions, was uploaded to a repository and there were two problems, Tim. Not just the one problem. There were two major problems here.

The one problem that is quite common is – and this has to be solved – the repository was never meant to be publicly seen but it was posted as public. A public repository. And a might say to yourself, well, why would you ever want to post something publicly. Well, open source code is you want other people to see it. You want to collaborate not just with an internal team, you want to collaborate with the world. And that’s great. The problem is that when you mean for it to be private and never be seen by anybody but your private team, you don’t want to share it with the world.

TCTim CallanSo, let me clarify, Jason. This is not so much somebody misusing GitHub and accidentally publishing something publicly that they shouldn’t be. This was somebody misunderstanding what code was meant to be public and what code was not. Is that correct?

Jason SorokoThat is correct. That was one of the major problems because there’s a ton of code up in these cloud repositories and a lot of it is set to private, which is great because that’s the way it should be and we don’t get to see what the second problem is.

And the second problem is, as I said, the systems, computer code is logging into databases, etc., not unlike this code we are talking about here in this problem, in this context. There was a key – and we are talking about a symmetric key here. Not an asymmetric private key but a symmetric key that was used to log into a database system was contained within that code. And I will tell you – this is something that we’ve talked about this podcast before – it’s not a best practice. You don’t hard code any kind of key into your source code. And, unfortunately, in this case, it was the double-whammy of they put the secret into the source code and then that source code was made public. And that enabled access into a fairly important system which gave various kinds of information about customers and some other things as well. You can read the article. But those are the two main problems with this, Tim, and, boy, I gotta tell you, I think both of those things need to be solved. It’s shocking to me that we keep reading about these problems. And, from a PKI standpoint, Tim, that private key is so important as well. Even this article was about a symmetric key. It doesn’t matter. We want to call it out whenever we see it.

TCTim CallanIt could equally well have been a private key. It just happened that it wasn’t in this case and, in fact, if you even just go back to just the Uber breach which is just a few episodes ago, one of the ways that that breach was enabled is that somebody did find asymmetric keys embedded in code. And then they could then use those.

Jason SorokoTim, exactly right. And going back to the article here to reference this, to give you an idea of just how crazy this is, this source code was uploaded in December of 2017. It wasn’t discovered as public – even though it was public the entire time – wasn’t discovered until middle of September of this year, 2022.

TCTim CallanThat was the other interesting thing that I was gonna touch on but you already got there is that this sat for a long time. Now, is there any way of knowing if this key or these keys were maliciously used?

Jason SorokoI think that’s a big question. Obviously, if an attacker would have gone and gotten all kinds of customer credentials and posted it up on the dark web or put it for sale, well, that’s typically the way that we get signaled, hey, there’s been a breach. In this case, this is the scary part, folks. And that’s an important question, Tim, because there is no way to know because since the credential was given away, you just have to assume whoever was logged into that repository was logged in the same way that a legitimate user would.

TCTim CallanThey look exactly the same.

Jason SorokoAbsolutely. So, it’s a terrible combination of I bet you, Tim, there’s probably enormous amounts of source code out there that has hidden secrets within the source code. We hear about this all the time. But when you combine that with the fact that you might have unbeknownst to you posted publicly rather than privately, that’s the bad combination.

TCTim CallanSo, by the way, while I’m thinking about it, the Uber breach episode is Episode 247. So, just very recently. So, you can go back and listen to that. But I also call people’s attention to our Episode 240, which is about a Hyundai asymmetric key problem but that wasn’t the same problem. It was similar in that it was keys and it was auto manufacturer and it kind of yielded the same result but that was due to a different error in Hyundai’s case.

Jason SorokoIn Hyundai’s case, they had used a known published private key from a very well, an extremely well-known public example from a NIST document. And so, therefore, unfortunately, it was a very easily guessable key and white hat researchers – they go look for this. And if the white hats are doing it, what you have to assume is the black hats are looking for that as well.

TCTim CallanYou want the white hats to find it before the black hats do.

Jason SorokoThat’s the hope.

TCTim CallanOr if not before at least they can find it and shut off that entryway for the black hats.

Jason SorokoThat’s right. Tim, I think in reflection with everything that we’ve just said, even in looking back at some of the previous attacks or previous white hat findings or even this finding which is now being reported for Toyota, one of the reasons why we call this out – this isn’t about naming and shaming. It really is about drawing attention to common problems. And you and I, we’ve done a well-known webinar. We’ve talked about this on podcasts. You know, pitfalls of PKI. Easy problems that can be made and I think this is just another one of these pitfalls of being a developer. When you are handling secrets, handling credentials, what are mistakes that - - learn from others. Learn from yourselves. And I think that’s why I think we are calling this out, Tim. It’s not about the name and shame. It’s about, hey, let’s learn from our mistakes.

TCTim CallanI agree with all of that. And, I just wanted to real quick point out that the https://www.sectigo.com/resour... is our Episode 104.

Jason SorokoVery good. Thank you.

TCTim CallanHow is that, Jay? We plugged three podcasts in a single. I think that’s some kind of record.

I think it’s a pretty simple story. It’s a story we’ve told a lot but, like you said, this does continue to happen now doesn’t it?

Jason SorokoIt does continue to happen and we are gonna call it out even if it’s public keys, private keys. Like why in the world make the distinction. There’s so much automation code going on, Tim, that’s logging into all kinds of systems. There’s secrets. There’s credentials everywhere. Sometimes it’s those dastardly keys that are just so easy to put into source code. Please check into secrets repositories, credential vaults. That’s the right place to put. Use an abstraction layer for your secrets.

TCTim CallanAnd the amount of automation is going up and not down. And if you’re a believer in the everything is code philosophy then there’s just more code and so, this particular risk feels to me like it hasn’t really been figured out. It hasn’t really been resolved and at least for the moment we should not expect this to go away.

Jason SorokoI don’t think it’s gonna go away. I do think though it does show that the maturity level of a lot of different developers – even though they might be writing great code, the maturity level when it comes to security matters is still low in many places and those skillsets are very unevenly spread across developers and across even teams where there’s peer review. You gotta watch yourselves, folks. These are very, very solvable problems. This doesn’t take a lot of money or a lot of technology. This just takes better best practices.

TCTim CallanThat’s a good point and we’ve touched on that a lot. That people who ultimately are doing PKI – let’s say, for instance, if you are in a DevOps environment, who are probably very good developers but they are not particularly grounded in security or PKI and that’s where you see a lot of these problems occur.

Stay informed with expert insights

Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!