Root Causes 217: What's the Deal with the Recent Okta Security Breach?

Hosted by

Tim Callan

Chief Compliance Officer

Jason Soroko

Fellow

Original broadcast date

April 14, 2022

In March the LAPSIS$ hacking group convincingly announced a breach of Okta systems, potentially exposing Okta customers to additional compromise. Despite Okta's initial statements to the contrary, it ultimately turned out that up to 366 Okta customers may be affected. Our hosts walk through the events of the attack, how it unfolded over time, and how this breach was revealed.

Podcast Transcript

Lightly edited for flow and brevity.

Tim CallanWe wanted to talk about a news item that earned a lot of press in late March, and I believe got a lot of headlines, and we just want to clarify, talk through and clarify exactly what went on with it in one neat and encapsulated place, and what we are talking, of course, is the Okta breach. If my memory serves, I think it as March 22. Does that sound right, Jay?

Jason SorokoMarch 22, Tim, is I believe the date that the attackers released screenshots, and that’s what set off a lot of these events.

Tim CallanThat’s when the story broke. It’s not, certainly there was stuff that happened in advance of that and arguably, possibly well in advance of that. But that’s when the story broke. So, it’s March 22, people get up, they’re expecting a normal news cycle, and instead, they get some news. What is the news that we learned, Jay?

Jason SorokoWhat we saw on Twitter was some screenshots that were attributed to a hacking group. I’m only going to mention their name once, just because I’m not even sure of the proper pronunciation, but Lapsus$, L-a-p-s-u-s-$ is one of the names that they go by. Microsoft even has a different name for them. So for the rest of this podcast, let’s call them the attackers.

The important part to this is that the screenshots seemed to suggest that they had achieved a pretty high level amount of privilege within Okta’s systems. So, in other words, Okta being the big one-time – the, the – sorry.

Tim CallanSingle sign-on.

Jason SorokoOkta being the big single sign-on cloud provider and of course, so many enterprises use them for probably a very good reason. And these screenshots, as I say, seemed to suggest that the attackers had achieved some level of super administrator privileges within their environment. And I think, Tim, what’s important to note here, and this is what has come out through a lot of the news is that the victim in this case was not Okta systems directly. However, it was – what was in the initial Twitter release by the CEO of Okta had basically said, yes, this is one of our co-processors. And what that turned out to mean was, it was a third-party

Support agency called Sitel that actually had been compromised. So, what we’ve learned, Tim, and, I’ll just tell a little bit of the detail, and we can talk through some of this. It looks like what the initial facts are, and these are the important, these are high level facts, is that somebody at Sitel was socially engineered at some point, and their credentials had made it on to the dark web. The attackers, the second set of attackers, being this Lapsus Group, used those credentials to then be able to get into systems within that third party provider who had, of course, the ability to do things such as resetting passwords of Okta customers.

Tim CallanAt that point, once you can do that, basically, the implication, to me at least, would be that the entire Okta environment might be available to you. And once the entire Okta environment is available to you since what it does fundamentally is give you access to other systems, then the implication there is that there might be other downstream system that you could gain access to as well. Is that right?

Jason SorokoThat’s right, in fact some of what we’ve seen subsequent to the initial news releases, is that yes, there were some other movements that the attacker had made like that. Not just within the systems of this Sitel but also within some of those customers. And so, this is me being very careful because there is a lot of information floating around in the Twitterverse and in news releases that are based on leaked information from incident handling reports and what with all, Tim, so I want to keep this on a slightly higher level than just the actual facts of the attack because that’s interesting enough, but I think what’s not talked about enough is the actual series of responses by Okta to this event. So you had the initial response from the CEO which said, hey, this is something that affected a co-processor of ours, a third-party, that happened back in January. We’re aware of it, and we had seen an initial investigation report.

Tim CallanMy recollection is that they said very early on, like maybe in that first message from the CEO, this is not an ongoing attack. There’s not an ongoing threat. Correct?

Jason SorokoExactly, Tim. Exactly. And so, then, of course, the Twitterverse and others basically put a little bit of pressure and said, geez, some of those screenshots are kind of scary. It suggests that a lot more went on, and how can you be absolutely sure that there wasn’t something worse or that you do have an issue with some of your customers? Well later, on October 22, Okta apparently received the complete investigation report from Sitel and ended up revealing that a certain percentage of customers probably needed to be contacted and might be affected.

Tim CallanNow, this is the same day, later March 22. Correct?

Jason SorokoThen a little later - the exact timing, that part of it’s not in my head, but what I can tell you is that there was an online - I think it was a Zoom call between Okta and interested parties where, yes, it was revealed that 366 customers might be affected. So it did go backward from the initial there’s nothing for anybody to do and no, oops, it looks like now that we’ve looked into this, it’s worse than we thought. And the other customers, a certain percentage of our customers needed to be contacted, and we’re going to go do that. And so, that left a bad taste in some people’s mouths. I think that was the really, the big story here. The big non-technical story was just how - I think, and this is just me guessing as to what happened. Maybe somebody from Okta could correct me, but I think the initial response was this a happened to a third-party. It was pretty small and contained, and we’re aware of it, and then once the screenshots were released by the attackers and scrutiny started to emerge, Okta looked into it further and realized, oops, there’s something bigger here. And I think, Tim, that’s the heart of the story right now.

Tim CallanI think also another thing that I noticed that was interesting about this is that the initial screenshot poster who had some form of social media channel that they used to put these screenshots up, actually went up and put up subsequent posts basically attempting to discredit the claims that Okta was making about this breach and in particular, attempting to convincingly convey that though Okta is saying there is no ongoing threat and it wasn’t really a big deal, there actually is an ongoing threat, and it actually was a big deal. And so you got this kind of debate happening in these two different channels between the official word of the company and this attacker, this committer of a breach, this breach attacker who is on their own channel disputing the things that the company is saying.

Jason SorokoCorrect, Tim. And in fact what was interesting is beyond just what amounted to a correction by the attackers, which kind of looked like bragging but really it was an admonishment of Okta for, hey, you’re underplaying this. Within that same time period, March 22, Microsoft actually wrote and published a blog about what they called DEV-0537. That’s their name for the attackers. D-E-V-0537 and what was very interesting is that – I’ll make the small note here. In Okta’s own blog with their online response, they had said, there was a compromised laptop at our third-party. And even in the note from the attackers that was part of the admonishment, they said no, no, no we were using something else. It wasn’t a laptop. It was actually a virtual desktop-type of system, and Microsoft went on to actually help to confirm what the attackers were saying, making it clear that no, these guys prefer to do things sort of the easy way, which is to harvest credentials that are available on the dark web and then subsequent to that, start harvesting things like VPN credentials, BDI credentials, RDP credentials, going as far as sometimes to use MFA replay attacks, SIM swapping, etc. and using insiders. And it turns out that in the case of Okta, with their third-party, it looks like it was RDP credentials that had been harvested with something called the Redline malware, which is a stealer malware which basically would be able to potentially steal those credentials. So, I think it’s really a case of not looking at the implications carefully enough, and I don’t know if Okta was downplaying it in their own minds or if they had a legitimate reason to downplay it as much as they did, but I think - -

Tim CallanOr if it was just being spun.

Jason SorokoI’m not even going to speculate. I’m just saying what the possibilities could be. But certainly, it turns out that the implications were larger than had been initially talked about. And as time went on, more and more of the facts came out and then, Tim, just a factoid here, but the Twitterverse was rich in information with all this stuff, it turns out that that initial incident report, which I think somebody had claimed it had been done by Mandiant, actually went into fairly good detail about what the actual events were by the attackers to be able to achieve their objective. And that brought up a lot of information that was pretty consistent with the screenshots that we ended up seeing from the attacker’s initially on March 22.

So, the story continues, Tim. And just for those of you who haven’t followed along super closely, there have been arrests related to this attack, and apparently, and this was, I think Brian Krebs has reported on this. I know the BBC has reported on this. But an unnamed person in the UK has been arrested as well as some of, apparently, some of this person’s compatriots in South Africa. The reason we don’t know the name of the person in the UK that was arrested is because they’re only 16 or 17 years old.

Tim CallanSo, that’s interesting because that gives you a very different profile. We all have this image in our mind of the sophisticated attacker being a career criminal who is probably in some place where law enforcement is relatively weak or at least, cybercrime law enforcement is relatively weak, and what they’re doing is they are capitalizing on this status that they have. But that’s not with this at all. This almost sounds like your classic computer prodigy hacking for the joy of it including this bragging, if you will, or this correcting the record aspect of what was going on.

Jason SorokoTim, let me attempt to frame this because you’re giving a couple different ways to potentially think about it. Here’s the way that I’ve thought about it after now looking at all this, and I think there’s a little more background I have to give because this set of attackers is also attributed with having attacked Nvidia as well as Microsoft itself. Apparently they lost some source code to the organization. And I could just go on and on. In fact, I don’t even know if anybody truly has the full list of organizations that have been affected by this particular hacking group. And one of the things that seems to be common is that this group is successful, fairly boastful, they're not afraid to communicate, in other words, their methods of communication were not like some stealth organization, and if you take a look at those Mandiant notes, those supposed Mandiant notes, and I’m not claiming they’re true, I’m just claiming a lot of people have read them, and they seem to line up. One of the very common reactions by people that I respect in the cybersecurity community have basically said, look, their methodologies are pretty rough. In other words, the usage, the direct usage of Mimikatz , the direct usage of certain other kinds of systems to be able to get in steal credentials, etc. if you were the pro career criminal, super hacker-type, you might use other tools. You might use other methodologies in order to be more secret, in order to be more whatever. And just the fact that these guys chose the easy way, Tim, and this is expressed very clearly in the Microsoft blog, and I really invite people to read that in their blog about DEV-0537, published March 22. This perhaps, Tim, is the lesson to be leaned here, and maybe the crux to this podcast, which is I think everybody has a particular mindset about what hackers are like. Even certain categories of hackers and what they’re like. But there’s absolutely, absolutely a type of hacker out there that’s very, very scrary who doesn’t have to be ultra-skilled, doesn’t have to be gifted, doesn’t have to be any of the things that we typically attribute to these types of people. What we really need to keep in mind is perhaps cybersecurity is so poor overall that very, very minimal, somewhat “easy” methodologies to use are the ones that are perhaps the most effective against modern organizations with a lot of resources because there’s still a lot of back doors open. And this idea of the genius hacker, it’s not necessary.

Tim CallanI mean one of the things that occurs to me, and I think about this a lot, is if you are a large sophisticated born-in-the cloud enterprise - Okta is a perfect example of this - your attack service is vast. Everything’s digital. You’ve got many different systems and vendors and they’re all attacking each other, and in a lot of ways, the attacker has a badly unfair advantage, which is they only need to find one chink in the armor, and they need to only find one way in. And you need to armor everything without exception. And one little seam or one little molecular gap is enough to let somebody in, so your protection needs to be perfect, and that’s hard. That’s a hard thing to do.

Jason SorokoWell, Tim, I’ll tell you what. I’ll even narrow it a little bit. Not only, obviously, this asymmetric problem of cybersecurity, which is what you're describing, absolutely true. 100% true. But I guess what I’m trying to point out is that it’s even a bit worse than that because I think people get the sense that even the basics are being covered by competent defenders, and I’m not sure that that’s entirely true.

Therefore, if you’ve got legit username and passwords to very, very important administrators floating around the dark web, you’re already sunk.

Tim CallanSure. The battle was lost sometime in the past when those credentials were harvested, not when they’re used. At that point, you already lost the battle. You're just waiting for the follow on.

Jason SorokoAnd not only harvested because as you say, Tim, that’s if you’re dealing in a username and password world, that’s an entire possibility. We’ve talked about this multiple times in this podcast - keyloggers, stealer malware, all these things. Username and passwords are so bad for many reasons. We’ve mentioned that so many times. The problem is that these were searchable credentials, and what was very interesting to me is that there was a group, and let me get their name. It’s in my notes here, Tim. There was a group that actually had looked up the dark web after this attack had been made available, the information had been made available, and they looked up some of these subprocessors, coprocessors of Okta, and actually were able to find those credentials on the dark web which is just amazing to me. So in other words, there’s a lesson to organizations out there. There are professional legit, white hat type of groups who scourer the dark web for these kinds of things. Maybe you should plug into that level of intelligence and get feedback on a regular basis – hey, do I have any of my credentials floating out there right now? I think that’s a big lesson learned from this.

Tim CallanThen, so a couple more questions. One that probably is very predictable, which is presumably if they had been not using a simple shared secret authentication model that this attack would not have succeeded? Is that right?

Jason SorokoIf you take a look at how the attack was actually done in this case, that is correct. That is correct. A stronger form of authentication such as a true passwordless probably would have defeated this.

One of the issues, of course, that could be argued here is that this exact same attack group has been known to actually put out ads and hire people internally to these organizations to do things such as click yes on a push notification for MFA or to type in an OTP on behalf of the attackers. So in other words, these guys also use insider attacks which can bypass some of that. But on the other hand, as I say, the attack as it was done would have been defeated by a stronger form of authentication. You’re correct.

Tim CallanBut let’s return to that because that was another thing I was going to bring up. I remember seeing one of these ads, and it’s pretty incredible. Like, literally, they are posting advertisements, asking people to reach out to them if they’re interested in selling access into the trusted systems of the company they work for. Recruiting moles inside of companies. It’s kind of amazing to me.

Jason SorokoBy the way, I’ll give Microsoft credit for detailing some of that, so again, go check out that blog. But, absolutely. Insider attacks are real, and it’s not even, it’s funny. When people hear the term insider attack they think it’s the person inside doing the hacking, when in reality, it’s just one moment of sabotage.

Tim CallanThey’re just opening the door. They just got to unlock the door.

But also what’s interesting about that is when you think insider attack, you imagine like a guy knows a guy kind of thing. No, these are complete strangers. These are complete strangers that are being recruited online to do this.

Jason SorokoCorrect. People who look at ads that are probably put in very, very odd places. And responding positively. Like, I don’t know. It’s as if – is it a disgruntled employee, is it an employee who’s just looking for a few extra bucks? The damage to your employer is gigantic. It’s not something that anybody should be motivated to do, but goodness, it seems to happen.

Tim CallanIt seems, like, just with no information, just thinking about the psychology of it, it has to be either just a complete and utter mercenary attitude, not caring in any way or somebody who’s got some kind of bone to pick.

Jason SorokoThere’s a bone to pick somewhere. You’re exactly right. The thought of just making a couple of bucks doesn’t even seem right to me. It’s somebody who’s probably disgruntled in some kind of a way. Goodness.

Tim CallanWe don’t know how many bucks. Maybe it’s a lot of bucks. We don’t know that. But anyway, I thought that was amazing too as part of this story. It’s the first time I’ve been really exposed to that practice as a real thing that was really happening, and I thought that was an eye opener for sure.

Jason SorokoThere’s a lot more to this. We were even talking about the AWS cloud credentials that had potentially been found, lists of names of administrators that were found. I mean this gets into the mud raking of bad practices by either the third-party to Okta or Okta itself, and, I don’t want to – this isn’t about getting into that, and therefore, we’re kind of going to break our own rules of not getting right to the root cause, but I think that we need to let the investigation to this flesh out a little more, and then we can probably get into the root cause of this in the future. But I think there were some interesting aspects to the story we could start talking about now.

Tim CallanI agree. I mean, it was – there’s enough going on that it was worth talking about now. The story is not done at all, and hopefully it will develop, and the public will learn more about this. Sometimes we don’t. Sometimes it all just goes under the covers, and nobody ever tells you anything ever again. But if there are trials and stuff, then it might be that there’s public information. We’ll keep tracking it, and I think if there’s more that’s worth reporting, we can come back and give an update then.

Stay informed with expert insights

Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!