Podcast
Root Causes 95: Cryptographic Key Vaulting
Hosted by
Tim Callan
Chief Compliance Officer
Jason Soroko
Fellow
Original broadcast date
May 28, 2020
For PKI to be secure, private keys need to remain private. In this episode we explain "vaulting" for keys or other shared secrets. We touch on the vulnerabilities that secrets vaulting fights against and the common use cases for vaulting.
Podcast Transcript
Lightly edited for flow and brevity.
Tim CallanSo, today, we are going to talk about key vaulting.Jason SorokoFor those of you who work in DevOps, this is probably a subject that you already know a heck of a lot about, maybe too much, and really, it's about where do you put your secrets in order to automate. So, Tim, a lot of our podcasts talk a lot about automation. Automation, especially with respect to cybersecurity, and really particularly around key management. And so, key vaulting or secrets vaulting, or just simply, you know, the vault, is a concept that's used quite often in automated systems so that the system that is running autonomously, without a human being can grab its necessary secrets to log into APIs, log into, you know, TLS sessions, basically any form of authentication that's necessary. There's some credential that's needed. Where do you get those credentials? The best way to get it from is a vault.Tim CallanRight. So, you have those secrets, they're stored somewhere consistent and safe and you know where they are. And, again, the main thing is safe. There's something about that vault that makes it, shall we say, more difficult for someone to come and steal those secrets?Jason SorokoYeah, because the alternative is pretty rough. Let's say you've got an Ansible script, and you're automating some sort of cloud implementation of your application and you are logging into - - your automated system is logging into databases, might be logging into the cloud system itself. It might even be provisioning SSL certificates. That's a lot of systems to have to log into, and so, instead of hard-coding those credentials, quite often using passwords, perhaps for database systems, legacy database systems, and instead of putting private key information into your Ansible script, you can store those things in the vault. And, unfortunately, still, to this day, we are seeing hard-coded credentials within automation scripts. We're talking about Chef, Puppet, Ansible, you know, name, your favorite flavor of automation scripting, and you're definitely seeing a lot of credentials stored in there. Even Python languages.Tim CallanAnd when the credentials are in the script, obviously, first of all, anyone who has access to the script can go find those credentials, right?Jason SorokoAs soon as the bad guy gets a hold of your automation script, it's absolutely game over because you've given them the full recipe of how to get into everything.Tim CallanAnd on the other hand, with the vault, what the script does is the script calls the vault and indicates which secret it wants, but the secret itself isn't in the script.Jason SorokoCorrect.Tim CallanTherefore, if you get the script, you don't get, you might get other information that's useful to you, but you're certainly not going to get those credentials.Jason SorokoThat's right, Tim. So, like everything else in computer science, it comes down to an abstraction layer is a good idea. And so, you want an abstraction layer between your scripting logic, and the actual credentials themselves. The credentials themselves should be safely ensconced within a vault, and called when necessary, and never, ever should be hard-coded within your actual logic.Tim CallanSo, what are the main? I mean, you kind of touched on a couple of them already, but like, what are the main - what do I want to say? Environments. Use cases. What are the main circumstances in which secrets vaults or key vaults or credential vaults are being used?Jason SorokoSure. I would say the number one is in DevOps, as we just said.Tim CallanRight.Jason SorokoAnd that's particularly obvious, because if you are implementing automated cloud systems, then those automations will necessarily request credentials of some kind for all kinds of purposes.Tim CallanRight.Jason SorokoAnd I would say number two, and this is interesting, Tim; this is not a topic we've brought up yet on this podcast, but robotic process automation.Tim CallanOk. Ok. That's a topic we need to - - that's one of the things I've been meaning for us to hit, so can talk about that.Jason SorokoYou can just imagine, you know, back, you know, it wasn't that long ago, let's say yesterday, which wasn't that long ago, you might have a fair amount of your staff doing all sorts of repetitious computer commands. So, in other words, your HR department, your finance department, these kinds of departments like that, perhaps which in order to say onboard a new employee in the HR department, or in order to close the books at quarter end, as a finance employee, you might have dozens of commands.Tim CallanYou have a set of predictable steps that you must always follow.Jason Soroko100%. There's just tons and tons of predictable repetitious steps, that what happens if you just got rid of that?Tim CallanYeah.Jason SorokoWhat happens if your employee just triggers one event, which then causes an automation of the ten other things that had to happen in order to complete the task.Tim CallanRight, which is essentially a script. Like whether or not it's done through a scripting language or something else, at the end of the day, that's what it is, right? It's a recipe book. And guess what? That automated task may have to access various systems and that means it's going to need credentials.Jason SorokoWell, think about it, Tim. I could be a DevOps employee who in order - - I could choose to manually implement my cloud application, which is like, you know, let's say 100 steps. Or, I could close the book as a finance employee at the end of the quarter, which is another 100 steps. Each one of those things absolutely could be codified and in each of those steps, there probably is some sort of log into this, log into this, log into this, right? And every time you have to log into a new system, you have to grab a credential from somewhere. Well, for a human being, either you are using a client certificate from your computer, or your mobile device, or you're having to remember a username and password, which you're typing in manually and in the automation scripts, right, in robotic process automation, or in DevOps automation, those credentials have to come from somewhere. Ideally, they're coming from a vault.Tim CallanGotcha. Ok. So that's another that's another good one. When you go through this process, it strikes me that probably for single sign-on services, you have to have a key vault or credentials vault of some kind, right? Because they're doing the same thing.Jason SorokoYeah. That's it. That's it, Tim. It really is a very, very similar kind of concept. But the thing is, robotic process automation technologies have some vaults. There are vault technologies, say in HashiCorp. We have our own with our DevOps tools. And so, therefore, you know, the vault that you will use will probably be specific to the tasks that you're trying to solve.Tim CallanWhich means you're using multiple vaults. So, you used this example, I've got some DevOps use cases going on and I've got a vault for that, or more than one and I got a vault for my Ansible and my vault for my Puppet, possibly and then I might also have a vault for, like we said, some other application like my single sign-on apps. I might have a vault for keys I'm using on the public cloud service, like Azure or AWS. So, do I have lots of vaults? Is that what really happens in today's enterprise?Jason SorokoNo. Well, I would say you as a DevOps employee are probably running with one vault. You as somebody who might be triggering a robotic process automation set of steps, you're probably dealing with just one vault. And enterprise may have a very small handful, one or two, and depending on what use cases are being used but you as an employee are probably just dealing with one.Tim CallanSo, tell me if I'm crazy, Jay, but this strikes me as a potential problem. You and I have talked in the past about concepts like rogue certificates, or even rogue CAs, right? Somebody in the company stands up a CA, they're running that for some number of certs, other people don't know that CA exists and because of that, problems arrive. There's no governance or nobody keeps an eye on it, and the patches don't get deployed, etc. Is there the same potential here? Can we have rogue vaults?Jason SorokoTim, let me let me scare you. This is worse than a horror movie.Tim CallanOk. Please do so.Jason SorokoSo please, anybody who's a CIO, CSO, listening right now, this is where you got to take a note and maybe do some homework because there's a really, really good chance that your DevOps employee might have set up a Certificate Authority without you knowing it and the employee might not even understand the full implication of what setting up a Certificate Authority actually means. They might have just set up, you know, an OpenSSL instance, which is pumping out certs happily, and goodness knows what certificate profile is being used and where that CA is actually residing. I've taken a look, Tim, at robotic process automation software documentation, and guess what? The instructions are very similar. If you just want to, you know, set up a root certificate, just fire up OpenSSL, here's some Linux commands, you know, there's no guidance in terms of what an appropriate bit length is, you know, RSA versus ECC, etc. And just oh, this will just make it work. So, on you go. And you and I know what kind of a nightmare that potentially is in terms of governance and, you know, what kind of door that opens to the bad guy. Because what we're talking about here is these are the keys to your kingdom and when you're setting up just rogue CA's all over the place - and that's what your employees may be doing if they're following the instructions. And heck, Tim, I wouldn't even count on the consultants you have in to be doing this correctly. They might not be PKI experts.Tim CallanRight. And even if they are, they're not going to be here in a year. They’re consultants.Jason SorokoDid those consultants spend the time to set up full blown visibility from the CA they set up for a vault for either RPA or for DevOps and give you visibility into your main PKI governance for your enterprise? I doubt it.Tim CallanAnd did they document and did they train the appropriate people and did that training and that knowledge continue to perpetuate correctly inside of the organization without the telephone game after those experts have left?Jason SorokoGood question. I bet you - - these are the questions that you as CSO, if you can't off the top of your head, know the answer that question - that's homework for you.Tim CallanAlright. If there's a vault and the vault has all the material in it, all the really good stuff, it seems like that now becomes a really, really, really good target inside the enterprise. So, how do we know that our vaults aren't being stolen by bad guys?Jason SorokoThe way that you implement the vaults, and every vendor is going to have their own instructions for how to do things. But I would say that the most - - it probably at this point in time, it's probably important for us just to say, make sure that your credentials are not hard-coded and just sitting out there and you don't have some OpenSSL, rogue SSL, CA just sitting out there. In terms of how to protect your vault, Tim, how about we have that as another full-blown podcast?Tim CallanI love it. So, I think and I think that's a really good point, Jay, and we talk about this a lot, which is perfect isn't - - perfect shouldn't be the enemy of good, right? So, if we can sit and cook up with some hypothetical and ask hard questions about well, what if the blah, blah, blah, blah, blah, well, gee, it's better than sitting in your Ansible script. Right? So, if that's what we're choosing, then let's choose the one that's unambiguously better. And I think that's a good point. That's a great place to start and we can pick up in a future episode and we can get into more depth on this.Jason SorokoPerfect, Tim, and that looks like one we should put on our to-do list and we'll be happy to have that podcast soon.Tim CallanExcellent. So, thank you very much, Jay.Jason SorokoThank you, Tim.Tim CallanThank you. This has been Root Causes.Stay informed with expert insights
Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!
