Podcast
Root Causes 361: The Premise of on Premise
Hosted by
Jason Soroko
Fellow
Original broadcast date
February 9, 2024
In this episode we examine commonly held belief that on-premise systems give system administrators greater levels of control and that that is better for security or other reasons. We explore the pros and cons of extra control, to what degree it is a benefit, and if it's worth it.
Podcast transcript
Lightly edited for flow and brevity.
TCTim CallanSo, the title of today's episode is The Premise of On-Premise. What does that mean?
TCTim CallanAnd I'm not sure as we discuss those two groups, PKI software and CLM, I don't know that all the same factors apply to both of them. I'll be interested to get into that. But before we do anything more just for the listeners, what is the premise of on-premise?
So what really is the premise of on-premise? It is control. And a lot of people might say it's a question of control versus cost, but I want to expand that greatly. It's more than that. In other words, is control worth it? What are you getting for more control, Tim?
TCTim CallanRight. So the idea is that if I have it on my own hardware running on my floor, then ultimately, I have more control. That as soon as I put it into some kind of public cloud, I have less visibility. I have less decision making visibility, and my control goes down. Is that right?
TCTim CallanAnd there is an implied assumption there that more control is better, which I think we need to discuss whether or not that’s true.
So, November last year, November 2023, Gartner actually put out a public report saying that cloud adoption by 2028 will no longer be disruptive but will be essential. That's a big statement.
And let's think about now, like the financial industry being a really good example of hey, you wanted to sell into the financial industry, you had to have an on-prem product, otherwise, they wouldn't even look at you. Those are folks that they're absolutely, you know, control freaks for good reasons and want to be as secure as possible for very good reasons. But let me give you some examples. Nasdaq's GEMX options trading platform is now entirely in the cloud. You're starting to see an inflection point right now, Tim, where even some of the most stalwart industries such as finance are moving operations to the cloud, because I think they're asking the question, is extra control worth it? And are you getting anything other than more control when you keep things on-premise?
TCTim CallanWell, and again, is more control even better? Like, and you and I have discussed this in the security space in particular in the past. But do I really think that I - and maybe if I'm one of the largest banks in the world, I do think this. But if I'm a regional bank, and across a couple states, do I really think that I'm better at this kind of thing than the people at Google or AWS or Microsoft?
TCTim CallanOr Sectigo? Sure. Right. Yeah.
You know what they're good at? Giving customer service to their banking customers. I certainly as a Sectigo employee, I can't do what they do but they shouldn't pretend to do what I do either.
TCTim CallanRight. And just as I wouldn't think about sitting and building my own servers, right? Why am I doing this? Yeah, exactly.
Speed to market. I think if you go to cloud, Tim, it's remarkable right now the amount of innovation that is just absolutely being driven by, my goodness, I can stand up systems so quickly and not have to worry about individual servers in my server room. It's just truly incredible.
TCTim CallanSure. Speed to market. I agree. That's one.
So I think we could pass that real quickly and move right on to I'm going to combine three important IT topics in one. I'm going to say updates, as well as availability and resiliency.
So you know, is there anybody out there who's enjoyed a patch Tuesday?
TCTim CallanYeah. Uptime and security. Absolutely.
TCTim CallanOk. Can I try to add one or two more?
TCTim CallanFuture proofing out. So do I think I'm going to be the better person to stay current with developments, either in general in hosting a cloud platform as opposed to the folks at AWS? Do I think I'm going to be better at staying current in terms of a specific niche, as opposed to the folks that offer that SaaS product? So I think future proofing is a big one that we need to talk about for sure.
TCTim CallanAnd then lastly, is just efficiency. You know, you and I just recently did a pair of podcasts about the pernicious IT skills gap that plagues every industry, every segment, every geography in the world and this helps me mitigate the impact of that skills gap.
Let's talk about now two things. One is, I think when you're dealing with a, not just a general public cloud, but if you're dealing with security, something that's fundamentally important in your infrastructure, such as PKI, CLM, etc., if you go on-prem, you're not gaining a security partner, necessarily. If you're dealing with somebody who is managing that for you in their environment, and is essentially then to you a cloud service, you're not gaining a security partner. And I think that's an opportunity cost of on-prem to some degree.
And some of that, of course, wraps in efficiency and innovation and scalability. Right? That's part of what it is. But security is something really specific.
And I think there's one more I wanted to add other than security itself, which is, you know, for those of you who think that backwards compatibility, those of you who are telling me, Jay, I need to have my on- prem because of backwards compatibility, I'm going to argue to you that you actually don't have to, in the case of PKI specifically, don't have to give up backwards compatibility when you're dealing with a cloud PKI provider. Somebody who can stand up private Certificate Authorities for you.
So in other words, what's the opportunity cost of control? I can tell you one thing, that it's not. You are absolutely not going to lose by backwards compatibility by working in the cloud. And there's ways that I'd like to talk about that, Tim. But that is the list.
TCTim CallanOk. Are there advantages to control that we want to list?
TCTim CallanThat’s the first thing I thought of is I have no choice. Someone who has influence over my success has told me I must, and I can't appeal it. Right?
TCTim CallanThat's the first thing that occurred to me.
So what is the real need for control? Oh my God, I almost hate to almost say it, but I'm gonna say it. I think it's a need that is more connected with human nature, emotion and psychology. In other words, I need to wrap my arms around the server to feel like it's safe and I don't think that's true. But you know, I can see how the psychology of some people think that that is what they're getting out of control.
TCTim CallanYeah, absolutely. And we see so many aspects of this. This is how come in horror movies, the scary monster is always the one you don't see. Right? It's this lack of knowledge, lack of information, lack of control. It's just fundamentally anxiety creating for people in a real basic way.
If you think Patch Tuesday is a good thing. It's not. It's a huge risk every time you go in and do your official patches. And I'll tell you, it's no different than if you're running CLM software or PKI software. There's gonna be updates. If you're not hosted in the cloud, if you don't have the experts doing it for all of you and making it seamless, there is risk that I don't think people are calculating correctly.
And Tim, I gotta tell you, I know you keep up on it. I keep on it. But this whole world about infrastructure as code, we've had podcasts about that. Topics such as continuous improvement, continuous delivery, CICD pipelines. I think you need to rethink how you consider how cloud operations and how cloud applications are actually implemented. And it just blows the doors off of anything that happens on-prem. And that's just the truth.
In other words, I find it interesting, Tim, that the best controls now for how to actually implement software are now the state of the art is pointing to the cloud. Not on-prem.
Let's talk about security partners for a moment, right? Do you really want to roll your own? If you do, my God, you better have a very, very specific reason.
TCTim CallanRight. And this goes back to my earlier point about do I really think I'm better at this then? Who employs more career security engineers? Amazon or me? And unless I'm one of the largest financial institutions in the world, the answer is Amazon.
TCTim CallanWell, and this is the one we just hit. If you want to take it back to PKI, we just did an episode about should a private PKI service provider, do anything the customer wants, and what we came down on the end of is probably not. There should probably be some boundaries and you should say no, I won't give you that product. The very fact that you're working with someone who is expert in that area, is an advantage over sitting and trying to make these decisions yourself. Maybe you do make a poor decision.
And you and I had a podcast about what happens if the customer asks for things that are just crazy. Is it ethical for us to allow that to happen? It touches on that topic. And I would say that you want to speak to a security partner in order to understand what the same defaults are for your environment. Don't do it yourself. And let's go further, Tim. Do you have staff that can set up and maintain an HSM? That's such a specialist thing.
TCTim CallanAbsolutely. Is your staff going to try to learn on the job? Are they going to learn as a little bit of a hobby? Would you rather have somebody, you know, doing that? Taking a Pluralsight course and then cracking his knuckles and getting into it? Or would you rather have somebody who has done this hundreds of times?
TCTim CallanOk.
TCTim CallanYeah. Sure. Absolutely. Yeah.
TCTim CallanI mean, and that's such an interesting one. Just as an example of the more general principle, you know, it's something that's going to happen to everybody. There's no avoiding it. Nobody knows exactly what it’s going to shape like. Like nobody in the world. There are, you know, real, sort of intense, technical and mathematical and standards based conversations that are happening. How is the average IT department ever going to track that stuff? Like, that's just, that's a hard ask to ask somebody to be current on what's going on with PQC and what do I need to do to make sure that my organization is ready for it, and I take a sensible PQC strategy. That's just a hard ask for anybody.
But keep in mind that when you talk about PKI, you can also subordinate a modern issuing CA that's in the cloud to expand your capabilities. And so therefore, there's lots of ways to augment, monitor your legacy environments, as well as to actually create new issuing environments that are modern, able to integrate into all sorts of use cases and operating systems that legacy on-prem was never able to do.
TCTim CallanAnd doesn't that also tie into a future path of actually getting off of those legacy systems that I don't want to be on? That I can augment that with a modern, cloud-based SaaS-based solution with the idea that somewhere along the line, I'm going to pull the plug on that thing on my floor, and just use the SaaS?
I don't think you have to give up anything anymore. By going with a modern, cloud-based PKI, I think that you get all that backwards capability and you get all the future and innovative capabilities, and future proofing that you were just talking about Tim. You can get all the benefits. I'm gonna make this statement. Anybody who is architecting something net new right now, even in big finance, for example, I'm not sure that on-premise is your best solution. And I'd argue it comes down to psychology.
TCTim CallanSo what about the scenario - and maybe the answer is this isn't actually real scenario - but what about the scenario where I agree with you, I'd much rather do it this way but I feel I have some kind of external, let's call it regulatory, or standards or SLA requirement such that I feel that I cannot.
TCTim CallanMaybe you need to think about what you can do to open their eyes for them. Who can you contact? What body can you get involved in? You know, like places like PCI, like you can go join them. You can walk in and say, I think this is a bad regulation, it should be changed and become a member. Organizations can do that. And the big well-resourced ones do.
TCTim CallanSo even if I feel like I'm stuck by a regulation today, number one, maybe I can get involved in explaining why that regulation was fine a decade ago or two decades ago but isn't fine now. The other thing is, if I do believe that it's only a matter of time before these regulations change, maybe I start thinking about how do I gear my strategy and my architecture and my decisions now, so that I'm not going to be locked in and stuck, but I will be able to take advantage of these changes when they do occur.
TCTim CallanYeah. You’ve got to imagine. And that would be another one. If the problem is your auditor, and your auditor’s interpretation of the Reg, because there's two things, right? There's the actual wording of the regulation and then auditing firms, either individually or in consortia, come up with their own interpretations of it. It's like, you know, it's kind of like there's a US Constitution and we all know what the words say but then there's a body of law that interprets what those words say. The same kind of thing happens with these regulations. And the, again, that's another opportunity to be involved. You can sit down with your auditors. I've done this with auditors, you know, on the WebTrust side. You can sit down and explain why you don't agree with their viewpoint and sometimes you can change their viewpoint. And that's also a path forward to think about if you've got a regulation as a problem.
So let's arm you then, Tim. Let's arm the audience with the big elephant in the room, the objection you're gonna get, in terms of interpretations with regards to security. Ok. And let me just call this one right now. I have spoken to people who really truly believe and this is - - It's funny how this comes down to emotions and psychology and topics of faith, rather than real, measurable truth. But if you really believe that you can put better controls around a server on your on-premise system compared to a cloud environment, well, yes, in the cloud, it is someone else's server, but the controls that are put around it are typically going to be far better just even due to the way that things are specialized and isolated.
Tim, you and I talk a lot about putting better locks on the door.
Well, if you have a mixed enterprise environment, as well as this, you know, crown jewels of PKI and CLM, what kind of locks can you put on the door? What kind of isolation can you put into place? If you can't isolate it enough, are you able to put the necessary monitoring into place? Do you have the dedicated staff to enable all the security technology? The answer for all that is, look, no, you don't? You don’t.
TCTim CallanWell, and also, isn't there something fundamental about that architectural decision, which is just the compartmentalization, the unavoidable compartmentalization of your network functions? The unavoidable walling off that occurs. You've just made lateral movement, so insanely, much harder. And isn't there just inherently value there alone?
Look. Human error. You have seen things like storage buckets, right, that have been exposed publicly. Well, that's not something you see in a dedicated cloud PKI or cloud CLM. Right? You're just not seeing that. In terms of configuration errors, yes, you can absolutely have cloud configuration errors that lead to certain kinds of security problems, but you know what, I consider that a scratch because you can just as easily have those mistakes on-premise.
And, you know, I gotta call this out. I think, Tim, one of the biggest bugbears people have about putting stuff from the cloud - in other words, somebody else's server - is how many times have we heard about insider staff at a cloud provider being malicious?
TCTim CallanYeah, um, I can't think of one.
TCTim CallanAnd that would be hard. There are a lot of controls in place to prevent that. Even down even down to simple things like locks on cages and cameras in server rooms. Right? And, also, in terms of IT-based, you know, software-based controls and logs and log reviews and there's a lot, there's a whole lot of effort that goes in to making sure that somebody can't just waltz in with a USB drive and walk out with your data. It's not that easy.
I'm gonna throw one more in. It's gonna get into the weeds a touch, Tim, but we've talked about, you know, because of the fact that a lot of these isolation technologies between a multi customer environment in the cloud have to do with things such as - let's just call it this. Flaws that are taken advantage of by bad guys within the virtual machine isolation.
And I can tell you that I've seen those talks at BlackHat and those are ridiculously difficult types of attacks. Here's the thing. They can be mitigated by patching of the VM systems at the cloud level, you know, all at once. And furthermore, there are other controls. Remember, I talked about better locks on the door? You still have to authenticate it. And so, in other words, this would be a very difficult attack. I'm never going to call it impossible. But if you were to compare it against lateral movements on-prem - give me a break. I saw the BlackHat talks around, you know, red-pilling and blue-pilling VM environments, and those were done by some of the smartest white hats in the world. And the mitigations for those kinds of things are so sophisticated at the big public clouds and at any of the providers that you're dealing with, Tim. So I would say it's a very in the weeds argument, but I wanted to call it out, right. Let’s compare it to on-prem, give me a break. That's a 10 times harder attack than a lot of the attacks that will happen on-prem.
So Tim, I just wanted to now just, you know, get into the security context a little more. I think some of the fundamental flaws on the on-prem argument are your enterprise network is probably nowhere near as isolated as you think it is.
I think that endpoint security controls are really a speed bump to the worst attackers that are going to be going after important targets such as your PKI, or your CLM, for example. I've also seen the argument from some people with the, you know, the premise of on-premise saying, hey, you know, I actually, I don't have a problem with staff monitoring my on-premise platform, because I've shopped that all out to a SecOps provider in the cloud. Well, isn't that hilarious, Tim? It's kind of like, oh my God, you now have staff that have, you now can wrap your arms around your servers. That's nice. But you're now remotely monitored and administrated by the cloud. And so what the heck is the difference?
TCTim CallanRight. Yep. Absolutely.
Tim, I want to call it just a few last things in terms of security. Let’s get really specific with PKI, for example.
TCTim CallanSure.
You know, there's a reason why we have elaborate key signing ceremonies in our industry. And the reason we do that is because we are better than all y'all. We do audit ourselves. We do all the technicalities.
TCTim CallanSpecific logging, specific process, specific logs, specific visibility, specific hardware. Yes.
TCTim CallanWell, and HSMs ain't cheap. Like that could be a big expense item for something that you use relatively little, and you might find yourself using 1% of the capacity of that piece of hardware, but paying 100% of the price tag.
Tim, physical security. Right? My God. What does it take to be a private – not only a private CA, right? What does it take to be a public CA, Tim Callan?
TCTim CallanYeah. There's hundreds of pages of regulations around it and it includes physical security and network security and process and, you know, employee background security and all kinds of things. Absolutely.
TCTim CallanRight. And the risk reward arithmetic, which is part of what you're talking about here, is different for a public CA, or for that matter, for a SaaS managed PKI cloud service provider. That's a mouthful. Than is for the average medium sized business. You may have in your medium sized business, have a different, you know, set of costs that you're prepared to indulge - monetary costs, personnel costs, etc., than you're going to see in those other services. So this is where consolidating that investment in one place where one company makes sure that they're really doing everything titanium plated, and then they service a larger number of customers could just be a fundamentally better way to allocate resources.
So if you're a CIO, if you're a CISO, if you’re Risk Officer, if you're a Director of IT, if you don't know whether your developers or ops staff are just simply choosing defaults in whatever private CA software that they're using - you shouldn't be doing it. If you can't answer that question - stop what you're doing right now. Because I can tell you right now, a Sectigo – and again, not an advertisement, just trying to give an example - we have vast, vast experience in all these use cases that you're setting up private CAs for, so helping to set up your certificate profiles and you trust models properly so that - - you know, because these are going to be systems that live on a long time.
TCTim CallanHere's a litmus test I would say, Jay. If you don't know what your CPS says, you probably shouldn't be doing this. Let's start with that.
TCTim CallanOr if you don't have a CPS - -
TCTim CallanYes. Exactly. Yeah. Good point. Well, I'm not going to tell you. You guys have to look that up on your own.
TCTim CallanYeah. Rogue CAs. We call them rogue CAs. You and I have used that word in the past. Yeah.
TCTim CallanAbsolutely.
TCTim CallanRight. Yeah.It's so ubiquitous. It's so available, right? And, you know, we're all doing some kind of DevOps cloud architecture. And it's just right there.
TCTim CallanRight. Yeah. And you may. You may. Ok, Jay. So we could say that the premise of on-premise is a false premise?
TCTim CallanYeah. I think so. And I think you can see where there was and you can see where people who cut their teeth, got their education, formed their attitudes and their habits in a certain era that isn't that far in the past could be making that assumption without reexamining it.
TCTim CallanAnd even there, right, even that we should think you may have resources and options you're not thinking about like we discussed. And also, you shouldn't expect that to sit static forever. Like those, even the most Luddite regulatory body, somewhere along the line is going to catch up.
TCTim CallanRight. All right. Ok, Jay. I love it. This is a great topic. I think this is a new one for us. And we'll probably be returning to it but that was a really good overview and framework today.
TCTim CallanAll right. Thank you, Jay. This has been Root Causes.