Podcast
Root Causes 67: Definition of DevOps and DevSecOps
Hosted by
Tim Callan
Chief Compliance Officer
Jason Soroko
Fellow
Original broadcast date
February 21, 2020
Our hosts are joined by senior DevOps engineer David Colon to explore what DevOps means in today's enterprise. They cover diverse aspects of the DevOps phenomenon, including cultural implications, "configuration drift," definition of release velocity, and DevSecOps. Plus of course how DevSecOps intersects with PKI.
Podcast Transcript
Lightly edited for flow and brevity.
Tim CallanSo, maybe for starters, maybe a good place to start is, you know, DevOps means - - seems to mean different things to different people. So, how do you view this word and tell us what DevOps means to you.
David ColonSo, DevOps to me means more of a cultural change in the organization of your development and operations teams. One might look at it as a shared responsibility where a traditional approach would be you would have a network team, an infrastructure sys admin team, a database team and what ends up happening is each one of those teams have different responsibilities and they are accountable for different things that do not regularly align with business goals. Same can be said for development teams. Usually, product managers talk to developers, developers now have to interface with all these silos. So, one way to break down that organizational structure is by having the right skilled folks in the same team so they can share the responsibility and accountability to align with the business goals.Tim CallanYeah. So, you know, in a way it seems like a simple concept, but it seems to be a concept that at least in some enterprises is causing people heartburn. Why is that?David ColonI believe part of it is change. I remember when I first started in the industry, and I don’t mean the PKI industry, I just mean general tech. One of the advice that was given to me was to be specialized. Be specialized in networking or database and that never really resonated well with me because I came from a startup culture where I wore many hats and I also felt that the jack-of-all-trades wasn’t a bad thing to have and it’s starting to prove out as time has evolved because a lot of DevOps Engineers are very skilled in multi-disciplinary actions and you can always get that consultant or expert to help you out on like really deep level technological stuff.Tim CallanYeah, and I think we are gonna come back to that point because one of the things about DevOps is you’re setting up a whole environment and so, you know, need to at least be good enough at all aspects of it or you’re liable to run into problems.David ColonYep, and that’s right. Another way of kind of looking at the pain point is these teams; they were structured probably geographically in different places so it makes it very hard to have them collaborate locally. So, if you took let’s say you made a super star team of network admin, a sys admin, a couple developers and had them associated in a let’s say product or service-oriented approach as opposed to a specific discipline those players may be in different offices, may not even be in the same time zone. So, it makes collaborating in real-time a bit difficult but thankfully remote conferencing technologies make it a bit easier nowadays.Tim CallanSo, is there also an org structure component to that where my boss isn’t the same as your boss and maybe my goals aren’t the same as your goals and maybe my process isn’t the same as your process?David ColonExactly. That’s what ends up happening. Network team really cares about making, you know, packet pushing or pushing packets I should say and, you know, the efficiencies of that may not have any big business impact. Does lowering the latency matter for that business product? Maybe, maybe not, depends on the service you offer.Tim CallanRight and this is where we have seen another trend, we have seen in IT which is big is imbedding, right. To break down your classic monolithic IT department and rather have departments that are there to serve business goals, business units, departments, and things along those lines in order to more closely align those things.David ColonYep. That’s exactly it.Tim CallanSo, to that question so how do we get around these problems? Like, I mean they seem like important matters, if your goals and my goals aren’t the same that could really make a difference. What is the right way to structure a DevOps practice so that we are not having that problem?David ColonSo, I have already alluded to it a little bit. It’s that shared accountability. I think it really goes a long way. If you put a group of different skilled engineers on the same team, they are now on the hook for that product so naturally all their business goals are aligned. So, it doesn’t matter how efficient the network is. If it’s not gonna benefit the business much now that network engineer can now focus a little bit more on the automation aspect.Tim CallanNow that can be taken too far though, right? Because I mean one of the things that we have seen, certainly we have seen classically, and I still see this and let’s take it back to security. Although, it wouldn’t have to be security which is to say somewhere you have a CISO who says, hey, you’re not having a problem today but that doesn’t mean you won’t have a problem tomorrow and so what we are gonna do is we are gonna make sure that you are secure even though maybe you and the BU aren’t thinking about that. So, we still need to have some level of that external expertise accountability and part of this process as well, right?David ColonYeah. So, one of the cool things about having a product specific team is whenever they have free time meaning let’s say they finish all their features from product management and they get to clean up, which I know sounds like a dream in most organizations, the good thing about when they are focused, when they do have that downtime, they also have this autonomy where they realize, hey, I know I implemented a solution and the security was alright but I wasn’t happy with it. Let me revisit that and make it better and that’s the beauty of being focused on one specific product or service or at least maybe two but not more than that. Because when you are focused on many products you are going to be pulled in many different directions and not be able to revisit that thing that, you know, you just got it done just to meet the deadline, but you never got to architect it the way you wanted to.
Jason SorokoSo, Dave in terms of outcome what in your thinking, your experience in your daily job, what do you think is one of the outcomes that DevOps specifically has been giving you? Is it velocity of being able to put things out? Is it - - and is there a different outcome for different sizes of organizations?David ColonI believe the outcomes are mainly the same. So, there is this really great book if I am correct that’s called Accelerate DevOps and it’s a statistician’s approach to measuring DevOps in high performing companies. They were able to identify a couple key patterns and one of them you alluded to which is velocity. Now, when I first read the book, I thought of velocity being as the number of production builds but what I quickly realize is it means the number of builds built against any environment. It could be development, which is what it’s normally for. So, a developer does some code, and they want to make sure it doesn’t break anything so they push that code in and automation magic happens and it quickly gives instant feedback to the developer on whether the code did good or bad.Jason SorokoThat’s really handy, Dave, to know because, first of all, of course, Tim and I have spoken about this in webinars and previous podcasts before so it’s consistent with what we have been saying but now when you are talking about types of environments, the complexity of environments, complexity of operating systems you are talking about the advent of yourself in tech, I remember myself in the earlier days where just about everything was Microsoft Stack and we weren’t having to worry about the Cloud, we didn’t have much else except server blades full of Microsoft servers and we knew what dependencies were available to us regardless of what we were building. That’s not the kind of world you are living in today, is that right?David ColonYeah. It’s a lot different. One thing about DevOps is a lot of the concepts in DevOps existed before it was a coin phrase. Companies have been practicing configuration management and those tools existed before DevOps and DevOps just helped accelerate this and put it under a certain umbrella. One thing that because advantageous with the Cloud is this concept of immutable infrastructure. So, when you are using configuration management tools you have this concept called configuration drift where what’s in source code may be a little bit ahead of what’s in production. Where with the concept of immutable infrastructure you basically destroy what you have and bring up what’s new. So, a good example of this using your Microsoft analogy and I have never really been a true Microsoft sys admin is that dreaded Windows update. Instead of logging onto like a Windows server and doing an update, immutable infrastructure, what happens in immutable infrastructure workflows is you actually create a whole new build that has the updates already built in and then you just change let’s say a load balancer to point to the new server and if things break you can change it back if you haven’t destroyed the old server or just build up the old server from an older revision and then point to it again and see what update actually broke it. The same philosophy is true in a Linux kind of workflow.Tim CallanSo, what about - - so we also hear a term which is Dev Sec Ops, which obviously is DevOps with the word security stuck in the middle. So, define Dev Sec Ops for us, Dave.David ColonSo like DevOps, Dev Sec Ops has security baked in through the whole process, so I’ve normally seen a lot of the products and services around the pipeline. So, one big thing that DevOps borrows from is the manufacturing industry and I believe it comes from Toyota. So back in the 80s when Toyota and Japanese manufacturers were taking over the automotive industry in America, one of their, I guess innovations, was around quality control and pipelines. They had like this factory line of how cars are constructed, and a lot of DevOps borrows from that methodology. So, in the past, if you had let’s say a sys admin team and you told them, hey, build me a Linux server, you might have that team if you ask everyone, they might build you different flavors of Linux and by having a pipeline you get the thing exactly the same every single time. Which is much like a factory assembly line. With that said, security is usually Dev Sec Ops and their tools are usually part of each part of the pipeline. So, it checks the code, it checks the infrastructure for vulnerabilities. Now, that’s how I have seen it today but what it really means to me is kind of what I have always thought of a security conscientious company is that security is baked into every part of the process. So, every discipline should be security focused.Tim CallanYeah and this seems like a simple concept right and this isn’t a new concept, I think to any of us on this podcast and probably most of our listeners which is, you know, design in security from the very beginning, but certainly the way I hear it is that now that we are in these new environments and now that in a lot of ways our responsibilities have changed in terms of whose doing what and how we are doing it that sometimes that idea gets forgotten or lapses or isn’t really properly done and to some degree what the word Dev Sec Ops is just to say, no guys, all of these ideas that we had like design security in from the ground up are still valid and still apply and we need to do that in this environment and this methodology, too.David ColonYeah, and that also boils down to a concept known as like each organization or each team’s definition of done. If a product manager says hey build me feature x and give it to me tomorrow people will definitely neglect security, but if there is a good policy and governance that says you cannot call feature x done until security has checked it out whether it’s through a manual security team approach or using security automation tools which DevOps being automation focused Dev Sec Ops also tries to be automation focused.Jason SorokoSo, Dave, you’re a practitioner of DevOps within a PKI company, which as Tim said, gives you kind of a unique advantage to see the whole spectrum of DevOps and security working together because at least you are very, very conscious about it and I would love to hear your take in terms of one of the - - what’s an important aspect. There is all kinds of security aspects within Dev Sec Ops. I have actually spoken to analysts who had a great chart of ten different areas that somebody needs to keep in mind. Keeping one in mind is hard enough but in Dev Sec Ops there was at least ten. One of them that comes to mind for me is the identity management. This is a PKI podcast. A lot of people listening to this are familiar with things like TLS certificates and mutual TLS authentication. When I am thinking of containers and especially these orchestrated containers, I am thinking about making sure that those certificates are getting into those containers the right way and the visibility of the certificate profiles underlying those certificate authorities. Would love to get your take on how important that is and especially with respect to the definition of Dev Sec Ops and where that fits.David ColonYeah. So, that’s definitely one that I kind of wish I had a direct line of access to product managers back when we were Comodo because as a sys admin back then I wanted to use the PKI infrastructure to authenticate something like our SSH login process and I know the question that you have asked was pertaining to containers but I just wanted to divert it just to something simple that most people understand which is SSH logins. So one common practice that I saw in a lot of organizations is if I am a brand admin and I want SSH into a production server I am gonna get that SSH key fingerprint that you trust in. I have never met anyone or seen anyone stop and say, hey, can someone verify that key fingerprint? One thing I didn’t know when I joined Comodo, the CA business was that PKI has solved this problem already and it authenticates identities. So, Open SSH actually had that baked in but one of the things that was hard for me to find is a way to automate PKI especially on- premises or if there was any paid solution out there that did this, and that same struggle was felt in a lot of different areas like containers. Containers are very useful and immutable infrastructure and that pipeline that we just about that’s exactly where it would be in when part of that pipeline step is to, that container gets brought into life, it checks in, it authenticates and now it has identity. So, whatever services it needs to talk to it’s already authenticated and trusted.Jason SorokoI think that SSH analogy is perfect. So a day in the life of somebody who might be a developer or even on the ops side, depends on how you want to put the Venn diagram, that as I think you have spoken about before Dave, the day in the life of that kind of person would involve all kinds of certificates right? The SSL certificate to the webservers, the load balancers, SSH out to the Cloud resources and now we are talking about mutual TLS authentication certificates which might be very, very short lived. So, Tim, you’re of course, you have got all the experience in the world with public and trusted certificates, some of these lasting two years to 90 days, SSH can last perhaps too long especially if they are keys and they are not wrapped in a certificate, Dave, right with an expiry date. They could be lying around for a very long time and now we are dealing with what could potentially be rogue certificate authorities with not very well thought out certificate profile structures, which is something that I know Tim and I have talked about quite a bit in the past on this podcast. So that’s a lot to have to worry about and that’s just one of the ten categories that I alluded to earlier Dave. So, is that a good way of thinking of just a day in the life of a DevOps practitioner in terms of identity management, what they might have to think about?David ColonYes, that’s definitely a good view into it and shows how much is on our plate and why being specialized isn’t necessarily a good trait to have. I think it’s important to be specialized in one discipline but be general on everything else because you do have to think about all these aspects and how they relate to one another.Jason SorokoSo, I think you talked about toolsets, so you know I think in the earlier days of DevOps it was, you know, there was an enormous proliferation of tool sets. I know that’s one of the challenges that you have probably had to deal with. But in terms of Dev Sec Ops is there another layer that we need to have, or would it be better for you to have security tools, in other words, this identity management concept that’s baked right into the tools you are already using today?David ColonSo, this is a conversation I have had with a couple engineers especially that are focused on the security side, not even in the Sectigo organization where this concept of is it better to have like an on-prem tool versus a centralized Cloud tool and I think that concept definitely affects your decision of how you architect things because if you had this Cloud-based tool and you have this pipeline what happens if the Cloud service is down? Your pipelines just halt and that is not a very good thing. Therefore, I can see how some people may want the on-premises approach to this. The problem with the on-premises approach is there isn’t that many products traditionally that existed for this sort of problem - that aims to solve this sort of problem. Therefore, a lot of shops ended up creating their own solutions. I know one of them that I did for my own home lab was using Cloudflare’s SSL library to do my open SSH server authentication stuff.Jason SorokoYeah. When you are having to cobble together a tool, toolsets from around the internet things can get difficult. This whole principal that you talked about which I think is really core to what makes DevOps powerful is that immutable releasing of software, that getting rid of problems such as configuration drift, I really like that term. I think that really the underlying toolsets that you are talking about really shouldn’t be - - I would hate for them to have to be cobbled together. I hope that the very near future gives a better pane of glass into these kinds of things that I know that are on your plate. Things that bring automation so that immutability really is possible and this big promise that DevOps gives us can be delivered. Especially is a way that we no longer, Tim, must talk about Dev Sec Ops as a separate subject, right?Tim CallanYeah. Like in an ideal world nobody would ever say Dev Sec Ops because it would be pointless but the very fact that this is part of our vocabulary, I think is an indicator that we are not all the way to home plate on this one.Jason SorokoI think that’s right. So, Dave, where should we take this conversation in a future podcast?David ColonI would love to expand on the SSH server access mainly because you alluded to certificates expiring and a common trend is no one should have access to production servers but if you need access how do you audit it, how do you ensure that the person who gets access their authorization expires in a timely manner. So, I would love to talk about short live access to production systems and how PKI can help in that aspect.Jason SorokoIt seems like a modern take on the principal of least privileges, Tim.Tim CallanYeah. Yeah. So that sounds good. That sounds like an excellent topic for us to pick up that’s nice and meaty that deserves to be a podcast on its own and, at this point, I think this was a good introduction to the idea of Dev Secs Ops and how PKI and DevOps intersect with each other and thank you very much, Dave.David ColonThank you, Tim.Tim CallanThank you, Jason.Jason SorokoThank you, Tim.Tim CallanAnd this has been Root Causes.Stay informed with expert insights
Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!
