Podcast
Root Causes 341: The Trouble with Security Questionnaires
Hosted by
Jason Soroko
Fellow
Original broadcast date
November 13, 2023
The practice of sending security questionnaires to technology vendors is exploding, and with it dysfunctional behavior is on the rise. In this episode we describe how security questionnaires are changing and the pitfalls associated with this emerging practice.
Podcast transcript
Lightly edited for flow and brevity.
TCTim CallanAlright. So Jason, today, I would like to talk about security questionnaires.
Jason SorokoSecurity questionnaires. Right on. Going out to vendors and things like that, I guess?TCTim CallanExactly right. So a company before it buys an enterprise product will ask, will fill out what, in general people call a security questionnaire. The footprint has expanded beyond security. That's where it started. But sometimes these vendors will send out these security questionnaires and ask you to fill them out. These are not a new thing, and I can't put any hard numbers behind it but as somebody who sees a lot of these things, I feel that there are some trends or one kind of macro trend that's going on in the security questionnaire space or effort, practice, that I just really want to talk about.
Jason SorokoLet's do that. It's an interesting topic. It's I think it belongs in this podcast, because it's in the general term of security, but also, some interesting things will come up here. So that's good, Tim.TCTim CallanSo I see these, because I'm in charge of compliance, and there's always compliance questions. So they flow through me, among other things, as these things come in. I feel like fairly recently, I'm gonna say in the last six months or so, that this whole practice has transformed in several directly related ways.
The first of which is the frequency at which they come in definitely has increased dramatically. We are seeing multiples of the number that we were seeing a year ago at this time. So the number of people who are deciding to use these questionnaires with their vendors has really skyrocketed. That's item number one.
But also, in addition to that the complexity and length of these questionnaires also has skyrocketed. So where once upon a time, like a year ago, you'd get a questionnaire and it might be 15 questions and if it was 30 questions, you'd say, whoa, big one, right. I just finished one yesterday - 200 questions long.
The first of which is the frequency at which they come in definitely has increased dramatically. We are seeing multiples of the number that we were seeing a year ago at this time. So the number of people who are deciding to use these questionnaires with their vendors has really skyrocketed. That's item number one.
But also, in addition to that the complexity and length of these questionnaires also has skyrocketed. So where once upon a time, like a year ago, you'd get a questionnaire and it might be 15 questions and if it was 30 questions, you'd say, whoa, big one, right. I just finished one yesterday - 200 questions long.
Jason Soroko(chuckle)TCTim Callan200 questions long. To make a security questionnaire that is 200 questions long, you have to ask some pretty esoteric stuff. So I'm literally getting questions in questionnaires these days and it's like, your backups of your business productivity systems are they stored on tape or disc? That's a real question that I got on a real questionnaire. And, one of the things I go is I say, is this a question you actually need to ask? Like, is there a meaningful difference in that answer, that in some way is going to aid the vendor selection process?
Then the third thing that's connected - not just the number of questions, but directly related to it, that the nature of the questions, in a lot of ways is getting really bizarre. I'm being asked questions that I just scratch my head, and I can't fathom how this is meaningful information. I can't fathom how it really matters. And so these three things going on together, I think are part of a bigger trend, right? Which is, for some reason buyers are getting it in their heads that they have to have this incredible set of information for every vendor.
Then the third thing that's connected - not just the number of questions, but directly related to it, that the nature of the questions, in a lot of ways is getting really bizarre. I'm being asked questions that I just scratch my head, and I can't fathom how this is meaningful information. I can't fathom how it really matters. And so these three things going on together, I think are part of a bigger trend, right? Which is, for some reason buyers are getting it in their heads that they have to have this incredible set of information for every vendor.
Jason SorokoIt could quite possibly be. You know, Tim, what interests me about this, and this now, now that you brought this up, this convinces me more that this is a good topic for this podcast, because how many times how many times have you and I brought up you and I giving a really solid advice for procurement people for security offerings. We've talked about that quite a few times. And the question you ask right there, like the question example you just gave, is an example of a low to zero value answer. Any answer you're gonna get is going to be that's, I mean, go ahead and argue me. Please, please. Go ahead. I have no problem taking on that because really it has a lot more to do with how you operationalize your backups. It's not so much the whatever technology you’ve got things backed up on. And in fact, you might be using both of those things for different purposes. There's the real answer, right?TCTim CallanWell, and I’ll return to that problem. But keep going with your point. But remind me, let's return to that point because it's a valid point. But go on.
Jason SorokoHere's the thing. You've got a situation where good procurement questions are absolutely an important part of what you should be doing as a customer of these technologies. We've said that many times. And in fact, we've given - - go back on many podcasts - - I wish I could give you the numbers.TCTim CallanWe’ve got examples.
Jason SorokoWe got really solid examples. But these things should - - they always should be extremely well thought out in how you ask the questions and why ask the questions.TCTim CallanRight. So one of the things I think, as I'm going through this is, I think, come on, there's no conceivable way that the answer to this question is going to make a difference in the vendor selection process, like realistic. Then the second thing I say is I'd say well, ok, then these people who come up with these excessive or these strange questionnaires or these unnecessary questionnaires, they're hurting themselves, because in these 200 questions, how many of these do you really need? How many of these are actionable? 10, 15, 20. So how much time are you spending wading through 180 meaningless questions, instead of focusing on the 20 questions that matter?
Jason SorokoThis is this is sloppy procurement. Come on, guys. You can do better. You can do better. You know, vendors of security software, we have been used to being asked every question under the sun for years. We prefer to be asked the real ones. So just, come on, let's stop wasting everybody’s time here.TCTim CallanRight. I mean, that's an interesting thing. So I see these a lot, right. I have a lot of opportunity to think about them. And I've started to come up with some, some ideas, some hypotheses about what's going on here. One of them is, I think that to some degree, people view it as free. Like, it's free to ask more questions. It's fast to write a question, right? You just say whatever is, explain in detail your disaster recovery plans, right? Explain in detail your disaster recovery plans. Seven words. Right? It’s easy to write seven words, and then you sort of throw the work back on to the potential vendor. Right? So all the vendors have to go away and have to write an essay about this. You kinda go, I don't care. It's free. Doesn't matter to me, right? I don't care. But that isn't true because then these inputs come in and are you going to take these inputs? Are you going to internalize them? Are you going to make sense of them? Are you going to somehow use them to make a best possible choice? And again, some of them are legit. You may feel that this is a mission critical system, and I need to know what your disaster recovery is. Ok. Great. That's one that you need. Right? But you don't need all these things.
In the past where I saw these questionnaires were very focused on kind of how are you going to ensure that the SaaS service that I'm purchasing from you continues to be up and running? We have watched this spread out and expand. Right? The example I gave you earlier is about other things - business systems. You really want to know? Is it really your business? Is it really important to you as a procurement agent to know that my CRM has business continuity built in and is going to remain up and running? Why? Why is it that you're asking questions about my CRM at all?
So that's another thing that we have and connected to what you said is, I feel like a whole lot of the stuff is very thoughtless. Like it's sort of a knee jerk, ill thought out, just sort of make a big list of things and ask about all of them without anybody saying, is this information actionable? Does this information make my purchase decision better and more effective, right? Instead, the attitude seems to be well, it's perfectly free. It's really easy to write a question about what's your business continuity plan for your CRM, right? Where is your CRM stored? And then, they have to go do the work to answer that question. I don't necessarily use it or care. It seems to me what you said. It seems to be ill considered. I feel like that a lot of these questions are just sort of knee jerk and I don't want to quite use the word random, but just disconnected to the real criteria that matter.
In the past where I saw these questionnaires were very focused on kind of how are you going to ensure that the SaaS service that I'm purchasing from you continues to be up and running? We have watched this spread out and expand. Right? The example I gave you earlier is about other things - business systems. You really want to know? Is it really your business? Is it really important to you as a procurement agent to know that my CRM has business continuity built in and is going to remain up and running? Why? Why is it that you're asking questions about my CRM at all?
So that's another thing that we have and connected to what you said is, I feel like a whole lot of the stuff is very thoughtless. Like it's sort of a knee jerk, ill thought out, just sort of make a big list of things and ask about all of them without anybody saying, is this information actionable? Does this information make my purchase decision better and more effective, right? Instead, the attitude seems to be well, it's perfectly free. It's really easy to write a question about what's your business continuity plan for your CRM, right? Where is your CRM stored? And then, they have to go do the work to answer that question. I don't necessarily use it or care. It seems to me what you said. It seems to be ill considered. I feel like that a lot of these questions are just sort of knee jerk and I don't want to quite use the word random, but just disconnected to the real criteria that matter.
Jason SorokoThey are disconnected from the criteria that matter. Absolutely. There's no question. But you know, really good procurement people - of which there are plenty out there - I've worked with them. A different breed of people. You know who you are. But you're good, good people. The example that Tim is giving and hopefully it's not, that's not common, and it's not everybody, but it's, you know, you're calling out a trend, Tim, and it's unfortunate to see. We’re just calling out here. For those of you who are in the business of procuring security software, or security offerings of any kind, your procurement process is very important. And the questions you ask your vendor really should be well thought out. It looks like these kinds of things got copied and pasted from a late 1990s manual, right?TCTim CallanWell, I think that's a big thing is it all feels very much like a copy paste job.
The other thing that I feel, seeing a lot of these things, is that there's no discrimination based on the nature or the form of what's being asked. Nobody is distinguishing. I'll give you an example. One time, I got a security questionnaire. I got passed over by a sales guy and I asked the question, what product. Because you got to understand what is someone trying to buy from us and it turned out that what they were proposing to buy was a single wildcard certificate. And I said, no, I'm not filling out the questionnaire. They can buy it or they can’t. And I kicked it back, and they bought.
But, if somebody was thinking about it for even a second, they would realize that a certificate is a different beast, and asking questions about my internal failover plans is irrelevant because they have the certificate. It's theirs. It's running in their environment. All this other stuff isn't part of the consideration set. And that was somebody who clearly just wasn't thinking at all. At all. And so you'd see that too. A lot of this is kind of this paint by numbers approach. Someone gets told, they go to a conference, and they sit in a seminar at RSA or wherever and they go to one of these speeches, and someone says you should do security questionnaires. And here's why. Then the person who gave that speech probably has a well-considered viewpoint that comes from experience and knowledge and study and stuff but then the people who sit there kind of come home with a real basic idea, like we should do security questionnaires. There's no more thought about the how and the why. It's just this kind of really over simple paint by numbers approach, as opposed to stopping and understanding, what am I doing? How am I doing it? Why am I doing it? And what do I think I'm going to get out of it? And that stuff seems to get lost, right? When somebody is asking questions about business continuity plans, and what they're actually going to have as a cert that shows a disconnection between understanding what you're asking, and how it's going to affect you.
The other thing that I feel, seeing a lot of these things, is that there's no discrimination based on the nature or the form of what's being asked. Nobody is distinguishing. I'll give you an example. One time, I got a security questionnaire. I got passed over by a sales guy and I asked the question, what product. Because you got to understand what is someone trying to buy from us and it turned out that what they were proposing to buy was a single wildcard certificate. And I said, no, I'm not filling out the questionnaire. They can buy it or they can’t. And I kicked it back, and they bought.
But, if somebody was thinking about it for even a second, they would realize that a certificate is a different beast, and asking questions about my internal failover plans is irrelevant because they have the certificate. It's theirs. It's running in their environment. All this other stuff isn't part of the consideration set. And that was somebody who clearly just wasn't thinking at all. At all. And so you'd see that too. A lot of this is kind of this paint by numbers approach. Someone gets told, they go to a conference, and they sit in a seminar at RSA or wherever and they go to one of these speeches, and someone says you should do security questionnaires. And here's why. Then the person who gave that speech probably has a well-considered viewpoint that comes from experience and knowledge and study and stuff but then the people who sit there kind of come home with a real basic idea, like we should do security questionnaires. There's no more thought about the how and the why. It's just this kind of really over simple paint by numbers approach, as opposed to stopping and understanding, what am I doing? How am I doing it? Why am I doing it? And what do I think I'm going to get out of it? And that stuff seems to get lost, right? When somebody is asking questions about business continuity plans, and what they're actually going to have as a cert that shows a disconnection between understanding what you're asking, and how it's going to affect you.
Jason SorokoLet's talk about the real purpose of these kinds of things. Obviously, if you just look on the surface, Tim, it's, hey, let's try to get the best offering possible, right?TCTim CallanRight.
Jason SorokoThere's a lot to be said there. The question you just asked is really, those are in the spirit of, you know, can the vendor that we're dealing with, sustain themselves? Like real basic business sustainment and business risk. There are some fair questions about that, frankly. Instead of just operational risk, I'd ask a lot more about financial risk, especially if the deal is a bigger one. Those are some much more meaningful questions. But the spirit of alright, let's go, let's say now that there's a problem with the vendor, and you need to cover your butt somehow. How did you come down to the decision of making a decision to buy this or invest in it? And oh, well the vendor said this, they must have lied to us - which could have legal, perhaps some kind of a legal implication or the opposite, which is you know, we just didn't procure this very well. Well, I want to make the argument is, unless you've really thought out what you're really trying to get out of this, you know, your outcomes, what are the true outcomes you really want? And instead of just late 1990s wrote copy and paste procurement ideas. I think that this doesn't help to protect your butt whether you're making the procurement or even whether you have to answer the question to some auditor inside. Hey, how did you make a decision about how to acquire this security technology investment? Just having a tick box type of operation in your procurement doesn't satisfy anything. There's really no purpose to have a meaningless set of procurement questions that really don't get to the heart of your desired outcomes.TCTim CallanAnd I think, also, the problem there is you lose focus on what are the questions that are meaningful? Let's play a little hypothetical. Let's suppose that there was some kind of rule that you could only ask a dozen questions. Well, you would make sure that they made every one of those dozen questions count. Those dozen questions would be meaningful questions that really helped me understand. But because there's this kind of expansiveness strategy of ask every single thing you could ever conceive of, then you see a lack of someone doing what you’ve said, of really considering, what am I going to find out? And how am I going to use that information? And that goes away.
Jason SorokoAs soon as you see 200 questions, that it wasn't well considered. As soon as you see a question about do you back up to disk or tape that it wasn't well considered. And it's a huge lost opportunity because, boy, sometimes procurement questions you can learn a lot about even how some of your own misconceptions about security software, let the vendor answer a good question well and that has an enormous amount of value. You can learn something from it.TCTim CallanAnd then connected to that I think this, what does this matter to me is what I already alluded to, which is this sense of asking questions that are kind of getting out of the realm of what you normally ask. So for instance, we now get some of these things and they're asking questions that we would never answer in a million years. Like, questions about our financials. We're a private company. We don't answer those questions. We just don’t. Or, things that by policy, we don't share. And they ask. And maybe the rationale is, well, if you ask, maybe they will share. But again, I come back to, if you're asking people questions that they shouldn't tell you the answer to, what does success look like? Like you'd think a good vendor or actually the vendor you would want would be the vendor who says, no, I'm not going to tell you that because it's a bad practice to tell people that, so I won't. And yet, I have to suspect that the person who asked that question, didn't ask it as a test to see if I knew better than to answer it. I have to suspect that the person who asked that question was expecting to get an answer. And so thereto, you’re like, why? Why am I being asked questions that are company confidential, that don't have to do with my ability to service your business? And what do you think you're gonna conceivably get out of that and rather than that, aren't you just kind of burning a lot of time and effort on both sides for the vendor and the purchaser without really coming to any productive results?
Jason SorokoYou got it, Tim. It's a troubling trend. Thanks for bringing it up. But, look, maybe, maybe, Tim, we should revisit cybersecurity procurement tips and tricks. And let's help people to come up with maybe not the full 12 questions, but, a good two or three or four that are just killer in terms of the space of PKI, certificate lifecycle management, etc. Maybe we can do that at some point.TCTim CallanI love that idea. That's a great idea. Let’s muse on that. Hopefully, that'll be a future podcast. I just think this is a trend. I've seen enough of these things that I feel like this is something that's going on, not just one or two isolated incidents, and I just wanted to call it to everybody's attention.
Stay informed with expert insights
Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!
