Root Causes 358: Security Questionnaire Sins

Hosted by

Tim Callan

Chief Compliance Officer

Jason Soroko

Fellow

Original broadcast date

January 30, 2024

In this episode we present a catalog of "security questionnaire sins," which are avoidable problems and errors that frequently occur in the security questionnaires enterprises send to vendors. Categories include difficulty of access, poor technical implementation, poor policies, and poor questions.

Podcast Transcript

Lightly edited for flow and brevity.

Tim CallanSo, our Episode 341, The Trouble with Security Questionnaires, which you remember, we did just a couple of months ago, it touched a nerve. I'm gonna say. I got a lot of responses from a lot of people. And they were kindred soul kind of responses. It was people who are dealing with the same thing and the capsule summary of the feedback I got was, yeah, yeah, exactly. That's right. Yeah. And it got me to thinking that that episode was kind of a, I would say it was more emotional than content based. It was like us kind of talking about how these things are badly run, and they're painful, etc. and it wasn't really kind of a systematic catalogue of what goes wrong with these things.

So over the last couple of months, I've been keeping a document and as I discover or run into one of these problems, since I look at a lot of security questionnaires, I've just captured them in this document. Gave it a little bit of organization and I thought maybe today what we could do is we can do something that's a little more organized and rigorous and essentially, present a catalogue of what I'm calling security questionnaire sins. Does that sound like a good idea?

Jason SorokoWow. Ok. Love it, Tim. Let's do it.

Tim CallanI always like an episode with a little bit of homework. Ok. So security questionnaire sins. So as I said, I see a lot of these things and there are a lot of things that aggravate me, that make it unnecessarily difficult. And here's what I'm gonna say about the sins.

These are very avoidable mistakes that either, one, make it unnecessarily difficult for the vendor to provide the information that the customers are looking for, or number two, render that information that's being provided less useful than it should be. And again, I think these are avoidable. I think these are absolutely things that could be gotten right, and they're just not gotten right and maybe at the end, we can spend a little bit of time, opining about the reasons why maybe they're not gotten right. But first of all, why don't we just get into the sins?

Jason SorokoLet's do it, Tim.

Tim CallanOk. So there are four categories. I'll give you the high categories, and then let's go into them individually.

The first category is making it insanely hard to sign on. That is actually amazingly its own category. It's such a prevalent problem. The second one is poor technical implementation. The third one is poor policies. And the fourth one is poor questions. Maybe I should have called the first one poor sign on, but whatever.

So section one, making it insanely hard to sign on. And these are not necessarily in any particular order. They're just in the order they're in. So item number - one time passcode for every login. And so I sometimes get these things. And literally - - and you can't just sit and fill one of these things out. They're giving you 100 questions. They're obscure and baroque. You go away, you do some research, you come back, you put some questions in, you go away, do some more research. Sometimes there's follow up questions that you don't see until you answer the original question. You can't sit and do this in one thing. So you're all you're forced to log into these questionnaires multiple times. This is for an online web portal version. You're forced to log in multiple times. And sometimes you have to log in every single dang time with a one-time password. So I'm dealing with this OTP thing every time I login. If I log in every day, right? I can't say, you know, ignore this for the next seven days. That's not an option. If I log in twice, if I use it in the morning, and then I go away, and I come back in the afternoon, I'm forced to do a new one time password.

And my point here is there's a disconnect between the level of security that is warranted, and what they're giving me. They're acting like they're Fort Knox. But what is this amazing secret that's behind this thing where they're so worried that an advanced persistent threat is going to come get in? It is me filling out a webform answering their questions. So perhaps is the level of difficulty in accessing this out of proportion to what is appropriate? And in so doing, they're making it much more difficult for me as a vendor.

Jason SorokoYeah, I think just the security architect just wasn't thinking very hard or doesn't exist.

Tim CallanI think there's a lot of this. I think when we go through this, a lot of this is going to be a lot of wasn't thinking very hard or doesn't exist.

The related one, and this has actually occurred to me, is actually making me install a specific app on my phone because that's the only way you're prepared to authenticate me is through this app. I can't use Microsoft authenticator, which shipped on my phone, which I use for all kinds of other things. I've got to use your app that you're requiring me to use. So now I've got your app, and Microsoft authenticator, and maybe another app from another vendor and literally, I would find myself going in to log into one of these questionnaires and not remembering which app I was supposed to use, and having to move from app to app to app collecting a one-time passcode and putting it in to see which one was going to let me go in. That's terrible. Right? In order to protect what? Your online form that I'm going to fill out with my information.

Jason SorokoYeah.

Tim CallanSo yeah.

Jason SorokoYeah. That's great. That's great. I think that quite often that's from security architecture where their own employees have a proprietary system, and they haven't figured out that for the rest of the public BYC, you know, I guess what it is, Tim, is you were to really fill this, you know, fill in the blank on this one, to me, it's a difference between a B2B and BYC and I think that filling in the security questionnaires should probably fall under BYC, which means not only should your usability be better, but also forcing people to use some sort of B2B type of authentication system is, it's unnecessarily onerous.

Tim CallanYeah. It’s definitely onerous. Right.

And then the next ones are connected to this. So the next two kind of go together.

The first one is absurdly complex password requirements. And you know, you and I hate passwords and we agree that password complexity is oftentimes the enemy of security, because it forces people to do things like, you know, put them on files on their computers and you see this here. People give you password requirements with all the dogs and ponies you'd expect. You know, upper, lower, special character, no repeated numbers, can’t reuse the same thing, change every 90 days. I’m like, again, really? Why? Why?

And then the worst one than that is absurdly complex password requirements that are unstated. So I've run into this. Where I go, and I put in a password, and it says, no good. And I don't know why. And so I add a special character and I try it again. It says no good. And I don't know why. Right? And, and like, if you are going to do that, at least tell me what the requirements are. And then when I go to log back in, have a little line under the password field that says, by the way, these are the things we demand of you, right. But no, they don't do that. And so you get in that whole problem and that difficulty. And I think that's connected to the OTP thing.

And I think that all kind of falls in that same group of what you said - Number one, it's a disconnection from the value of the secret that you're protecting, or the value of the material protecting and how much security you should put in place.

Second, is brain dead or non-existent security architecture. And then the third one that might be as just kind of lazy reuse of what you already did somewhere else. You put something together for your own employees that are doing highly sensitive stuff and you just kind of grab the same thing and slap it on this, giving it exactly zero thought to how it suits your needs in this different use case. So those four bullets go together in terms of making it hard to sign on.

Now there's one other bullet in terms of making it insanely hard to sign on, which is, all of these things come by way of email. You get an email, and that's how you get access. And these emails are often very poorly handled. They could be enigmatic, and cryptic. They come out of the clear blue with no warning and no explanation. Sometimes they look like a phishing email, right? I will literally get something that comes from some organization, and it says, you know, gotta do this. There's a deadline. It's in three days and urgent, urgent, exclamation marks and bad consequences if you don't. Click on this button right here and I go nah, I'm not an idiot. I'm not clicking on that. What is it, 1998? And, of course, it's not a phish. It's a badly designed email that looks like a phish and up to and including things that come from strange email addresses that aren't the vendors email address. They're running this through some third party so you get something and it totally looks like, you know, uh, a phisher’s spoofy email address, you know, instead of the company name.com, its company name-securityquestionnaire.com. And you're like, I don't believe that. And, and, like the worst example of this, I would say, is one where literally the from name, the friendly name was just the word happydesk as one word. And I didn't open it. I get weird emails from all sorts of people all the time. Somebody is trying to email me from someone called Happy Desk. I don't have time for that. And it wasn't until follow up came that I understood that this out of the blue, over the transom, unsolicited email was the thing I was supposed to pay attention to.

So once again, like badly done. All of this stuff is badly done. It's badly thought through and the same way there's no security architecture, there's no communications architecture. There's no communications design and planning around any of this.

Jason SorokoGeez. I tell ya. You know, folks, a lot of this has been solved. The way to communicate with people, there are ways to do it. I was thinking about some of the emails that you've received, you know, magic links are a thing. It's a way to get people logged in very quickly, then you can establish a better authenticator, and for the love of everything, folks, security architects, please listen carefully. Passkeys exist now. Especially for B2C scenarios, like Tim is talking about, let's start using them.

Tim CallanYeah, I agree. Passkeys would be really nice here. That's a perfect, perfect use case for a passkey.

Second one - poor technical implications. So I do - - these all take two forms. Either you get a web form, or you get an Excel doc. Once in a blue moon, you get a Word doc, but almost never. You get a web form, or you get an Excel doc.

And first one on the list - poor technical implications. Web forms that don't work. That are not coded correctly. That they won't submit, or they pop up errors that you can't rectify, or they pop up errors that make no sense. Or for one reason or another, they just plain don't work. And this is the thing. Like literally, you get web forms that don't work. Where the submit button won't un-gray. And you don't know why. And what do you do? Right?

Lack of clarity on how to submit no results is one of a big one. Like I just had to deal with one yesterday. It made no sense. I got in, I filled out the form. It was multiple pages. It was real complex. Lots of questions, checkmarks next to the ones that were all completely filled in, and you go through, and in the end, there's a checkmark full of everything. And the Save Draft button doesn't turn into a submit button. And I couldn't figure out how to submit and I could save the draft all day long. I sent something to the vendor. I said, hey, I saved the last version of the draft. They said no, you got to submit it. And I said, how do I do that? And they said, oh, we don't know. And it turned out it was weird and bizarre and you had to like go do something phenomenally counterintuitive, in order to submit it. And why? Why? Right? People have been building web interfaces for decades. Why don't you just do the stuff that makes sense and works?

Spreadsheets. Locked spreadsheets with implementation errors. This is another one, right? They want you to fill out a field, but you can't. The field is locked, and you can't put things in it. I've seen this. You know, there's an open text field and they're asking for an explanation, but we can't fill out the text. It's locked. Or locked, drop down lists that are wrong, where the results in the drop down list don't match the question that's being asked. Right? So they ask you, you know, one to 10 question and the drop down list is yes, no, right? Where just there's no translating it, you simply can't answer it correctly. Or, I've seen dropdown lists that are for select all questions. And these can be either on Excel documents or in web forms now, right. So it'll say select all that apply, and you got to drop and so there's 12 options. Which of these security tools do you use and your, oh well, 11 of the 12 but I can only select one. So you select the first one and move on and you know whoever is getting these results is getting garbage results.

I've seen other options where once you select other they want you to then describe/explain what it is, but they don't enable a text field. So it's like other/explain and there's no explain. There's no place to explain And then you go, ok, well, I guess you're not gonna get any of your explanations are you? And all of these things are there. Like this happens all the time.

Nonsensical instructions is another one. Just badly written instructions. They just don't make sense. You just don't know what they're asking you for. So I see a bunch of this, too. And again, I think it goes back to the same things which is no thought, no attempted quality, kind of a lazy approach and no thinking through the alternate consequences. And this one really gets me because in an earlier life, one of the things I used to do is I worked on a lot of survey instruments. And when you're paying $75.00 to somebody to fill out your survey, you make damn sure that every question is clear and obvious and you're getting answers to the questions you think you're getting answers to. And these guys don’t do this.

Jason SorokoHey, I've seen it. Tim, I just wanted to add a little bit to what you just said. I've also seen examples of, if you're going to send me a spreadsheet, which is bad enough, quite often, there are macros that you're forcing me to enable. My answer to that is no. Never. Forget about it.

Tim CallanRight.

Jason SorokoAnd I've also, you know, in a previous life of mine, I used to love to do hacking demos, and one of my favorite ways of doing it was through PDF forms, because of the inherent weakness of a lot of PDF implementations, and so therefore, whenever I receive a PDF form to fill out, it's literally like, wow, I might as well just let my computer - - you know, I might as well just hand it over to Hacking Incorporated. You know, it's just, it's bad. So, quite often for that purpose, I will fill out PDFs on an iPad, and not on a PC. Period.

Tim CallanRight. And this makes you like, like, I don't think this is the case. But then my mind starts going to the fact that, ah-ha, well, maybe the real point here is I'm supposed to refuse to enable the macros, and that's how I win. Right? And sending them an email back saying, no, I won't fill out your form. But you know, of course, that's not what they're doing. They're just, again, doing a bad job, doing a lazy job.

So number three - poor policies. So the first one is extremely short turnarounds where like you send me your 150 page questionnaire, and you tell me it's due in 10 days. Really? Like, what if I'm traveling? What if I'm on vacation? What if I've got a really bad couple of weeks? Did you really need to give me only 10 days? You honestly couldn't have sent this to me a month ago? Are you just not on top of your own business? Is that what's going on? Or sometimes ridiculously unrealistic due dates. I've gotten due dates that are like the next day. I've received emails where the stated due date was in the past. One email where the stated due date was in the past. And you're like, ok, well, I know I'm not doing that because I don't own a DeLorean. So once you come to that conclusion, what's going to happen? I'm going to miss the due date. Now that I'm missing the due date anyway, I just have to speculate on what a real due date is. Three weeks from now? Ok. Let’s take that guess. Right? So by giving me a due date that's the day after tomorrow or giving me a due date that's in the past, what you're telling me is, you're not going to tell me the due date. That's really what you're telling me? And then I make up my own due date? And if it's not the due date you want, now we have a disconnect. Why don't you tell me the real due date from the beginning?

Asking for non-applicable data. Asking for things that don't apply to me. Um, you know, I've seen some where they ask you a question like, are you, you know, A or B, and you say, A, and they turn around and they start asking you questions about B, right? And you're like, no, I just said, I wasn't that. I don't have an answer for that question. I'm not that.

Failure to accept well understood audit vehicles in place of these forms. If I've got a SOC audit that has the answer to every single thing that's in your form, why can't I just give you my SOC audit, right? Or, in our case, a WebTrust audit. Now, I know, most vendors don't have WebTrust audits. But WebTrust is actually more thorough than SOC. And we do have WebTrust audits, and they're a matter of public information. So how come I can't give you my WebTrust audit? Right? My WebTrust audit is like 100 pages long. Can I just give you that? And nope. Nope. Why? Why? What's the purpose of that policy?

And then the last one, this one drives me crazy, is this doesn't really occur with existing customers, but it sometimes occurs with pipeline opportunities, which is an unwillingness to sign an NDA before you turn around and you ask me to fill out your technical questionnaire. I'm like, ok, let me get this straight. You're going to ask me to tell you all kinds of things that I wouldn't tell the public about my environment, my practices, my policies, my protections, etc. Stuff I absolutely wouldn't go publish in the newspaper and you won't sign an NDA. Really? And so again, this is a problem and I push back. I'll go back to the sales team and say, no, until you have an NDA, I won’t to tell them. Because you gotta. How could we not?

Jason SorokoYou know, that really, really speaks to me about inexperience.

Tim CallanYeah.

Jason SorokoThese might be interns or something who might not realize how business works.

Tim CallanYeah. Or it's somebody who is down in the bowels of the IT department, who has never met anybody in legal department, who doesn't do this stuff and they're not finding out. I think a lot of this when we talk about kind of lack of security architecture, lack of fundamental best practices, and how to send emails, this feels like a sysadmin with a certain skillset that has been assigned a task and isn't connecting to the organizational knowledge that knows how to do this task, right?

Jason SorokoYeah. I think you're getting really close to it, Tim. It sounds like people in a business who know that they need a survey will assign somebody to it, but that person is not qualified to do it because doing it properly does require something that isn't the lowest level person.

Tim CallanAnd it requires skills that aren't the skills that you learn when you get your IT degree. Right? It requires knowing how to connect with legal and again, knowing some basics about how business is done, knowing some basics about how communication works and that isn't necessarily what you learned when you went to school or on your first five years on the job and you don't know it, and you don't find people or you're not assigned a group or you're not given support by the organization, by the people who do understand this stuff, right? These guys are sending out emails to a bunch of their vendors and there's nobody from marketing who knows how emailing is done, who is helping them. And so you get bad emails, and it breaks down the process. So all that's great. So great. So we can excuse them for all of those things.

The last one I have a little trouble with though excusing them for which is poor questions. And I get a lot of poor questions on these things. It is a common, common occurrence. And so here's some examples.

First one - asking things the vendor should not answer. I cannot tell you the number of people who have asked for my full penetration testing reports, or my full vulnerability scanning reports. Well, I don't give those out. Nope. Can't have it. Right? Or things like that. Then questions that I shouldn't answer. And connected to that is questions that I shouldn't answer, or questions that are absolutely irrelevant to you that you have no need to know. Why are you asking me to ask you a question that you have absolutely no need to know? And my best example of this, and I think I said this the last time we did this after Episode 241 was the vendor who wanted to know if our backup was on tape or disk?

Jason SorokoYes, I remember that.

Tim CallanI'm like, you don't need to know. It doesn't matter to you. You will not make a decision differently based on that answer. Why are you asking me?

Jason SorokoYeah. Copy and paste from 1990s questionnaires for sure.

Tim CallanRight? You don't need to know. Another one that drives me crazy, is overly vague, freeform text, what I'll call essay questions. People will literally put things like describe your incident response policy. And I'm like, uh, describe my incident response policy? I could write a 40 page document on that. I don't think you want me to paste a 40 page document into this textbox in this web form.

So now that I think that's not what you want - also, I don't have time to do that - what do you really want? What does an acceptable answer look like? And now I'm guessing. Right? If the form was more clear on what they were looking for, guessing wouldn't be required. Right? So that's another one that occurs. Or hyper detail. The opposite of that. A level of scope and detail that is not useful.

So for instance, here's one that I got out of a forum just the other day. Are audit trails sent to a centralized server? And maybe this is the same as asking me things you don’t need to know. I'm like, why do you care? Why do you care about if audit trails are sent to a centralized server? It is not important or relevant to you. Right? And, and I don't even know. Like, I don't know. Am I really gonna go track this answer down and find out? And why? Why? What is going to be different? How is this going to help anybody?

Jason SorokoI almost feel like some of these are candidates for AI. It's like, hey, just boil up an answer for me here.

Tim CallanYeah.

Jason SorokoBecause it almost doesn't matter what the answer is.

Tim CallanRight. Right. Are audit trails sent to a centralized server? I could say yes. I could say no. I don't see how my answering either of those answers makes anything different. So why are you asking.

Questions where I'm deeply skeptical that the person who is going to be dealing with these results has the technical acumen to evaluate the response.

So again, a good recent example, what DDoS Protection Service are you using? I'm like, are you a DDoS expert? Are you really a DDoS expert? Are you going to look at the answer that I gave you and come to some kind of judgment about whether I've chosen the right service or the wrong service for my needs? Because there are DDoS experts in the world. I used to work at VeriSign. I know a bunch of them. Most people with sysadmin jobs out there in the world are not DDoS experts, and they're not going to be able to do anything with that answer because they don't have the knowledge they need to answer that.

Jason SorokoRight. Every system is unique. Every system is totally unique. The answer can be totally different. How could you evaluate possibly.

Tim CallanYeah. And therefore, why are you asking? Right?

Here's a big one. Questions where the question, the literal question that is being asked is clearly not the intended question. I'll give you an example. Do you depend on third parties to deliver this service? Well, everybody should always answer yes. I get electricity from the power grid. I have an Internet service provider that gives me bandwidth. I buy software. I buy hardware. Right? There is no way - - I’m probably in a colo. There is no way I should - - Anyone who says no, like unless you're sitting in a room with a soldering iron putting servers together, you need to say no. Right?. But of course, that's not the question they're asking. I get that. But then they should ask the question they're really asking, because now what I have to do is I have to look at that question. I have to say, well, I know that's not the question you really mean to ask me. So what question do I think you need to ask me? And then I answer that question. And maybe I'm not answering the question that you meant to ask and either way, it's just making it much more difficult for the vendor to do their job to give you the information that you want. Again, it's back to the same thing, lazy, thoughtless, not connecting to people who can help them make the stuff better.

Ambiguous questions with no disambiguation. So my favorite one around this is about like kind of access policies. So I get a lot of questions about what are your password policies? What are your minimum password strength policies? Or what are your MFA policies? And I say, well, which password are we talking about? Are we talking about your password for your users to log into my SaaS service? Are we talking about your regular users or your admins? Are we talking about my people and their passwords to log into things and which things are we talking about? Just general access to all my business systems? Or what the trusted employees need to get into the systems that are running your data and your service? Because they're all different answers. And I can answer every one of them. I know the answer to every one of them. But I don't know which one you're asking me.

Jason SorokoYep. What's the scope?

Tim CallanWhat's the scope here? Ambiguous questions that could use disambiguation.

And then this, I've run into this one more times than you imagine. Forced document upload where no document exists or is appropriate.

So here's the scenario. They say, do you have the following? Do you have a - - I'm going to make up something we don't have. Do you have, you know, a HIPAA? Do you do need HIPAA requirements? And I click no. And the next question, which is a required question is, upload your latest HIPAA audit. And I'm like, no. I just said we don't meet HIPAA requirements. I don't have a HIPAA audit. I said no. And yet, it's a required thing. I literally have created a Word doc that I have in my hard disk that says, we don't have this. And I upload it to get past those forced uploads and move on. And again, why? Why? It's the same thing. It's just badly done. Right? These are all avoidable. These are all things.

So that's, that's my list. And that's my rant. And, um, you know, I think we've identified a lot of the reasons why. Right? The reasons why is there - - it's the things we said. It's the people who are doing this view this as a toss off. They don't think they have to do it well.

Jason SorokoYeah.

Tim CallanThey just want to get it done with the minimum possible effort. They don't bring in other people. They don't sit and think. They don't test. They don't get feedback. They don't show it to somebody and say does this make sense? They just knock it out and move forward with no consideration of the idea that there's a quality difference between doing it well and doing it poorly.

Jason SorokoYeah. It looks like nobody has even tested the survey.

Tim CallanYeah.

Jason SorokoIt’s just a read through and catching some of these logic errors that you're talking about. And for - - you’re so right in what you said about if you're going to bother to ask people questions, be real careful what questions you ask and really truly think it through. Every moment of thinking is going to pay off.

Tim CallanYeah. And I think maybe the other thing that's connected to this is a sense of, well, it's not my prompt, right? If I have a survey instrument that's difficult to use, or ambiguous, or it's hard for someone to log in or onerously difficult, it's not my problem, right? They have to figure it out. It's their problem. Let them figure it out.

And so, you know, first of all, is that how you treat people? Right? But even if you do think that way, you know, this is a two way street. This is information you want. And IT departments have to buy security tools and that means that somewhere along the line, they got to deal with the vendor, and you know, they need this - - if they actually need this information, then they need this information. And if the vendors have a difficult time giving them that information that actually makes the security professionals’ job more difficult. So, you're also making your own job more difficult.

Jason SorokoWith that exact thing being said, Tim, here's what I'm thinking right now. I think we need a third podcast on this subject.

Tim CallanOk.

Jason SorokoWhere - - take, you know, different set of categories and then say, hey, here's how to get to the meat of what we know, we think, you're asking for. Effectively. And with your experience as a survey person, you lay it out in terms of how it should be done, and some of the logic errors that are easy to make, and therefore double check for these things. And I think if you're in the business of putting these kinds of things out, I think there's some real basic rules that you could that you could follow. And I think the real meat to it isn't just how do you build a survey. It's what is truly important as a procurement specialist when you are evaluating a vendor. Let's really get down to, from a security standpoint, what really is important. And here is how to ask that question. And I think that that's something we actually are qualified to help with.

Tim CallanI love it. I think that's a great suggestion. Jason. So we will return to this topic yet a third time. And I think getting clarity on that, less about the talk about what's wrong and more about the spell out what something that's right looks like. Great idea. I love it.

Jason SorokoGood.

Tim CallanAll right. So probably enough for today. Thank you once again for tolerating my ranting on this particular subject, which obviously I have a lot of emotion about. And thanks, Listeners. This has been Root Causes.

Stay informed with expert insights

Subscribe to Root Causes for engaging discussions on PKI, digital security, and best practices for protecting your organization's critical assets. Don’t miss an episode!