Missing a vulnerability
Friday 13th May 16
Calling testers, put your hand up if have you ever missed a finding on a test. If you've still got your hand down, are you sure? If you are an app tester, did you test a site using ImageMagick before the 4th May 2016, or a network tester, did you test a network using OpenSSL before April 2015. If you did either of these and didn't write up ImageTragick or Heartbleed then you missed two very serious issues but you missed them along with the rest of the security community. I'd consider these acceptable misses.
What about if you missed SQLi in the search box on the homepage of a site or if you missed an open root share from a domain controller? Both also serious problems but I'd consider missing these a very big mistake.
So where does the line change from something being acceptable to miss to it being unacceptable? There are going to be lots of things that come into play here, length of test window, size of network, complexity of the application and all sorts of other things. We also have to accept that we are all human and have bad days.
I guess the answer is similar to what they use in various other disciplines, would another tester of similar skills to what the tester is supposed to have, have spotted it. I say "supposed to have" as there are testers out there that pass themselves off as having a much higher skill level than they really have, either deliberately to deceive or out of an honest belief they are better than they think they are.
There is also the problem of specialities, some people are better in some areas than others. Someone who has done lots of testing on PHP is more likely to pick up a vulnerability in a LAMP app than someone who has spent most of their time looking at .Net apps. Something obvious to one may be completely new to the other.
Whenever I hear of someone I know testing an app that I tested, especially if there have been no changes between the tests, I get nervous. What if I missed something? What if I missed something that was really obvious? What if they investigate it and find that they were exploited through it? Something like this could be the end of a career or, at least, a very big step backward. I've missed the odd little thing here and there but nothing big yet and I hope that I never do but the worry is always there.
I spoke to one friend who said that he has a very good level of professional indemnity insurance and if he ever did make a big mistake that he would tell the client to make a full claim then walk off and find another career. That seems a little extreme but I understand his feelings.
Something that does happen occasionally is when one tester finds an issue that the other doesn't feel is an issue so saw it but didn't feel like it needed including in a report. I've had this a few times where a second tester reports on something that I wouldn't consider enough of a risk to put in the report. I've never had this be a problem but sometimes have had to explain why it wasn't in the original report and talk it through with both the new tester and client.
I guess what I'm getting at is that everyone misses things, some deliberately, some by honest accident, some by incompetence. Some you can get away with, some could really spoil a good career. I don't like being worried but think that ultimately it keeps me on my toes and helps to make sure that I do my best on every job.
So, does missing something worry you? What do you consider is acceptable to miss? Have you missed something big and had to face the consequences? This blog doesn't have a comment system but please email me and I'll collate replies into a follow up article, it would be good to get other perspectives on the issue.
To finish off, I think it is worth remembering that technologies change and new attacks are developed all the time. As I said at the start, some issues are acceptable to miss as the whole community is missing them and, as long as you work to the best of your ability, be honest about your skill levels and keep good records of what you did, then things will hopefully work out if something does go wrong.
