Debunking vulnerability assessment myths: Part 1

Experts discuss commonly held misconceptions within the industry


Editor’s note: This is part one in a two-part series on clearing up misunderstandings people have regarding vulnerability assessments. Part two, which will examine who should conduct vulnerability assessments, will be published next week.   

The man on the other end of the phone was nothing if not enthusiastic.  “So,” he said, “you folks do vulnerability assessments?”  “Yes,” was the response.  “That’s great,” he said, “we have an outstanding new security product we would like you to test and certify.”  “Well, I’m afraid we don’t do that,” he was told.  “I guess I’m confused,” said the man.

As vulnerability assessors, we encounter this kind of confusion dozens of times a year.  The problem isn’t solely that security managers are confusing testing/certification (and other things) with vulnerability assessments, it’s that by not understanding vulnerability assessments in the first place, they are probably not having them done, or at least not done well.  This isn’t conducive to good security.

The purpose of a vulnerability assessment is to improve security.  This is done in two ways: 1) by finding and perhaps demonstrating vulnerabilities (weaknesses) in a security device, system, or program that could be exploited by an adversary for nefarious purposes and possibly suggesting countermeasures and 2) by providing one of the 10 or so major inputs to an overall modern risk management approach to security.

Myth:  A vulnerability assessment is a test you pass.

You no more pass a vulnerability assessment (VA) than you “pass” marriage counseling.  Passing a VA can certainly not mean there are no vulnerabilities, or even that all vulnerabilities have been mitigated.  Any given security device, system, or program has a very large number of vulnerabilities, most of which you will never know about.  Every time we look at a new security device, system, or program a second or third time, we find new vulnerabilities that we missed the first time, and vulnerabilities that others missed, and vice versa. A VA is never going to find all the vulnerabilities, but hopefully assessors can—by thinking like the bad guys—find the most obvious, the most serious, and the most likely to be exploited.

What people sometimes mean when they say that they passed a vulnerability assessment is that they took the results of the VA as one of the inputs, and then made a subjective, context-dependent value judgment about whether their security is adequate for the specific security application of interest.  While it may be completely necessary and reasonable to make such a judgment, that decision belongs in the domain of the security manager, not the vulnerability assessor.

Myth:  The purpose of a VA is to accomplish one or more of these things:  Test performance;  do quality control;  justify the status quo;  apply a mindless stamp of approval;  engender warm and happy feelings;  praise or accuse somebody;  check against some standard;  generate metrics;  help out the guys in the marketing department;  impress auditors or higher ups;  claim there are no vulnerabilities;  endorse a product or security program;  rationalize the expenditures on research and development;  certify a security product as “good” or “ready for use”;  or characterize the ergonomics, ease of use, field readiness, or environmental durability. 

Certainly, some of these issues are very important and may have a bearing on security vulnerabilities, but they are not the focus or the purpose of a VA.

Myth:  A vulnerability assessment is the same thing as a threat assessment.

Threats are who might attack, why, when, how, and with what resources.  A threat assessment (TA) is an attempt to identify threats.  Vulnerabilities are what these threats might exploit. 

Myth:  A TA is more important than a VA. 

This content continues onto the next page...