Debunking vulnerability assessment myths: Part 1

Experts discuss commonly held misconceptions within the industry


Effective VAs and TAs are both essential for good security and for modern risk management.  A TA, however, entails speculations about groups and people who may or may not exist, their goals, motivations, and resources.  TAs are often reactive in nature, i.e., focused on past incidents and existing intelligence data.  Vulnerabilities, on the other hand, are right in front of you (if you will open your eyes and mind), and can often be demonstrated.  VAs are thus typically more proactive in nature. 

If anything, an effective VA may be more important than a TA.  If you get the threats exactly right, but have no clue as to your vulnerabilities, you are probably at significant risk.  If, on the other hand, you get the threats at least partially wrong (which is likely), but you have a good understanding of your vulnerabilities and have mitigated those you can, you may well have good security independent of the threats.

Myth:  These techniques are effective for finding vulnerabilities:  Security survey (walking around with a checklist), security audit (are the security rules being followed?), feature analysis, TA, design basis threat (DBT), fault or event tree analysis (from safety engineering), Delphi Method (getting a consensus decision from a panel of experts), and the CARVER method (DoD targeting algorithm). 

The truth is that many of these techniques—while very much worth doing—are not particularly effective at discovering new vulnerabilities.  The last four aren’t even about discovering vulnerabilities at all, but rather are tools to help decide how to field and deploy your security resources.  None of these make much sense for testing security because the logic in using them that way is circular.

Myth:  Safety or safety-like analyses are good ways to find vulnerabilities.

Safety is a very different kind of problem because there is no malicious adversary attacking deliberately and intelligently at the weakest points.  Safety issues aren’t completely irrelevant for infrastructure security, but they are limited in their ability to predict many malicious attacks.

Myth:  These things are the vulnerabilities: The assets to be protected, possible attack scenarios, security delay paths, or security/facility features. 

These things are important in analyzing vulnerabilities and understanding your security, but they are not vulnerabilities in and of themselves.

Myth:  One-size-fits-all. 

Obviously, no single test or certification could have much meaning across a wide range of security applications.  The same thing is true for VAs.  Whenever possible, they should be done in the context of the actual security application and adversaries of interest.

Myth:  Past security incidents will tell you all you need to know about vulnerabilities. 

Looking only at the past is a good way to overlook the risk from rare but catastrophic attacks.  Moreover, the world is now rapidly changing, and what was once true may no longer be true.  Good security requires imagination, peering into the future, and seeing things from the adversary’s perspective.

Myth:  A software program or package will find your vulnerabilities. 

There is nothing wrong with using a software program as a VA starting point, as a checklist, and as a way to stimulate your thinking.  But with security, the devil is in the details.  No security program or package is going to understand your particular security application, facility, personnel, and adversaries in sufficient detail to adequately identify on-the-ground vulnerabilities.  A software app is unlikely, for example, to recognize that frontline security officer Bob falls asleep every day at 3 p.m.

Myth:  Vulnerabilities are bad news. 

Vulnerabilities are always present in large numbers. Finding one means you can do something about it.  This concept is a tough sell to security managers, but it is the correct way to look at vulnerabilities and VAs.

Myth:  You can eliminate all your vulnerabilities.

The unfortunate fact is that some vulnerabilities can’t be fully eliminated, you just have to live with them (and that’s alright as long as you are aware they exist).

Myth:  The ideal scenario is when a VA finds zero or just a few vulnerabilities.

 The reality is that any such VA should be redone by assessors who are competent and/or willing to be honest with you.

Myth:  A VA should be done at the end, when the product is finished or the security program is ready to be fielded.