Debunking vulnerability assessment myths: Part 2

Characteristics to look for in vulnerability assessors, things to include in a VA report


Other Misunderstandings About VAs

There are other common VA problems and mistakes that should be avoided. Sham rigor—thinking that the VA process can be done in a rigorous, formalistic, linear, reproducible, and/or quantitative manner—is a common problem. In fact, effective VAs are creative, right-brain exercises in thinking like somebody you’re not (the bad guys). The VA process is difficult to formalistically characterize, reproduce, or automate (see the Vulnerability Pyramid).

Another common VA mistake is to focus on high-tech attacks. In our experience, relatively low-tech attacks work just fine, even against high-tech devices, systems and programs. It is also a big mistake to let the good guys and the existing security infrastructure and strategies define the problem—the bad guys get to do that. We must also be careful not to let envisioned attack methods solely define the vulnerabilities—it ultimately has to work the other way around.

Placing arbitrary constraints on the VA in terms of scope, time, effort, modules, or components is also a common mistake. Often, software experts are brought in to look at the software, mechanical engineers to look at the physical design, electronics experts to examine the electronics, etc. While there is nothing wrong with using experts, the fact is that many attacks occur at the interface between modules or between disciplines. An effective VA needs to employ a holistic approach and people who can think holistically.

There is nothing wrong with testing and certifying security devices, systems, and programs—assuming the tests and certifications are relevant, meaningful, and well thought through. But testing and certifying is something quite apart from undertaking a vulnerability assessment. Be sure you understand what a vulnerability assessment is and is not, how it should be done and by whom, and why it is important to do it.

About the Authors: Roger G. Johnston, Ph.D., CPP, is leader of the Vulnerability Assessment Team at Argonne National Laboratory. He was founder and head of the Vulnerability Assessment Team at Los Alamos National Laboratory from 1992 to 2007. Roger graduated from Carleton College (1977), and received M.S. and Ph.D. degrees in physics from the University of Colorado (1983). He has authored over 170 technical papers and 90 invited talks (including six keynote addresses), holds 10 U.S. patents, and serves as editor of the Journal of Physical Security.

Jon S. Warner, Ph.D., is a systems engineer with the Vulnerability Assessment Team at Argonne National Laboratory. From 2002-2007 he served as a Technical Staff Member with the Vulnerability Assessment Team at Los Alamos National Laboratory. His research interests include vulnerability assessments, microprocessor and wireless applications, nuclear safeguards, and developing novel security devices. Warner received B.S. degrees in Physics and Business Management at Southern Oregon University (1994), and M.S. and Ph.D. degrees in physics from Portland State University (1998 & 2002).