Pass or fail: The out-of-the-box experience

Part two of the three-part IP Best Practices series (expanded Web version)

It is important to note the valid conclusions that are likely to be drawn by the IT evaluator. These are the points that physical security manufacturers and system providers, and their physical security practitioner customers, are generally not aware of.

Most security industry manufacturers, integrators and consultants are surprised to learn what can be validly concluded from the out-of-the-box experience. Tables 1 and 2 below are charts showing some simple evaluation actions for networked appliances and devices, including example conclusions that can be drawn for the initial evaluation steps from opening the box to examining the documentation.

Although these example evaluation criteria are being presented in an apparently formal fashion in Tables 1 and 2, such evaluations are often not very formal. They are done mostly against the background of common experience. The more experienced an evaluator is, the less forgiving the evaluator will be, because those points of forgiveness are likely to be points of pain and regret somewhere down the line. "Once bitten- twice shy" is an old and common expression. But "thrice bitten-no way!" is a more likely scenario for an experienced evaluator. If the security product will be used in any way to achieve regulatory compliance, the evaluation bar will be set particularly high, and with good cause (see the GAMP white paper referenced above).

The authors have spoken to that it would be unfair to judge their products on the out-of-the-box experience, because they have many successful deployments. But are they defining "success" in the same way that enterprise customers do? It is completely fair to the customer to judge the likely product deployment costs and efforts in large part on the out-of-the-box experience. It's not just the product that is being evaluated. It's the vendor as well, based upon how well the vendor enables its customers to be successful with low-effort deployments and cost-effective customer internal support.


Tables 1 and 2 are only a partial listing of considerations and findings from such an evaluation.

If you are an enterprise security practitioner who likes the performance or specifications of a specific networked security product, you would do well to have that product approved, or even established as a standard, by your IT department.

Before you or your systems provider submit the product to IT for approval or evaluation, be sure in advance that all of the out-of-the-box experience ingredients are included in the evaluation package. If you can't obtain them all, ask yourself whether or not the vendor cares enough about supporting your company's ease of deployment in an enterprise networked environment.

About the Authors

Ray Bernard, PSP, CHS-III is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities. He is founder and publisher of The Security Minute 60-second newsletter ( For more information about Ray Bernard and RBCS go to or call 949-831-6788. Mr. Bernard is also a member of the Subject Matter Expert Faculty of the Security Executive Council (



James Connor is founder and CEO of security technology consultancy N2N Secure, a security consulting firm specializing in migration of analog to converged IP-based Physical and Logical security solutions. He is the former Senior Manager of Global Security Systems for Symantec Corp.






Rodney Thayer is an independent network researcher focusing on network attack and defense issues as they relate to business infrastructure. Current security research (exploit development) includes product and infrastructure evaluations, and training/lecturing on computer security topics. Mr. Thayer's background is in engineering, deployment and evaluation of computer and network security solutions. He has participated in the authoring of IETF standards, written product reviews for trade publications, taught at venues like RSA and Black Hat, played Capture The Flag at Defcon (on a winning team), and has consulted for large and small enterprises and Infrastructure Operators.