Searching for the Perfect Fit: Advanced Test Planning

Oct. 27, 2008
Testing your systems the right way at the right times will keep your project on target

Editor's Note: This article is based upon the upcoming book from Auerbach Publications titled Physical Security System Acceptance Testing, by Ray Bernard and Don Sturgis, scheduled for release in the summer of 2007.

Security system technology continues to change at an increasingly rapid pace. Security products and systems continue to become more capable and more complex. Today's software-based and networked systems are so complex, in fact, that manufacturers cannot think of—let alone test—all possible conditions of operation.

The pace of technology change has also led to instances in which manufacturers have released products before sufficiently testing them either on the bench or in the field. Furthermore, the greater the capabilities of the system, the greater the complexity of the system setup work required. All of this means that the testing of installed systems is more important than ever before.

Today's electronic security systems are vastly different from the systems of a decade ago, yet most system providers and users apply the same approach to system testing that they have used for a decade or more.

Test Planning

The purpose of testing is to ensure that the system capabilities expected will actually be delivered.

The purpose of test planning is to ensure that the appropriate tests are performed at appropriate points in the project, so that problems can be identified and corrected as early as possible to prevent unnecessary schedule delays or cost increases.

Sound test planning and execution are the security practitioner's primary means of controlling a project (making sure it is meeting its schedule, cost and quality targets). However, most security practitioners are not aware of the full spectrum of testing that should be a part of a security system project, and thus generally expect that testing will occur only at the very end. By then, they'll have lost all opportunities for early detection of problems.

Many security practitioners hold the belief that milestone payments are an effective means to control a project. Yet many projects with milestone payments still go off schedule and off budget. Milestone payments are a series of pre-agreed sums, each to be paid only when work has reached a certain stage. However, unless appropriate testing milestones are included in a project, milestone payments don't necessarily result in a system that is installed and set up completely and correctly.

Inspection and Testing

The construction and manufacturing industries share a useful adage: “To get what you expect, inspect.” The same holds true for security system projects.

Inspection of installed system hardware and software is performed by walk-through examination. Inspection of the less visible elements—such as the functionality of installed software, the communications infrastructure, and the integration—is performed by testing.

Unique Aspects of Security System Testing

There are several unique aspects of physical security system testing.

• Selected features are tested instead of all the features.

• Scenario-based testing is performed for both normal operations and for security response. (A scenario is a brief description of an event.)

• The system goes live before the final acceptance testing is complete.

• Training is required before final testing, not after, because the system will be put into full use as part of the final test phase.

• Personnel from multiple work shifts (day, afternoon, night, weekend) should participate in the testing.

• To make best use of new systems and new technology, the security practitioner may need to change security processes and procedures, and new procedures may need to be developed. The related organizational functions must also be tested and/or practiced before the system can undergo final acceptance testing.

It is neither practical nor desirable to test all the features of a security system. In fact, some system features will be mutually exclusive. It is, however, important to test the features that you intend to use, in the way you intend to use them.

Security system testing should also incorporate tests that exercise the full set of organiza­tional processes and procedures related to use of the system. Omitting that element of testing can result in a security operations failure at the worst of times.

Testing as an Element of Project Risk Management

One of the largest project risk elements is the ongoing assumption that untested components and systems are installed correctly and will work as expected. Testing is a very important element of project risk management. No project risk management plan can be considered complete without it.

Security Stakeholders and Testing

Security system stakeholders may include senior executives and board members who are involved in project budget approvals, or who have responsibilities related to mitigating organizational security risks. Many a security manager has had to appear before the board or an executive team to explain why his or her security project has fallen significantly behind schedule, why its costs have increased, or why it can't quite seem to be finished.

For security projects that don't incorporate a testing program, a typical scenario goes like this. Progress is reported as satisfactory to the senior project stakeholders throughout most of the project. Then—as system final acceptance approaches—delays are reported and deadlines are missed. The stakeholders want to know why this is happening, since all earlier reports stated that satisfactory progress was being made.

Typically one of the stakeholders asks, “How are you going to prevent further slippage and get the project under control?” No response can be accurate unless it includes the incorporation of an adequate testing program into the project, as well as catching up on the testing backlog.

Security system stakeholders should require that high-level outlines of test plans, actual test schedules, and summaries of test results are included in reports about system and project status.

Security users have a right to a visibly well-executed project and can require that the system provider incorporate appropriate testing. A properly designed, properly executed test plan is their only means of ensuring visible and accurate evidence of project status.

Testing Strategies

There are several testing strategies to be considered. Rarely are testing strategies given conscious thought ahead of time. Here is a summary of testing strategies worth considering:

• Test direction strategy

- User-directed

- Vendor-directed (or system provider-directed)

• Test approach strategy

- Checklist approach

- Tutorial approach

• Test execution strategy

- System provider executes, user observes

- Both user and system provider personnel execute

- User executes, system provider observes

• Test focus strategy

- Technology focus

- Task focus

- Operations or workflow focus

These approaches are not necessarily mutually exclusive. For example, test direction and execution can be performed by the system provider until scenario testing, where the user takes over. Often the optimal test will be a combination of strategies in each category, applied according to what benefits the user the most.

The value of identifying test strategies is that they allow the security practitioner to step back from the technical details to consider the various impacts of testing, in support of the overall project goal: to take ownership of the system with the greatest degree of confidence and preparation feasible.

Security System Acceptance Tests

Five phases of testing are recommended for physical security systems: proof of concept testing, factory acceptance testing, site acceptance testing, operational acceptance testing, and ongoing system testing. The figure below presents a time line that relates the test phases to various project activities.

• Proof of Concept Test

The purpose of a proof of concept test (sometimes called a pilot test) is to demonstrate that a particular technology or one of its features functions satisfactorily for the specific security application.

Many organizations perform on-site proof of concept tests before making a purchasing decision. For example, prior to purchase, Symantec Corporation recently completed a successful pilot test that included installing software to manage physical security privileges globally across different brands of access control systems. The test included an audit of physical access control privilege conformance to its Sarbanes-Oxley compliance policies globally.

• Factory Acceptance Test

A factory acceptance test or FAT (sometimes called pre-shipment test, system test or more recently integration test) is often no longer conducted at the factory, but at the system integrator's facility or even at a user facility. For this test all of the major system components are assembled in one room or area of a building, even though they will be dispersed over many sites when the system is finally installed.

The purpose of this test is to prove that the major features of the systems function properly when connected together in the actual quantities required. It is preferable to substitute demonstrations of live systems at other sites for the FAT if such demonstrations can show the functionality and performance required on the same or a greater scale than intended.

• Site Acceptance Test

The site acceptance test (SAT) is also called a field acceptance test, because it is performed in the field. There is one site acceptance test for each physical location at which system components are being installed. It is a common but poor practice for site acceptance tests to be performed on an ad-hoc basis, without prior planning. This approach often leads to problems and dissatisfaction that can be troublesome to rectify. Proper test setup and user preparation and participation are important factors which, when done correctly, lead to smooth turnover of the system.

• Operational Acceptance Test

The operational acceptance test (OAT), which is sometimes called a field reliability test, is usually conducted as a 30-day test, the purpose of which is to ensure that the system operates reliably under continuous operations over an extended period of time. Sometimes test exercises (such as for emergency response) are included in the test period if normal operations are not expected to sufficiently exercise the system.

• Ongoing System Testing

Ongoing system operational testing (OSOT), also called ongoing maintenance testing (OMT), is performed periodically throughout the life of the security system. The frequency and extent of ongoing system operational testing varies significantly from one facility to another and is highly dependent upon the application of the system. For example, airports test their duress alarm systems once or twice daily, often as part of the work shift changeover.

Types of Tests

Any of the test phases described above will incorporate many different types of testing, such as:

• Functionality testing

• Security scenario testing

• Performance testing

• Stress testing

• Load and capacity testing

• Fault tolerance testing

• End-to-end system testing

• Standards compliance testing

The Cost of Testing

When you consider the cost of testing, make certain you also consider the cost of not testing. The cost of not testing is almost always the higher of the two—and it's an unplanned cost. The costs for planned testing are incorporated up front in the project budget and can range from 15% of the total project cost for very small projects down to 5% of the project cost for very large projects.

The Bottom Line

Security systems are a significant investment. Their purpose is to help protect lives and valuable assets. Those are good reasons to make sure that security projects are well managed. That requires testing.

Ray Bernard, PSP, CHS-III is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities. Mr. Bernard has also provided pivotal strategic and technical advice in the security and building automation industries for more than 18 years. He is founder and publisher of The Security Minute 60-second newsletter (www.TheSecurityMinute.com) . For more information about Ray Bernard and RBCS go to www.go-rbcs.com or call 949-831-6788.

About the Author

Ray Bernard, PSP, CHS-III

Ray Bernard, PSP CHS-III, is the principal consultant for Ray Bernard Consulting Services (www.go-rbcs.com), a firm that provides security consulting services for public and private facilities. He has been a frequent contributor to Security Business, SecurityInfoWatch and STE magazine for decades. He is the author of the Elsevier book Security Technology Convergence Insights, available on Amazon. Mr. Bernard is an active member of the ASIS member councils for Physical Security and IT Security, and is a member of the Subject Matter Expert Faculty of the Security Executive Council (www.SecurityExecutiveCouncil.com).

Follow him on LinkedIn: www.linkedin.com/in/raybernard

Follow him on Twitter: @RayBernardRBCS.