Ray Bernard, PSP, CHS-III, is a leading security consultant and author, who over 26 years has led many noteworthy security projects for international airports, nuclear disarmament facilities, sports stadiums, water districts, energy utilities, hotels, manufacturing plants and multiple-tower high-rise facilities (www.go-rbcs.com). Follow him @RayBernardRBCS
Today’s physical security devices and systems require a significant amount of configuration. Although cameras and other devices can be dead on arrival and hard drives can fail, those are not the common causes of ongoing device and system problems.
Poor configuration by unqualified technicians is a common factor in persisting system problems, as was the situation underlying this reader’s question:
Q: What ever happened to troubleshooting and root cause analysis? My company’s engineers do it all the time. It seems like my security integrator knows only two problem-solving actions: reboot and replace.
A: Unfortunately this is a too-common situation with multiple contributing factors, including end-users who shortchange their own security projects by failing to specify and insist on two things: (1) training and certification requirements for integrator project personnel; and (2) fully complete, as-built documentation and detailed trouble-shooting documentation.
A high percentage of troubled projects have personnel who are under-qualified in terms of experience and training, including project management (bitly.com/PhysSecPM) and technical training. In several recent project recovery efforts, many of the project team members had no product or general technical training or certification beyond 2008. Obviously, a lot has happened in terms of technology advancement since then.
Many technicians in other industries get formal training in troubleshooting, where they learn the importance of documentation in the troubleshooting process.
Documentation is Critical
Documentation is critical to troubleshooting efforts, and that is where efforts to fix troubled security projects can be stopped dead in their tracks. Many security projects have no as-built documentation at all, and some have only physical installation documentation and no records of system setup. Rarely does one find a configuration plan, which is a design document that explains how security devices and systems will be configured to achieve their functional and performance objectives.
During the acceptance testing process, all the commissioning work is checked against the configuration plan. Without a plan — which is where many security departments are sitting today — acceptance testing is impossible. There is no basis for determining if devices are set up correctly, and you will not have a starting point to use in managing future settings changes and upgrades.
The configuration plan is only step one. The next aspect is “change management” — which is a set of processes and procedures for managing the changes and upgrades to device and system configurations. IT departments usually have a change management (or configuration management) policy document, which describes key requirements that apply to making changes to deployed technology.
When as-built documents are poor or incomplete, the cost of troubleshooting skyrockets. I once observed a nearly six-hour service call that began with three hours of wire-tracing (the cables were not labeled), an hour of software-checking to find what the device and system settings were, and another hour of finding and reading manufacturers’ documentation. This research — which would have been unnecessary with proper documentation — was followed by a whopping 15 minutes of corrective action.
If the documentation was there, it would have resulted in a 30-minute service call — and it would have been a lot cheaper, as this call cost the customer more than 10 times what it should have. Because the technician did not label any wires and did not leave behind a detailed record of the service work, the next technician will probably have to repeat some or all of the research process.
Your situation is probably not this bad — after all, there are plenty of qualified security integrators out there. But without complete as-built documentation that includes device and system configuration information, you can bet you and/or your service providers are spending more on maintaining your technology deployments than you should be.
Write to Ray about this column at ConvergenceQA@go-rbcs.com. Ray Bernard, PSP, CHS-III is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities. He is founder and publisher of The Security Minute 60-second newsletter (www.TheSecurityMinute.com). For more information, visit www.go-rbcs.com or call 949-831-6788. He is also a member of the Content Expert Faculty of the Security Executive Council.