A decade ago, I had a front-row seat to a costly physical security deployment debacle. The integrator on the project had dispatched two employees to install a physical security system for a customer in another city. The technicians set up the cameras, rack-mounted the server, and discovered that the application was DOA because of assembly errors in the custom-engineered platform. The hardware builder couldn’t deliver a new unit for 10 days, the technicians had to fly home and back a second time, the integrator lost his profit, and the software vendor had to discount his next 10 orders to compensate.
That wasn’t the first time I saw an installation sidelined by snags with the hardware, and very little has changed since then. Whether you’re talking about video surveillance, access control systems, or other physical security systems, the hardware needed to serve up applications and store associated data can turn hard-won sales into crisis situations in less time than it takes for a camera to capture a break-and-enter. Meanwhile, costs escalate and the reputations of both the ISV and the integrator take a hit.
Hardware Woes (Sound Familiar?)
At the heart of the problem is the lack of transparency into server manufacturing and associated processes. Until recently, for example, there has been no way to determine whether the components in a commercial off-the-shelf platform match what was spec’d without opening the chassis to figure out why it won’t boot up. (And good luck with that if you’re not a platform engineer.)
I’ll have more to say about that as well as quality control in a minute, but first, let’s look at a few of the hardware issues that can send these projects careening off the rails. I’m sure you have seen them all and suffered the consequences:
- Configuration errors. Even the smallest change to a base platform from Dell, Supermicro, or any other supplier can cause a project to go awry. Optimizing a platform for a specific application is a manual process that typically lacks automated quality checks, so a server can easily go out the door with the wrong memory, networking card, software image, or any number of other errors that aren’t detected until the unit is in the field and the integrator is under the gun.
- Unapproved component changes. Sure, your hardware builder provides a bill of materials, but assembly technicians frequently substitute components that aren’t on the BOM without notification or permission based on what’s available. I know of one case where a server was delivered with 14 12 TB drives instead of the 10 16 TB drives that had been spec’d. The capacity was nearly the same but the open drive bays needed for future expansion were missing in action, forcing the integrator to send the platform back to the builder for a redo.
- Non-trusted components. In today’s world, it’s critical to know where system components come from because of concerns that countries like China may embed suspect chips capable of stealing information. Yet integrators and software vendors typically have no visibility into the provenance of their hardware components to ensure that their supply chain is safe, potentially compromising security implementations.
- Late delivery. The integrator again bears the brunt of this glitch in hardware procurement in the form of installation team scheduling problems, customer blowback for project delays, and penalties of up to $10,000 a day in the case of government projects. None of this wins the software vendor any popularity contests either. Advance notice of fulfillment delays could help soften the blow in some cases, but most builders lack a mechanism for alerting customers when they’re going to miss a ship date.
In a LinkedIn survey we conducted recently, 49% of respondents ranked misconfigured platforms as their biggest pain point in procuring servers for their physical security projects. Late delivery came in second at 38%, followed by component substitutions (9%) and wrong product shipped (4%). If this happened in any other industry, heads would roll.
Unsticking the Sticking Points
There are two common denominators in these issues, as indicated earlier. One is the absence of transparency, leaving both the security software provider and the integrator without visibility into the componentry and other details of the COTS servers they need for turnkey deployments. The other is poor quality control in an environment where the manual steps required to customize and image each box for the specific application create ample room for human error.
These problems can be solved if hardware manufacturers are willing to dedicate resources to developing the appropriate technologies and toolsets.
On the quality assurance front, for example, many errors can be prevented by automating quality checks as systems move from one work center to the other based on the build instructions for that particular platform. If any configuration step is overlooked, production software with the appropriate controls will be able to automatically detect the omission and prevent the builder from signing out and passing the mistake on to the next work center. Imaging, testing and verification processes can also be automated.
Safeguards like these can eliminate configuration errors such as missing or incorrect components, ensure that the correct software version is loaded and tested on the hardware, and thereby avoid some of the most common causes of hardware trouble.
On the visibility front, hardware manufacturers can take a page from Amazon and build customer portals enabling ISVs to track every aspect of platform engineering, manufacture, inventory and shipment. Users should be able to see SKUs, serial numbers, country of origin and EOL dates of every component in every box as well as the software version imaged with a click. BOMs should be stored and locked in the same interface. Approved engineering change orders should be required for any component switches and stored in the system for reference.
This kind of portal can also allow open orders to be tracked by their specific location on the manufacturing floor and automatically adjust due dates if a component or production issue causes a delay. Customers can then be automatically notified, eliminating multiple calls and emails to determine order status.
Innovations like these will lead to less troubleshooting, fewer wild goose chases to get information, and fewer customer complaints about fulfillment problems and hardware failures. That, in turn, will help protect your brand as well as your bottom line.
After all the effort you’ve put into developing and selling your physical security software, it’s time that hardware builders step up to the plate to eliminate errors that will leave you in limbo just as you reach the final inning. Some providers have already made the investment. When these policies become standard, every deployment will be a home run.
