There can be special considerations involved in deploying Uninterruptible Power Supply (UPS) devices and performing an orderly shutdown upon loss of main power to video system servers.
Q: Why did we still lose all our recorded video when we had a long power outage, even though the video server was connected to our UPS and should have performed an orderly shutdown before the UPS battery was exhausted?
A: The network attached storage was not also on the UPS and so was not protected from instantly being powered off upon the loss of power. Data corruption resulted.
For DVRs, at least partial loss of recorded video upon power outage was common, unless the DVR had UPS battery backup power and the power outages were short enough that the UPS could sustain the machines throughout the outage. Many DVR systems did not have backup power. Unless cameras and any connecting equipment, such as video amplifiers or network switches, can be run off battery backup or emergency power, it makes no sense during a power outage to keep video recording units running?whether DVRs, NVRs or servers. There would be no video data to record.
The main concern for the recording units should be to keep them powered long enough to shut them down correctly so that there is no data corruption. Powering cameras from local power sources close to the camera makes UPS backup power for camera infeasible — a UPS at every camera would be cost-prohibitive.
Using toady’s PoE-powered cameras, however, means that keeping the network switches powered automatically keeps the cameras powered. So, where maintaining video monitoring of critical asset areas is crucial — such as healthcare facilities — PoE power can be an affordable solution that enables cameras as well as their recording units to keep functioning throughout a power outage.
For most video systems, emergency power will not be provided during long power outages. Even where emergency backup power is available, if there is even a momentary problem, power system video servers could still lose power. Thus, for nearly all security video deployments, it is important to address UPS backup power and orderly shutdown for video servers and all related components.
Orderly Shutdown and Startup
The simplest scenario for implementing orderly shutdown is a single server computer containing the video storage hard drives, whose backup power comes from a UPS dedicated to that computer. A USB connection from the UPS to the computer provides the data interface, and software provided by the UPS manufacturer can be configured to both perform an orderly shutdown when a main power outage occurs, and ensure that the computer powers up when main power is restored.
Various options exist to allow for an appropriate shutdown sequence, for example: shutting down the VMS application, then shutting down SQL server (for example), then shutting down the operating systems and powering off the machine.
An example of a very complex orderly shutdown scenario is one with multiple video servers, and a UPS shared by IT servers as well as the video servers. The video servers are virtual machine (VM) host computers — each with several virtual machines running various applications, including the main VMS system, video analytics, SQL Server, network monitoring software, VM operating systems and servers, a video storage area network (SAN, where video data is written to) and any network equipment switches and routers that connect video and storage servers.
In a complex system, the shutdown time may be 30 minutes or longer. This may exceed the requirements of other UPS users in the data center, equipment rack or room where the equipment is located.
A UPS is required that supports network-based data connections from multiple computers, not just a one-computer USB connection. Typically such a UPS would support SNMP (Simple Network Monitoring Protocol), which IT would use to monitor the status of the UPS and receive notifications about changes in power status, which could be used to trigger power outage notifications to Security. Vendor-specific protocols also exist, usually for use with vendor-provided UPS management software.
The IT department would establish the shutdown and restart configurations for the IT servers on the UPS. Be sure to establish logging with real-time error notification for the shutdown sequence, so that contingencies for video outage can be activated, and any errors can be identified and corrected allowing quick manual intervention in the event of a significant shutdown error.
Large hard disk storage arrays, whether inside the server computer or in a separate storage system, can use up a lot of battery backup power. This means that accurate calculation of total UPS load and battery-backed uptime for the systems is very important. Fortunately, once the UPS battery is fully charged, the software provided with the UPS will report the Estimated Battery Time for the connected load.
The UPS must be able to support the total length of time required for the full shutdown sequence, including a good margin of safety for shutdown time. Error logging and notification adds more time to the overall shutdown process. Both the full shutdown sequence and startup sequence must be tested and timed, including notifications.
An orderly startup sequence is required when power is restored, which may or may not be the reverse sequence of the shutdown. Startup times will be different than shutdown times, and startup operations should also be logged with real-time notification.
Your security video system is a significant investment. Don’t allow power outages to destroy recorded video or corrupt server operating system files, when a good UPS system can prevent such problems for a fraction of the cost of the total system.
If you attend ASIS, take data about your VMS deployment (current or planned) to the show floor in Chicago, and ask candidate vendors how they would recommend you establish orderly shutdown and restart of all the server components of your system. Request an application note, white paper or specific written guidance — they really should have such materials.
Write to Ray about this column at [email protected]. Ray Bernard, PSP, CHS-III is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities. For more information about Ray Bernard and RBCS go to www.go-rbcs.com or call 949-831-6788. Follow him on Twitter: @RayBernardRBCS. Mr. Bernard is also a member of the Content Expert Faculty of the Security Executive Council (www.SecurityExecutiveCouncil.com), and is an active member of the ASIS IT Security Council and Physical Security Council.
