Planning enables central stations to survive Sandy

Despite widespread power and phone outages, redundancy helps central stations weather the storm

“We had two client central station companies that were on full disaster recovery stand by,” said Melissa M. Courville, head of marketing at Dice Corp. in Michigan. One company was 80,000 accounts in size and the other 50,000.

“These companies were processing overflow signals through the DICE Disaster Recovery Center,” Courville explained. “This methodology happens when clients don't need to do a full fail-over but their signal transmissions to the DICE DR allow them to send any overflow to us to grant them additional bandwidth to handle the extra signals they're taking on for processing.” Bay City, Mich., where DICE is headquartered, was in a safe harbor. While the East Coast was reeling, Bay City had a light drizzle and cloudy climate, not much different than any other late October day.

The day following the brunt of the storm, Dice had another central fail-over. This involved about 100,000 accounts. Courville said that incident was due to telephone line outages. The company immediately started the process of rolling over the IP infrastructure aspect of their business, which is another 40,000 accounts, she said.

That brought the figures to approximately 270,000 total accounts running on the DICE Disaster Recovery Center. “Our DICE DR team worked around the clock to ensure that our client profiles were already pre-loaded and ready to go in the impact zone,” Courville said.

Other client companies that are hot-redundant on DICE, with full redundancy capabilities, failed over to DICE and back again without DICE intervention as they are automatic.

By late Tuesday, long-distance carriers like AT&T and Verizon were having trouble getting toll-free traffic out of New England. The COPS staff discovered it could move traffic from New England to the West Coast and rerouted everything that way. Knowing what was possible and having a plan to implement it was a life-saver everywhere.

Going proactive

Mason Monitoring, located in Kings Park, N.Y. went pro-active right away. Mason sent an email blast to its customers acknowledging that Hurricane Sandy had ravaged the area. “The early reports indicate that this Super Storm has caused wide spread damage to many major cities leaving millions without power,” Mason’s website said even before the rains had stopped. “Through it all we’re proud to say that Mason is here…undeterred and on the job.

“While our neighborhood sits darkened and powerless in Sandy’s wake our central station remains a literal beacon of light,” Mike Cannatella told customers. ”Our back up plan in full effect, we march on…it is what we do. It is that kind of reassuring message, coupled with a subtle sales pitch reinforcing quality and resiliency that keeps customers calm.

“We were up 100 percent. We didn’t miss a heartbeat,” Cannatella added. Mason serves about 40,000 customers and much of their customer base in Connecticut was without power. This resulted in many air-conditioning failures and other alarms. Mason brought in a few more workers to handle the alarms.

Ironically, Mason had just posted a notice to its website a few weeks earlier. In August, their service area suffered a major power outage due to Hurricane Irene. Thanks to a brand new 100 kilowatt (412 continuous amps) John Deere diesel generator with a 400-gallon tank, they were unaffected. For extra redundancy and added peace of mind, they signed a four-hour service contract to replace or service that new generator. “Should that circumstance ever arise, our 10-hour battery backup supply will hold us for almost three times that duration,” the website trumpeted. Little did they dream that just two months later, they would be testing the system to its limits.

Mason also has a smaller, gas-powered unit. Cannatella figures they could be down for a month and still maintain service. Backup and UPS is part of being in the central station business, Cannatella said, noting that it is just like having a spare tire in the car.

At the most basic level, a 6 kilowatt generator could keep 20 computers running. However, most central stations go much bigger.

Post-mortem diagnosis