Disaster Exercise Management, Part 1: Performance Measurement

Studying the results of your emergency exercise is the key to obtaining the most from your training exercises

Timeliness -- Perhaps the most frequently cited and arguably most important exercise performance measurement category is timeliness. Timeliness measures that might come to mind first are those relating to the management of the exercise, i.e., the times that key events, tasks and decisions start and end.

However, the scope of timeliness measurements spans far outside the box of scheduling and project timelines. In particular, of greater importance to evaluating preparedness are measures of how much time was taken for specific "mission critical" tasks and decisions. For example,

  • From the time of first biohazard detection, how quickly did the security team seal off the building's ventilation system?
  • How many minutes into the disaster did the CEO decide to send people home versus shelter them?
  • For how long were the denial of service attacks allowed to persist before the internal network was restricted from outside connections?
  • At what point in the disaster exercise did the CIO become fully apprised of the situation?

Operational measures like these and others are needed to rate how well prepared the exercise team is to perform in a live disaster. Merely completing an exercise that ran according to schedule is not enough. Having precise measures of time intervals for mission critical tasks and decisions is essential to knowing whether or not the disaster response capabilities of the organization are ready to avoid business operations meltdown, property loss, impairment of critical infrastructure and casualties.

Effectiveness -- While speed of execution and punctuality are often key measures, how well mission-critical tasks and decisions are carried out may be of equal or greater importance. Such measures of effectiveness can address procedural compliance (e.g., Were evacuation procedures followed properly?) as well as the results of activities performed or decisions made during the course of the exercise (e.g., How many gallons of oil spilled over the containment barrier?, How many patients were diagnosed and inoculated in the first four hours?).

Human performance is not the only consideration, however. Other assets such as equipment, information systems, and protocols may also be appropriately placed under the effectiveness lens. For instance,

  • What percentage of the breathing apparatus failed during the exercise?
  • For how many members of the forward command team were protective suits made available before the team first entered the hot zone?
  • Of the 17 critical steps, how many were successfully carried out by participant #21?
  • Were the protocols sufficiently clear to be executed without hesitation or confusion?

Efficiency -- Efficiency measures direct our attention to how much was accomplished in relation to the number of resources utilized. Historically, efficiency has not been a top concern with respect to preparedness. However, in these times of shrinking budgets and an expanding set of threats, doing more with less has become an imperative. Measuring efficiency in exercises is a key first step in learning how to better utilize precious resources. Examples of efficiency measures are:

  • What number of security personnel were dispatched to the point of perimeter breach?
  • How many analyst work-hours were required to arrive at a correct interpretation of the intelligence?
  • On average, how many victims passed through each of the four decontamination stations per hour?

When critiquing exercise performance it is not uncommon to find a strong interdependency between measures of timeliness, effectiveness and efficiency. For example, a decontamination station will not likely be deemed timely or efficient if its people and equipment are highly prone to making errors. Having side-by-side measures of timeliness, efficiency and effectiveness sheds far more light on the root cause of any deficiencies than if only one or two measurements are used. In the decontamination station example, a prudent first step would be to lower the error rate through training and/or use of more accurate equipment. Simply adding more decontamination station resources could be a far more costly solution.