Disaster Exercise Management, Part 1: Performance Measurement

Studying the results of your emergency exercise is the key to obtaining the most from your training exercises


Having such measures will allow exercise directors and training instructors to set targets for improvement and build practical plans for achieving them over time. Diligence in this regard will surely pay off as and when a large-scale disaster reaches far beyond the capacity of available resources.

Learning -- Foundational steps for effective learning are to perform fact-based after-action analyses and disseminate results to participants promptly. Yet measuring how well the exercise participants learned from their experience can be far more subjective in nature than the other three measurement categories discussed above.

Numerous techniques can be employed. Administering "participant satisfaction" surveys following the exercise can be an effective tool in measuring the perceived level of learning. In some cases, more precision may be achieved by comparing results of proficiency tests taken before and after the exercise.

When the exercise can be repeated over time with the same participants, comparisons of performance for each successive exercise can show the degree of progress. For example, in the first exercise only 64 percent of participants fully complied with the established protocol, yet by the end of the third exercise, the same group of participants demonstrated a 97 percent compliance rate.

If performance from exercise to exercise remains stagnant, or worse, declines, then something may be drastically wrong with the process by which participants are given feedback on their performance and advised on steps to improve. Exercise directors and trainers should pay close attention to such indicators and adjust or reinforce their exercise practices accordingly.

Design with the End in Mind

Upon taking a closer look at the measurement process, the number of possible performance measurement data points can become unwieldy. For example, with each measure that applies to human performance, the exercise planner will have to address questions including:

1. Will the measurement be recorded for specific individuals, teams, functional departments, or by some other grouping or resources?

2. Will the measure be recorded separately for a series of distinct tasks or decisions?

3. In the case where measurements are taken by a human observer, will the observer's location be recorded with each measured value?

Clearly, the number of data points to define a single measurement can grow exponentially, and rather quickly.

While there's no set rule for important planning decisions like those called out about, the exercise planner should let the desired end result guide decisions in this matter. Knowing the type of after-action feedback he or she wishes the participants to be given can serve as a valuable guide. Yet exercise planners may find themselves at odds with an inadequate number of resources to record all of the desired measurements. In the end, a mix of prudence and innovative thinking will likely be the key to striking the right mix of measurements.

Conclusion

Incorporating a sound exercise performance measurement process can make the difference between a productive learning experience and a costly project with no material benefit. Indeed, an upfront commitment must be made and followed through to the after-action analysis and debriefing. But if a practical, balanced measurement process is implemented, the benefits of knowing precisely how your organization can become better prepared and do more with less will surely outweigh the effort and cost of such a process. So placing exercise performance measurement high on the priority list is imperative to delivering the payback that the exercise participants and your organization's stakeholders surely deserve.

Having guidelines and a framework for covering all bases of an exercise scorecard starts with the material discussed above. But there should be no mistake that there's more to implementing the exercise measurement process than is covered herein. Things like techniques and information technologies that are used to efficiently record performance measurements, systems that can automate the analysis and reporting of results, and an approach for setting practical targets for performance improvements are just a few considerations. Stay tuned for future articles in which these and other important topics will be addressed.