An overarching objective of disaster exercises is to shed light on how well participants and other resources will perform in the heat of a live disaster. While this "predictive" function is clearly bound by the degree of realism in the exercise itself, the extent to which one can assess how prepared we truly are is largely determined by what, how and when we measure performance during the exercise. Performance measures therefore play a vital role in closing "proficiency gaps" and determining when we have in fact reached the desired levels of preparedness.
In the author's experience, performance measures are all too often dismissed as unnecessary or given low priority in relation to the overall success of an exercise. Yet, if well crafted, a performance measurement program can be a relatively small piece of the effort and cost that goes into a disaster exercise. A first and essential step in bringing performance measurements to the forefront of disaster exercise management is having guidelines and techniques that can make the design, recording and reporting of performance measurements a straightforward and streamlined task. To this end, this first in a series of articles offers some guidelines and a framework for developing a practical exercise scorecard. In subsequent articles, related topics such as recording technologies, reporting tools and after-action analysis techniques will be covered.
Measuring for Success
What makes a good exercise performance measurement process? At a minimum, the exercise performance measurements should serve two ends:
1. Answer the question "Did the exercise achieve the desired performance levels?", and
2. Provide the exercise participants with a clear understanding of how well they performed and what they need to improve upon.
Any exercise lacking in either of these top-level objectives is unlikely to be deemed a success.
To get one's arms around the seemingly large task of planning and carrying out an exercise measurement process, a number of carefully orchestrated steps are needed.
Alignment with Performance Objectives
As a starting point, the performance objectives of the exercise must be clearly defined. For instance, are we setting out to test the speed with which our internal security group can detect a breach of the grounds perimeter, evacuate a building, etc.? Or are we out to verify that our company's executives will properly and at the right moment notify government agencies in the event of a never-seen-before disaster. With objectives in hand, the exercise planner can go about defining measurements that will ultimately tell us whether or not these performance objectives were met.
Along the way, the exercise type must be taken into account. For instance, a discussion-based exercise might not be concerned with response times so much as it would be with making the correct decisions and commands. A scorecard for the exercise might therefore consist of measures that focus on the correctness of decisions and compliance with protocol rather than how quickly decisions were carried out. In contrast, field exercises involving search and rescue or bomb diffusion would surely call for specific time-oriented performance measures.
The disaster type may also dictate what measures should be included in the exercise scorecard. An exercise involving the exposure of an air-borne biohazard to the company's corporate headquarters building would probably not be concerned with how effectively the company's web site can fend off a denial of service attack.
Coverage and Balance
Given the uniqueness of each exercise, there's no "cast in stone" list of measures for disaster exercises. However, there is good reason to have a set of measures which together cover four essential categories: Timeliness, Effectiveness, Efficiency and Learning. One might ask why all four categories are advised, especially when only timeliness and effectiveness appear directly related to preparedness. The answer rests on the presumption that all stakeholders have a desire to know (1) how swiftly the exercise participants will act, (2) how well they will perform their duties, (3) whether or not the same result could be achieved with fewer resources, and (4) how to become better prepared for a live disaster. The following explains the role of each category in an exercise measurement scorecard.