Related to the observer's evaluative function are performance measures. Performance measures provide a consistent means of evaluating participants' conduct during the exercise. Perhaps the most common set of measures pertain to timeliness. Examples would be "minutes elapsed from the call to the first security guard arriving at the scene", "time to evacuate 90 percent of the building's occupants", and "number of days to restore the company's stock to its pre-disaster level". Other measures can pertain to efficiency, capacity and efficacy. The set of plausible measures can be quite large, and because of this an entire article has been written by the author on this topic (see the SecurityInfoWatch.com article Disaster Exercise Management, Part 1, Performance Measures).
After the Action
Following the exercise, all observations, recordings and other sources of performance measurement should be collected and compiled. Depending on the size of the exercise and how many observations and recordings were made, this task may be quite laborious and should be planned for accordingly. It is generally a good idea to collect the information immediately following the exercise so as to allow a prompt after-action analysis.
The post-exercise analysis and feedback of results is commonly referred to as "after action analysis", "debrief" or a "hotwash". This step may necessitate a meeting of the exercise director, observers and certain support personnel to discuss and distill the findings down to a concise set of actionable recommendations. The findings may comprise assessments relative to expectations of the company's executives, employee's, shareholders, customers, company auditors and/or government officials. In general, the assessment should address performance in terms of timeliness, effectiveness and efficiency (see above cited article on performance measures for examples).
The findings of the analysis should be communicated to the participants in one or more "feedback" sessions. The session(s) should occur as soon as practical after the exercise and give the participants a clear assessment as to how well they performed and what areas they need to improve upon. Various techniques may be used for this purpose. At the discretion of the exercise director, the feedback may be given in reference to individuals, teams or an entire organizational unit. For example, it may be useful to play back audio or video recorded during the exercise and have group discussions with constructive criticism from the exercise director or observers. Summaries of observers' evaluations may also be presented followed by additional commentary from the observers. Self-assessment by the participants may be another effective performance rating tool.
The exercise itself should also be evaluated in terms of how well it met, or did not meet, its objectives. Certainly, the quality of the learning experience would be among the criteria of evaluation. Feedback from the participants will be instrumental in this assessment and should be actively sought by the exercise director and his team.
In cases where significant areas of improvement are identified, it may be appropriate to develop a plan to close these gaps. A variety of gap-closure steps may be advised. Examples could include the need for additional training of select participants, or the procurement of new safety equipment, or even the scheduling of further exercises to zero in on the "problem" areas. A repeat of the exercise might even be included in this plan so that after so many months the participant's will be tested again under similar or identical circumstances to verify closure of the gaps. Agreeing to specific dates by which key milestones will be met is a good idea before the session adjourns.
Exercises offer a powerful tool for preparing your organization for the risks of this day and age. When planned and carried out in a well-orchestrated manner and with a clear purpose they can give the participants a great learning experience that is second only to a live disaster.
About the author: William Comtois is managing director of Varicom, Inc., a consultancy and software company specializing in homeland defense and service logistics. He has over 20 years of experience in applying leading technologies and innovative process management practices to business and defense solutions. Over the past fourteen years, his work has focused on large service companies where he has lead numerous performance improvement, training and process management initiatives that have resulted in major breakthroughs in financial performance, service levels and disaster preparedness. He can be reached by phone at (212) 561-5782 or email at firstname.lastname@example.org.