Chapter 7, Risk Commission Final Report

Evaluating Results

Why Evaluate?

At this stage of risk management, decision-makers and other stakeholders review what risk management actions have been implemented and how effective they have been. Evaluating effectiveness involves monitoring and measuring, as well as comparing the actual benefits and costs to estimates made in the decision-making stage. The effectiveness of the process leading to implementation should also be evaluated at this stage.

Evaluation provides important information about:

Whether the actions were successful, whether they accomplished what was intended, and whether the predicted benefits and costs were accurate.

Whether any modifications are needed to the risk management plan to improve success.

Whether any critical information gaps hindered success.

Whether any new information has emerged that indicates a decision or a stage of the Framework should be revisited.

Whether the Framework process was effective and how stakeholder involvement contributed to the outcome.

What lessons can be learned to guide future risk management decisions or to improve the decision-making process.

A good risk management decision can be shown to have a significant impact on the risks of concern.

Monitoring health indicies can be one method of evaluating whether risk
management has been successful.

Tools for evaluation include environmental and health monitoring, research, disease surveillance, analyses of costs and benefits, and discussions with stakeholders.

Evaluation is critical to accountability and to ensure wise use of scarce resources. Too often, past risk management actions have had little or no evaluation or follow-up after implementation, even when evaluation was mandated.

Planning for Evaluation

Plans for evaluation should be built into the overall implementation plan to specify when evaluation will be conducted, who will conduct it, and what will be evaluated. In most situations, periodic evaluation will be important. The focus of evaluation may shift with the stage of implementation, because it often may take some time before the full impact of risk reduction can be measured. Evaluation might first focus more on progress and success in implementing the risk management plan. Later evaluations may focus on the success of the risk management actions in reducing risk.

In the past, evaluation, when conducted, has been performed by the regulatory authority itself. As with other stages of the risk management process, evaluation will benefit if stakeholders are involved, helping to:

Establish criteria for evaluation, including the definition of "success."

Assure the credibility of the evaluation and the evaluators.

Determine whether an action was successful.

Identify what lessons can be learned.

Identify information gaps.

Determine whether cost and benefit estimates made when evaluating the risk management options were reasonable.

"A good risk management decision can be revised and changed when significant new information becomes available, while avoiding "paralysis by analysis."

The Importance of Iteration

New information may emerge during evaluation that is of sufficient importance to indicate that parts of the Framework should be repeated. For example, revisiting a decision might be needed if a more effective risk management option or a less costly option of equal effectiveness is developed. Public comment, negotiation, information-gathering, research, or analysis of risks and options could clarify or redefine the problem, change the focus to a different problem, or identify other risks in a broader context. In such cases, the risk management process will not be sequential, but rather flexible and iterative as important new information, ideas, and perspectives come to light. The Commission’s Risk Management Framework provides that flexibility.

While an iterative process is important for incorporating new information, it should not become an excuse for taking no action. Decisions must be made, even when information is imperfect.