The importance of peer review in regulatory decision-making has been highlighted recently by the prominence of requirements for peer review in several regulatory-reform bills before Congress. Earlier versions of those bills included prescriptive instructions regarding the nature and duties of peer-review panels. Later versions of the bills have been less prescriptive. Peer review is an important and effective mechanism for evaluating the accuracy or validity of technical data, observations, and interpretations, and the scientific and economic aspects of policy recommendations and regulatory decisions.

FINDING 5.5: Peer-review activities in federal regulatory agencies are generally devoted to evaluating the quality of the science and the scientific interpretations that underlie a regulatory decision. The quality and interpretation of other technical information, especially that related to economic analysis and the social sciences, are generally ignored. Peer review has not been used to evaluate the use of scientific and economic information in regulatory decisions, however, and there are no procedures for evaluating the effectiveness of peer review itself. Several agencies do not have official guidelines or policies for peer review. Of course, peer review can be overdone; implementing a peer-review process for every regulatory decision or every step in a regulatory decision would lead to substantial delay and require excessive resources.

RECOMMENDATION: The role of peer review should be expanded to consider not only the quality of technical information, but the use of that information in regulatory decision-making. Peer review of economic and social science information should have as high a priority as peer review of health, ecologic, and engineering information. Clear, written guidelines for peer review should be established by regulatory agencies, and the effectiveness of agency peer-review programs should be evaluated regularly. The level of peer review should be commensurate with the level of scientific or economic importance and regulatory impact of the decision to be made. Peer review should be conducted not simply to seek legitimacy for agency decisions and positions, but to improve their quality. When peer review is judged to be unnecessary, an agency should provide an explanation and justification.

RATIONALE

Peer review provides independent views of an issue. When used well, peer review can serve as a system of checks and balances for the regulatory process. In the context of risk analysis, an open process for peer review can increase the credibility of and confidence in an assessment. Peer review can make important contributions to a collaborative decision-making process that involves stakeholders. Administrative details--such as how peer reviewers are selected, which agency products, regulatory options, or decisions will be subject to peer review, whether and how consistency among an agency's programs should be improved, and how the outcomes of peer review will be used--should be addressed by an agency's peer-review policies. EPA's program-specific standard operating procedures for peer review called for by its peer-review policy (EPA 1994) are examples of useful guidelines for peer review. Peer review of the output of the risk-assessment and options stages in the Commission's risk-management framework (section 2) is essential for all major rules under development. In some cases, peer review might be useful in the problem-formulation stage.

Good science can be used to justify bad regulations. Asking whether relevant scientific or economic information was cited appropriately in a particular regulatory process is critical. There appear to be no mechanisms in place that support peer review of the use of technical information at the policy stage. Perhaps scientific advisers to the EPA administrator, the FDA commissioner, or the OSHA administrator fill that role informally. Most peer reviews evaluate highly focused, technical topics because of the assumption that scientists and economists tend to lack an understanding of the history and philosophy of an agency's decision-making process. A mechanism for evaluating the descriptions and uses of scientific and economic analysis in the decision-making stage should be sought. The Commission does not suggest that the regulatory decision itself should be peer-reviewed, which, of course, is the purview of the judiciary.

Agency peer-review policies should include a regular evaluation process in which specific examples of an agency's use of peer review in its regulatory decision-making are examined. That evaluation would ask questions about how a peer review was conducted, whether and how the outcome of a peer review was used in a regulatory decision, whether the peer review was considered useful, and finally, how the process could be improved. A good example of agencywide evaluations of the role of peer review is described in the EPA publication Safeguarding the Future: Credible Science, Credible Decisions (EPA 1992b). Evaluations could be organized by the agency, such as EPA through its Science Advisory Board, or across agencies, such as by the Office of Science and Technology Policy or the risk-assessment subcommittee of the administration's Committee on Environment and Natural Resources.

Potential peer reviewers with clear conflicts of financial interest should be disqualified from service on peer-review panels that could directly influence regulatory decisions related to the products or interests of their organizations. However, it is difficult, if not impossible and unwise, to eliminate bias, which reflects views or positions taken that are largely intellectually motivated or that arise from a person's close identification or association with a particular point of view or with the position or perspectives of a particular group. The Commission believes that expertise, balance of biases, and inclusion of active, younger, and culturally diverse scientists, economists, and social scientists should be among the criteria for constitution of peer review panels. Explicit criteria for revealing and evaluating conflicts and biases are needed.

The person(s) responsible for selecting peer reviewers can have a great deal of influence on the nature and biases of the membership, the expertise represented, and, by extension, the outcome of the review. Those persons can also have a lot of influence on what is peer reviewed. That gatekeeper role should be structured carefully to ensure that a small number of people do not have undue influence on reviewers' characteristics or decisions or on what is chosen for peer review.

Full peer review is unlikely to be needed for every regulatory decision. The most-effective and most-efficient use of peer review should be made case by case, taking into account such issues as the extent to which the scientific information on which a decision is to be based might be considered controversial, the economic impact that a decision might have, and agency resource constraints. Peer review should not be used as a device to delay controversial policy decisions.