Chapter 6
The Role of Peer Review in Regulatory Decision-Making
Peer review is an important and effective mechanism for evaluating the accuracy or appropriateness of technical data, observations, interpretations, and the scientific and economic aspects of regulatory decisions. Peer review should provide balanced, independent views. When used well, peer review can serve as a system of checks and balances for the technical aspects of the regulatory process.
Improving the Quality of Regulatory Decisions
Finding
Peer reviews should be conducted both to enhance the credibility of agency decisions and positions and to improve their technical quality. Peer review activities in federal regulatory agencies are generally devoted to evaluating the quality of the toxicologic, epidemiologic, ecological, and engineering data and the credibility of the scientific interpretations that may be used in making a regulatory decision. The quality and interpretation of other technical information, especially related to economic analyses and the social sciences, are generally ignored.
Recommendation
Peer review should play a critical role in evaluation of the quality of technical information used in regulatory decision-making. Peer review of economic and social science information should have as high a priority as peer review of health, ecologic, and engineering information. The primary criterion for membership on peer review panels should be expertise in the area of concern; however, financial conflicts must be avoided.
Peer review of the scientific and economic data presented in the risks and options stages of the Framework (Section 2) is essential for all major rules under development. An open process of sharing the findings and conclusions from peer review can increase the credibility of a risk assessment and stakeholders’ confidence in the conclusions. Peer review might even be useful in the first stage of putting a problem in context, drawing in experienced ecologists, public health officials, and researchers.
The Commission believes that expertise in the technical area under evaluation should be the primary criterion for members of peer review panels. However, potential peer reviewers with financial conflicts should be disqualified from service on peer review panels that could specifically influence regulatory decisions related to the products or interests of their organizations. For example, if Monsanto manufactures a pesticide that is under review by EPA for potential classification as a likely human carcinogen, Monsanto employees or stockholders should not serve as peer reviewers, although they should certainly provide input to the technical analysis that is peer reviewed. If Ciba-Geigy manufactures a similar pesticide, Ciba-Geigy employees or stockholders should not serve as peer reviewers either, because they are competitors with Monsanto. These individuals often have a great deal of knowledge about the subject under discussion; however, and should be invited to share that knowledge in open sessions with the agencies and then, upon invitation, with peer review panels. Other industry scientists without scientists witout such clear financial conflicts would qualify.
Individuals with other kinds of financial interests may serve on peer review panels but must disclose those interests. Academic scientists working in the area of pesticide carcinogenesis may serve as peer reviewers but should recognize and disclose that their inputs to EPA’s decision might have an indirect impact on the nature or direction of their research. Similarly, qualified staff or representatives of environmental organizations that work to reduce the use of carcinogenic pesticides may qualify as peer reviewers, even if they may be perceived as enhancing their employment and their organization’s visibility.
In contrast with financial conflicts, bias reflects views or positions taken that are largely intellectually or socially motivated. It is difficult, if not impossible and unwise, to eliminate bias. The Commission believes that criteria for constitution of peer review panels should include a balance of disclosed biases and inclusion of active, younger, and culturally diverse scientists and economists. Explicit criteria for revealing and evaluating conflicts and biases are needed (see The Conduct and Effectiveness of Peer Review below).
Although economists have very different expertise from toxicologists and epidemiologists (but not bio- statisticians), we recommend a unifed peer reveiw panel, consistent with our recommendation to link assessment outputs and inputs for economic analysis (see page 99).
The person(s) responsible for selecting peer reviewers can have a great deal of influence on the nature and biases of the membership and the expertise represented; consequently, they can indirectly affect the outcome of the review. Those persons can also have a lot of influence on what is peer reviewed. That gatekeeper role should be structured carefully to ensure that biases affect the process as minimally as possible.
The Conduct and Effectiveness of Peer Review
Finding
EPA has a written policy for program-specific peer reviews (EPA 1994). FDA has an established policy for constitution of advisory panels, which function as technical review panels. Some agencies do not have official guidelines or policies for peer review, and essentially none has procedures for evaluating the effectiveness of peer reviews.
Recommendation
Clear, written guidelines for peer review should be established by regulatory agencies, and the effectiveness of agency peer review programs should be evaluated periodically. The extent of peer review should be commensurate with the importance of scientific or economic issues and the regulatory impact of the decision to be made. When peer review is judged to be unnecessary, an agency should provide an explanation and justification.
Official, written guidelines for the conduct of peer reviews help ensure the transparency of agency decision-making and enhance the credibility of agency decisions. Guidelines for consistent peer review procedures can enable agencies to address explicitly the questions and issues that are commonly raised during development of regulations. Administrative features—such as how peer reviewers are selected; which agency problems, risk assessments, regulatory options, or decisions will be subject to peer review; whether and how consistency among an agency’s programs should be improved; and how the outcomes of peer review will be used—should be addressed by an agency’s peer review policies. EPA’s program-specific standard operating procedures for peer review required by its peer review policy (EPA 1994) are good examples of such guidelines.
Peer review policies should also provide guidance to agency staff for effectively framing the responsibilites of peer review panels, which should at a minimum include determining whether all the relevant data were evaluated, whether the conclusions based on those data are justified by the evidence, and whether the conclusions are communicated in a manner that reflects the weight of the scientific evidence.
In some cases, alternatives to traditional peer review panels may be appropriate. For example, while OSHA uses peer review panels for some complex issues, it relies to a greater extent on trial-type rulemaking hearings, that can be quite rigorous. The two approaches should be compared and evaluated against criteria based on agency or cross-agency policies. At CPSC, a formal peer review process is required before issuance of certain rules related to cancer, birth defects, or gene mutations. CPSC employs peer review voluntarily in certain cases with scientific controversy; otherwise, this agency relies on the public comment pracess, usually two comment periods plus a hearing.
Agency peer review policies should also include a regular evaluation process to examine specific examples of an agency’s use of peer review in its regulatory decision-making and ask how the peer review was conducted, whether and how the outcome of the peer review was used in a regulatory decision, whether the peer review was considered useful, and finally, how the process could be improved. The General Accounting Office (GAO) of the Congress recently published a report, Peer Review: EPA’s Implementation Remains Uneven, which evaluated nine major agency work products that had been peer-reviewed under EPA’s 1994 peer review policy (GAO 1996). The uneven implementation of its policy was attributed to confusion among agency staff about what peer review means, what its benefits are, and how and when it should be conducted. In addition, accountability and oversight mechanisms were found to be inadequate. Another agencywide evaluation of the role of peer review is described in the EPA publication Safeguarding the Future: Credible Science, Credible Decisions (EPA 1992b). Evaluations can be organized by the agency, (as EPA does through its Science Advisory Board), across agencies, such as by the Office of Science and Technology Policy or the GAO, by the risk assessment subcommittee of the administration’s Committee on Environment and Natural Resources, or possibly by a rejuvenated Council for Environmental Quality.
Full peer review is unlikely to be needed for every regulatory decision. Implementing a peer review process for every regulatory decision or every step in a regulatory decision would lead to substantial delay and require excessive resources. The most effective and most efficient use of peer review should be decided case by case, taking into account such issues as the extent to which the scientific basis for a risk assessment or economic analysis might be considered controversial, the economic impact that a decision might have, and agency resource constraints. Peer review should not be used as a device to delay controversial policy decisions.
Finding
There appear to be no mechanisms for evaluating the use of scientific and economic information in regulatory decisions.
Recommendation
Advisory groups should be used periodically to evaluate the use of technical information and the results of peer reviews in regulatory decision-making. Advisory groups for this purpose should be composed of stakeholders, including those with financial stakes. Such advisory groups would review the process, not override pending decisions.
Good science can be used to justify bad regulations. Asking whether relevant scientific or economic information was cited appropriately in a particular regulatory process is critical. There appear to be no mechanisms in place that support review of the use of technical information at the policy stage, although scientific advisers to the EPA administrator, the FDA commissioner, or the OSHA administrator may fill that role informally. Most peer reviews evaluate highly focused, technical topics because of the assumption that scientists and economists tend to lack an understanding of the history and philosophy of an agency’s decision-making process. An advisory mechanism for evaluating the descriptions and uses of scientific and economic analysis in the decision-making stage should be developed.
In contrast to members of peer review panels reviewing pending matters, members of advisory groups would be permitted and expected to have conflicts of interest, financial or otherwise. Advisory groups of stakeholders would evaluate the use of the results of peer review in completed cases in the decision-making process as a lessons-learned exercise to support continuous improvement.