Calibration of Expert Judgment in Exposure Assessment: Personal Exposure to Benzene. K. D. Walker, P. Catalano, J. Hammitt, and J. S. Evans, Harvard School of Public Health, 665 Huntington Avenue, Boston, MA 02115
The recent movement of regulatory agencies toward probabilistic analyses of human health and environmental risks has focused greater attention on the quality of the estimates of variability and uncertainty that underlie them. Of particular concern is how uncertainty — a measure of what is not known— is characterized as it can play an influential role in decisions about the value of exposure research or of regulatory controls. When adequate amounts of relevant data are available, classical statistical methods for characterizing uncertainty may often suffice. More often than not, however, data are neither abundant nor directly relevant making it necessary to rely to varying degrees on subjective judgment. Since the 1950’s, methods to elicit and quantify subjective judgments about such uncertainties have been explored but have rarely been applied to the field of environmental exposure assessment. This study, conducted as an element of the National Human Exposure Assessment Survey (NHEXAS), had two phases (1) the application of subjective judgment techniques to the characterization of uncertainty in current estimates of environmental exposures to benzene in EPA Region V and (2) the calibration or assessment of the quality of those judgments using measurements of the same quantities from the NHEXAS field study in Region V that have recently become available. In the first phase of the project, reported on last year, seven experts in benzene exposure assessment were selected through a peer nomination process, participated in a 2-day workshop, and were interviewed individually to elicit their judgments about the distributions of residential ambient, residential indoor, and personal air benzene concentrations experienced by the non-smoking, non-occupationally-exposed population of U.S.EPA Region V. Specifically, each expert was asked to characterize, in probabilistic form, the arithmetic means and the 90th percentiles these distributions.
The second phase, the subject of this paper, has focused on the quality of the experts’ judgments — that is, how well they have been able to predict the findings from the NHEXAS Region V study. Two calibration measures were used: 1) surprise and interquartile indices and 2) a quadratic scoring rule in which two important attributes of the judgments, accuracy and expertise, are scored separately. The experts’ performance will be discussed and compared to the body of evidence on expert judgment in the literature. The study also examines the dependencies between and within the experts’ judgments and evaluates the impact of intra-expert dependencies on the significance of the calibration results.
Go to . . .