Cost-Benefit Analysis and Deep Uncertainty

This post was originally published on Legal Planet. Reprinted with permission.

Since 1981, cost-benefit analysis has been at the core of the rulemaking process. The Office of Information and Regulatory Affairs (OIRA), the so-called “regulatory czar” in the White House, must approve every significant regulation based on a review of its cost-benefit analysis. But cost-benefit analysis has had a major blind spot. It embodies techniques for analyzing possible harmful outcomes when the probability of those outcomes can be quantified with reasonable confidence. When those probabilities cannot be quantified (“deep uncertainty”), the analytic path is more difficult. This issue is especially important in the context of climate change, given the potential for tipping points to produce disastrous outcomes.

Decision science does not yet provide consensus solutions to the analysis of uncertain catastrophic outcomes. But it has advanced beyond the vague guidance provided by the White House Office of Management and Budget (OMB) since 2003, which may not have been state-of-the-art even then. In a recent paper, I survey these developments and explain how they might best be incorporated into agency practice in considering possible catastrophic risks.

The paper draws on the existing regulation for treatment of uncertainty in environmental impact statements, as well as the developments in decision science, to recommend new language for White House guidance to agencies. Agencies and courts have considerable experience in implementing the environmental guidelines, which should assist them in this other context.

For those who are interested, the language is pasted at the end of this post. The specific recommendation is less important, however, than the need to spotlight areas of deep uncertainty, rather than making them afterthoughts in regulatory analysis.

Proposed Guidance to Agencies on Unquantifiable Uncertainties

When the likelihood of an outcome cannot be quantified with confidence, you should provide (1) a statement that the likelihood of the outcome cannot be quantified with reasonable confidence;  (2) a statement of the relevance of the outcome to assessing the proposed regulation; (3) a summary of existing credible scientific evidence which is relevant to evaluating reasonably foreseeable impacts relating on the costs or benefits of a proposed regulation, and (4) the agency’s evaluation of such impacts based upon theoretical approaches or research methods generally accepted in the scientific community. For these purposes, ‘‘reasonably foreseeable’’ includes outcomes involving catastrophic consequences, even if their likelihood of occurrence is considered low, provided that the analysis is supported by credible scientific evidence, is not based on pure conjecture, and is within the rule of reason. (Note: This language is adapted from the Council on Environmental Quality’s guidelines for environmental impact analysis, which is used as a template because it is familiar to many agencies and well understood by courts.)

When an analysis of such an outcome is considered necessary, the agency should consult OMB about potential analytic approaches. The goal of the analysis should be to provide as much guidance as possible to decision makers about the significance of the potential outcomes in question. At a minimum, the analysis should provide a description of alternative scenarios. For instance, in assessing the potential outcomes of an environmental effect, there may be a limited number of scientific studies with strongly divergent results. In such cases, you might present results from a range of plausible scenarios, together with any available information that might help in qualitatively determining which scenario is most likely to occur. (Note: This paragraph is adapted from OMB Circular A-4.)

In developing your analysis, you should consider different methods of scenario development and the potential for applying techniques from decision science in analyzing the import of possible scenarios. You should reference best practices in scenario design and explain the reasons for your choices. You should also discuss how you will assess the outcomes of set of scenarios. Along with qualitative assessments, you may also want to consider techniques including the use of methods such as α-maxmin, robust decision making, and safe minimum standards (among others). Whether you choose to use a more formalized decision method or one that relies on more qualitative judgments and best professional judgment, you should bear in mind whatever guidance the governing statute provides about decision making under uncertainty, such as language requiring an adequate margin of safety.