This commentary was originally published by The Regulatory Review.
The Arkansas Department of Health Services hired a software company in 2016 to build an algorithm to automate the process of assessing disabled patients’ needs. The program failed spectacularly. For example, the algorithm reduced an amputee’s level of home care because the patient had no “foot problems.” Overall, the program negatively affected nearly half of Arkansas Medicaid recipients.
Artificial intelligence (AI) is now used widely in the federal government. According to the U.S. Government Accountability Office, use cases of AI in the federal government doubled — and for generative AI, increased nine-fold — from 2023 to 2024. With this growing use of AI, the risk of systematic errors with potentially harmful consequences, such as those in Arkansas, may increase. Similar issues have already occurred at the state level in Idaho, Texas, Michigan, and Wisconsin, as well as at the federal level at the Department of Homeland Security.
The Trump administration has reportedly signaled its intent to introduce AI into the rulemaking process. It is unclear whether and how agencies might rely on AI-generated information to inform regulatory decisions. But this development raises key administrative law questions — particularly as horror stories like those in Arkansas become more common.
Legal scholars have long recognized the challenges judges face when enforcing the Administrative Procedure Act’s bar on “arbitrary and capricious” decision-making — which requires that a government agency engage in reasoned decision-making — but these scholars have assumed that the decisionmakers under review are humans. Issues raised by AI may exceed the limits of existing administrative law doctrines. This creates a legal problem under the Administrative Procedure Act, which compels courts to hold unlawful and set aside agency action found to be arbitrary and capricious.
How can judges properly conduct judicial review of agencies’ regulatory decision-making that relies on AI? Observers cannot always determine how an AI model made its decision — the so-called “black box” nature of AI. Even when users prompt an AI model to explain its rationale, there is no guarantee that the reason the model produces is the actual reason the model used. Under the Administrative Procedure Act, judges must set aside arbitrary and capricious decision-making, but the black box issue can make it difficult, if not impossible, for reviewing judges to assess the reasonableness of decisions made using AI. This raises similar concerns to the problem of pretext that administrative law already recognizes, since both implicate the distinction between the stated and actual reason for a decision.
