This commentary was originally published by Notice & Comment.
Last summer, the Department of Housing and Urban Development (HUD) received a PowerPoint presentation introducing an Artificial Intelligence (AI) tool: SweetREX, named after its creator, a third-year undergraduate in economics. Consistent with the Trump administration’s stated goal of eliminating 50 percent of all federal rules by the first anniversary of President Trump’s inauguration, Elon Musk’s so-called Department of Government Efficiency (DOGE) had used to SweetREX to review more than a thousand of HUD’s regulatory sections and reportedly used it to write 100% of deregulations at the Consumer Financial Protection Bureau (CFPB). These were not simply idle recommendations for human decision makers to consider; rather, the AI recommendations were treated as presumptively correct, with staffers required to justify in writing any disagreement with what the model suggested.
The growing use of AI for such complex governance functions as regulatory decision-making is attracting attention from policy experts and policymakers alike. The example of HUD’s and the CFPB’s use of SweetREX, in particular, illustrates how such situations risk putting two existing bodies of public law—those governing the rulemaking process and procurement, respectively—on a collision course: in particular, rulemaking’s emphasis on transparency and procurement’s emphasis on quick commercial transactions that often do not include robust transparency audits.
The impetus for this potential clash is that the architecture of many of the AI systems that the federal government employs are developed by private contractors. For instance, rather than being developed entirely in-house, SweetREX is powered primarily by Google Gemini’s model. This means that the federal government is unlikely to have a full understanding of how the model was trained.
This gap in understanding is problematic in the context of agency rulemaking decisions because the law that governs these actions—the Administrative Procedure Act (APA)—requires the responsible agencies to offer an accounting of the deliberative process that undergirds their final decisions. Under that law’s “arbitrary-and-capricious” standard, which is used to evaluate the policy rationale for a regulation, the Supreme Court has held that agencies may not rely on factors Congress did not intend them to consider (i.e., “improper factors”). Of course, policing this standard is not easy even when agency decisionmakers are human. Hesitant to probe the minds of agency decisionmakers, courts have turned to the functional solution of evaluating the agency’s decision-making process.
However, AI complicates this analysis due to the “black box” problem. The inability to truly understand how an AI system reaches its decision, including what factors it weighs and how it weighs each factor. If the model relied on an improper factor, the agency that deployed it has effectively done so as well, with no way of knowing it has.
Read the full commentary at Notice & Comment.
