When the U.S. Small Business Administration issued a study last September claiming regulations cost the U.S. economy $1.75 Trillion in a single year, the agency trumpeted that the "report was peer reviewed consistent with the Office of Advocacy’s data quality guidelines."
But the peer review file included with the study was embarrassingly meager — comments from all of two individuals. The authors, economists Nicole Crain and Mark Crain, ignored a fundamental criticism raised by one of the two reviewers that struck at the very heart of their estimates of economic regulatory costs. The second reviewer's complete comment had the sort of casual quality to it that suggested a somewhat less than thorough review. The review, in its entirety: “I looked it over and it's terrific, nothing to add. Congrats."
When CPR Member Scholars issued a report in February critiquing SBA's study, they noted that the peer-review process was unimpressive. CPR co-author Sidney Shapiro sent a letter at the time to Karen Mills, the SBA Administrator, and Dr. Winslow Sargeant, Chief Counsel of the SBA's Office of Advocacy, calling on the SBA to withdraw the study and disavow its findings. The letter noted, among other problems, the inadequate peer review.
Dr. Sargeant responded with a letter in March, standing by the report and waving off the CPR report’s very thorough criticisms, although without responding to the specifics. There was perhaps little of particular surprise, except one bit about the peer review process. Wrote Sargeant:
Advocacy offered the study for review to a number of individuals with expertise in regulatory cost-benefit analysis, including former heads of Federal regulatory agency economic analysis offices and academics who have published in the field.
For Dr. Sargent, it was apparently just a minor detail that only two peer reviewers responded, as far as anyone knows, and one of the two peer reviewers took issue with the core methodology and the other devoted all of 11 words to a review. Today Shapiro wrote back to SBA, noting that while SBA apparently regards silence as an endorsement by peer reviewers, CPR does not.
Shapiro also reiterated one of the primary criticisms from CPR: the public does not have access to the data, equations, assumptions, extrapolations, and calculations that are necessary to understand and verify the methodologies, data, and assumptions that were employed in report. Shapiro calls on SBA to withdraw its sponsorship of the report until the agency makes the full methodologies, data, and assumptions available to the public, allowing the study to be replicated.
The SBA study, remember, was not privately funded; this was taxpayer dollars at work. The public deserves an explanation.
