WARNING: Individual Research Findings and Economic Models May Not Be Fully Grounded

Reposted from Legal Planet.

A couple of weeks ago, a major paper on the economics of government deficits turned out to have huge flaws. Matt Kahn and Jonathan Zasloff have already had something to say about this, but I’d like to add some thoughts about the implications for environmental issues.“Interesting,” you say, “But what does that have to do with the  environment?”  

I see two big lessons.  The first lesson is about the danger of overreacting to a dramatic research finding, especially when you really want to believe it because it confirms what you thought all along.  The second lesson is about how little economists know about the functioning of the economic system as a whole, as compared with their understanding of how individual pieces of the economy work. This is really important for large-scale issues like climate change.  I’d suggest use of the warning on the left by journals in the future. More about all of this after the jump. 

The paper in question purported to show that there’s a kind of deficit cliff — when government debt hits 90% of GDP, the bottom drops out of economic growth.  As a new paper showed, that finding had fatal flaws.  Due to a spreadsheet error, five countries were left out of the analysis.  Also, the results were pretty much driven by a single bad year in New Zealand, when government debt was very high and the economy was doing very badly.  (This was partly because the researchers only included that one year out of New Zealand’s history, maybe due to data availability, and also weighted each country equally no matter how many episodes of high debt they had or how they lasted). An additional problem is that the paper appeared in the American Economic Review, a very distinguished, peer-reviewed journal — but it turns out that the specific issue containing conference papers isn’t peer-reviewed, unknown to many of us.

The original paper created a huge furor, getting especially heavy play from politicians, journalists, and bloggers who were already deficit hawks.  ”At last,” they must have thought, “Scientific proof that our fears are justified!”  It’s easy to make fun of this, especially if you’re a Keynesian and thought they were wrong all along. But who isn’t prone to seize on evidence that confirms our strongest beliefs?  For some, it may be evidence about the evils of government spending, for others, evidence about melting ice sheets.  We all have the same impulse to embrace such evidence, and we just have to continually remind ourselves that any one paper is only one piece of the mosaic.  (You should be especially worried when, as in this case, the authors themselves hype the results.) What Larry Summers says about economics research applies more generally:

In the future, authors and journals and commentators need to devote more effort to replicating significant results before broadcasting them widely. More generally, no important policy conclusion should ever be based solely on a single statistical result. Policy judgments should be based on the accumulation of evidence from multiple studies done with differing methodological approaches.

That’s one advantage of the IPCC process — it’s conservative and may tend to understate risks, but it does represent a reliable synthesis of many different papers.  Unfortunately, there’s nothing like the IPCC in the economics realm.  This is one of many areas in which climate science, with all of its problems, is well ahead of economics.

There’s another important lesson to be learned.  As it turns out, the empirical evidence about the relationship between debt and growth is spotty.  There does seem to be statistical correlation between higher national debt and lower growth as you move from debt under 30% of GDP to debts above 100% of GDP.  But the relationship isn’t dramatic and there’s a huge amount of variation.  The evidence doesn’t prove whether there’s a general causal relationship in one direction or the other, or whether different situations (such as severe recessions) have very different debt-growth dynamics.  During the postwar period, have high debt levels caused lower growth, sometimes or usually?  Some economists seem to me to have much stronger arguments than others, but nobody has real proof.

Yet this is a much easier question than how climate change will impact the economy or whether the economic growth of the past couple of centuries will continue indefinitely.  First, we’re not looking to the past for evidence of economic behavior, we’re looking well ahead into the future.  Climate scientists test models by feeding in 1900 data and seeing how well their models “predict” the 20th century, but economists rarely if ever do so. And for good reason: our knowledge of how the economy works is nowhere near as good as our knowledge of how climate works (and neither one, of course, is nearly as good as our knowledge of how electrons and photons work.) .Second, climate change is a new phenomenon, so we don’t have direct evidence about its economic impacts on modern economies. Third, we don’t understand economic growth very well.  (For instance, the Chinese are seemingly doing many things wrong — much of the economy is socialist; property rights are poorly defined; IP piracy is rampant, the rule of law is weak.  Yet their economy is growing like gangbusters.)  Cost-benefit analysis of conventional pollution regulations can be hard enough, but we should be especially wary of economic studies that try to predict economy-wide outcomes or that involve long-term future trends.  The outcomes of such analyzes should be treated as plausible guesses, not reliable predictions.