The Great Econometric Illusion

I have shown this graph to colleagues, students and friends, and invited them to describe what it says to them about the relationship between y and x. Most people said one (or more) of the following things:

• There is no relationship between y and x.
• If there is a relationship, it is completely obscured by noise.
• The relationship could be almost anything.

Surprisingly, standard econometric methods say something completely different. They identify a ‘statistically significant’ relationship between y and x, with a slight positive slope of something between 3 degrees and 14 degrees.

This seems too good to be true. How can econometric methods turn such an amorphous scatter of points into a very confident statement about the relationship between y and x? The answer is that econometrics uses a very clever conjuring trick.

Obviously, there is a great deal of noise obscuring the relationship between y and x. Econometricians usually summarise this noise in a variable u, and it is conventional to assume that the noise u is independent of the value of x. This independence assumption plays a critical role. With it, econometric methods can translate this amorphous scatter into a narrowly-defined and statistically-significant relationship.

Many economists and econometricians, however, do not recognise the clever sleight of hand that has been used here. We are persuaded that this independence assumption is plausible because we know little or nothing about u, and therefore have no evidence to contradict the independence assumption. But even if independence is plausible, it is not necessarily the only plausible assumption; indeed, all sorts of hypotheses about the relationship between u and x could be equally plausible.  

Therefore, if we are to allow ourselves to be taken in by this conjuring trick, it is not enough to demonstrate that independence is plausible; we must also demonstrate that any dependence of u on x is implausible. But in most real econometric studies, dependence is just as plausible as independence.

What happens if we accept that the independence assumption is unjustified? In that case, we can no longer use the conjuring trick. Instead, we need to conduct a sensitivity analysis, showing how the estimated relationship between y and x depends on what we assume about the relationship between u and x.

How sensitive is this sensitivity analysis? The answer to that depends on the clarity or otherwise of the scatter-plot of y and x. In simple terms, if we have an amorphous scatter-plot like Figure 1, then the estimated relationship between y and x depends entirely on what we assume about the relationship between u and x. But, if we have a very clear picture like Figure 2, then the estimated relationship between y and x is little influenced by what we assume about the relationship between u and x.

Figure 2

We can make this discussion a bit more precise by introducing a simple measure of the clarity of these scatter-plots. It is called the signal-to-noise ratio, a concept that will be familiar to any hi-fi enthusiast. Most sciences have to make do with a signal-to-noise ratio a great deal lower than that achieved in hi-fi, but there is a general view that when signal-to-noise ratios get below 5, and certainly below 3, then it is very difficult to make progress.

In Figure 1, the signal-to-noise ratio is only 0.15. When the signal-to-noise ratio is as low as this, the estimated slope of the relationship can be almost anything, depending on what we assume about the relationship between u and x. In Figure 2, by contrast, the signal-to-noise ratio is 5, and in this case the sensitivity analysis finds that the estimated relationship between y and x is little affected by our assumptions.

What do we do in the case of a multivariate model, where we want to estimate the relationship between y and x, but recognise that these variables are also related to another group of variables, Z? Happily, thanks to the Frisch-Waugh Theorem, it is still possible to envisage the relationship in terms of the simple graphical approach shown above. Put simply, we extract from y and x any variation that is correlated with Z and use the terms y* and x* to describe what is left. Then we draw the same sort of scatter-plot as above, but this time using y* and x*.

Some econometricians may protest that Figure 1 is atypical, and that Figure 2 is far more typical of econometric work. Unfortunately, such an argument is not supported by the evidence. I examined some 2,200 econometric estimates taken from studies in the top economics journals and found that the median signal-to-noise ratio in these was 0.03. Fewer than 15 per cent of estimates had a signal-to-noise ratio of 0.15, or more. Fewer than half a per cent of estimates had a signal-to-noise ratio of 1 or above, and the highest value was 1.33. None of the studies had a signal-to-noise ratio anywhere near 5.

In short, it is Figure 2 that is completely atypical, while Figure 1 is very typical. Indeed, 85 per cent of the estimates in my sample were based on scatter-plots that are even less clear than Figure 1.

This is the ‘great econometric illusion’. Econometricians praise their technique as the most precise and robust available to economists, which makes other ‘woolly’ and impressionistic research methods redundant. But unless they are absolutely confident that the independence assumption is correct, and that any dependence is implausible, then this veneer of precision and robustness is a pure illusion.

So, what should we do? It is possible that econometricians will make progress in increasing the signal-to-noise ratios in their models, but the increase required is a factor of 100 (from 0.03 to 3). That is an exceptional challenge and will probably only be attainable in quite special circumstances.

To make real progress, therefore, something more radical is needed. We need to follow the example of medicine and create a field of economic anatomy that is to economics what anatomy is to medicine. This field would use direct observation and description of economic relationships, rather than the indirect inference of econometrics.

The great German anatomist, Friedrich Tiedemann wrote that, “Without anatomy, doctors are like moles. They work in the dark and their daily labours are mounds of earth.” ^[1] I think much the same can be said of economics without economic anatomy. The top priority for empirical economics should be a serious study of economic anatomy.

G.M.P Swann, Economics as Anatomy: Radical Innovation in Empirical Economics, Cheltenham:
Edward Elgar Publishing (2019)

[1]   Solow, R. (1983), “Comment by Robert Solow”, in J. Tobin (ed.), Macroeconomics, Prices, and Quantities: Essays in Memory of Arthur M. Okun, Washington DC: The Brookings Institution, p. 281

^[1]  “Ärzte ohne Anatomie gleichen den Maulwürfen. Sie arbeiten im Dunkeln und ihrer Hände Tagwerk sind Erdhügel”, Friedrich Tiedemann (1781—1861), quoted in Prückner (2017, p. 20).