Graphs as a tool for the close reading of econometrics (Settler mortality is not a valid instrument for institutions)
This paper is closed for comments.
Abstract
Graphing causal models while reading econometric papers can make it easy to understand assumptions that are vague in prose and to isolate those assumptions that are crucial to believe the main causal claims. The method is here illustrated with a close reading of the paper that introduced the use of settler mortality to instrument the impact of institutions on economic development. Two causal pathways that invalidate the instrument are found not to be blocked by satisfactory strategies. The estimates in the original paper and in many that have used the instrument since should be considered highly suspect.
This is a very interesting paper. It discusses the problems, the difficulties and the caveats that someone should have in mind in applying econometric analysis to the study of complex economic phenomena. In my viewpoint, it has two contributions. I will start with the more apparent one.
The paper introduces so-called ‘graphical theory’ as a ‘concise way of recording causal assertions’ in conventional presentations of structural equation models. It adopts as a case study an influential paper by Acemoglou et al. (2011), which uses the instrumental variables method of econometrics to support the argument that property rights have been an important causal determinant of economic growth. The paper introduces graphical analysis as a thorough and comprehensive way to follow and discuss whether the adopted instrument is valid. Graphical analysis can potentially support and/or challenge the estimated effects of an assumed causality between economic variables. More importantly, the paper introduces the idea of a ‘fatal graph’ which indicates all the links that need to be blocked or crossed in order for the use of the instrument to be valid. I find this method to be a quite interesting and useful tool for close reading and critical assessment of econometric papers.
For me, the most enjoyable part of the paper is the careful analysis of Acemoglou et al. (2011). I am quite familiar with this literature and I was able to follow the detailed discussion in the paper. For readers with no particular interest in this type of research, the paper is somewhat long, but functions as a thorough but extended case study. As mentioned in the conclusions, scholars with good background in econometrics should be able to spot weakness in the use of instrumental variables. For these people, graphical analysis could be a useful supplement. Scholars with no strong technical background are given an excellent illustration of the point by the paper, but the graphical method cannot offer anything more to them. The paper reads well and is quite informative and educational for the average reader. However, the use of the graphical method cannot replace a lack of econometric knowledge.
The second, rather implicit, contribution of the paper is to indicate the limitations of econometric research in the study of societies, should the latter be understood as complex institutional systems. Perhaps, this might be my interpretation but I consider it as contribution of the paper as well. The paper does great deal to challenge the conventional language (and the conventional practice of authority-building publications) in which causal identification is discussed in econometric research. However, one could also suggest that, even in cases when a bullet-proof instrument is used to estimate the effects of an assumed causal relation, there is still a powerful conventional econometric rhetoric to deal with social causality that is by no means affected. The enterprise of translating social institutions into corresponding proxy measurable variables in order to establish quantitative relations can easily be misleading in the study of social phenomena (Althusser 1990); sometimes, it might be even more misleading than an invalid use of an instrument. For instance, with regard to the case study of the paper, I am sure that many economists will doubt that the ‘institutional quality’ (whatever this term means) could become a single measurable variable to enter a causal relationship. In this respect, discussing the difficulties of using a valid instrument in the context of graphical method, the paper is quite revealing of the immanent positivism underpinning this type of research. Not only does it stress the shortcomings in the statistical testing of causal claims but also it indicates and demystifies the established way of talking about causality in the context of econometric research. The graphical method is only possible in a universe of quantitative inter-linkages between variables: it sketches both the assumptions of causal claims but also the limitations of thinking along the lines of quantitative causal claims.
References
Acemoglu, D., S. Johnson, and J. A Robinson (2001) “The Colonial Origins of Comparative Development: An Empirical Investigation,” The American Economic Review, 91 (5), 1369–1401.
Althusser, L. (1990) Philosophy and the Spontaneous Philosophy of Scientists, London and New York: Verso.
This is an ambitious paper. Its main argument is clear and difficult to disagree with: causal graphs make it a lot easier to specify hypothesised causal relations, detailing causal mechanisms, and making explicit the threats to causal inference. The author applies this idea to evaluate the analysis of Acemoglu, Johnson and Robinson’s influential Colonial Origins of Comparative Development paper (AJR hereon). The paper’s main argument is: AJR don’t use causal graphs; this in turns leads to ignorance at best and obfuscation at worst of an honest discussion of the causal pathways at play in terms of potential threats to causal inference; the robustness checks in the paper do little to ameliorate this consequence; so at best we don’t have the basis to believe, and at worst we should suspect AJR’s conclusions.
The first part of the paper deals with explaining causal graphs, but the bulk of the paper is the negation exercise: if AJR had used graphs, their findings might have changed or disappeared because they would be forced to control for new variables. The paper is thus mainly dedicated to discussing the threats to AJR’s conclusions, placing them quite literally on causal graphs.
The challenge throughout this paper is of limiting the critique to the parts related, and ideally ameliorated, by graphs. For instance, the discussion on language conventions does not strictly fulfil this criteria even though it is very illuminating: Causal inference is one part of econometric practice, which in turn is shaped by conventions in journal writing/publishing. As an example, the paper tells us that the word ‘fundamental’ appears in the opening sentence of AJR, and goes on to explain why the use of this heavy-weight adjective is in fact strategic, chosen in part to quell doubts about the analysis that follows. That’s probably true, but surely is a wider critique of how language is harnessed, usually to explain things but sometimes to ‘nudge’ the truth? Causal graphs might well have made more explicit potential pitfalls in the analysis, but surely that would only increase the dependence on a lingo designed to convince? The argument for graphs stands, or at least should stand independent of this.
The paper is at its strongest when it discusses real threats to AJR’s strategy (as opposed to known unknowns). The point that institutions are more complex is well taken, as is the discussion about why language and cultural practices should not have been ignored by AJR in their exclusive focus on counting descendants of European settlers. This point applies more widely too, and the stakes are perhaps highest when institutions are used in development economics to explain economies and societies.
A second critical weakness the paper identifies is AJR’s focus on adult mortality and interpreting this as the (only) effect of disease on health and economy. The paper points out that the effect of diseases on children are at least as great if not more serious than those on adults, and this effect opens up a clear pathway for a direct influence of diseases on economic prosperity independent of the link through settler mortality. Graphs do help make this sort of discussion explicit, and might even help in becoming aware of this sort of ‘fatal’ argument.
Its also perhaps an important time to be thinking about AJR’s argument and revisiting some fundamental questions about causal inference: AJR is recently quoted approvingly by Angus Deaton (speaking about Acemoglu, and his work on studying how institutions foster or inhibit growth): “He’s a very good example of the way things ought to be going, which is you do history but you know enough mathematics to be able to model it too. Banishing mathematics is not the solution,… …(T)he model is the cross-check on whether you actually know what you’re talking about.” This paper would say that graphs are critical too, helping structure a more plausible model for valid causal inference.
Notes:
The Angus Deaton quote is from the Financial Times: ‘Crash and Learn: Should We Change the Way We Teach Economics?’ Financial Times. Accessed 21 October 2016. https://www.ft.com/content/0dc9b416-8573-11e6-8897-2359a58ac7a5.
I thank you both for your comments, in which I find little with which to disagree.
Sunil Mitra Kumar is certainly correct that the discussion of word usage does not directly support the central point about the value of causal graphs. I included that discussion chiefly because I seek to illustrate the use if such graphs as a tool for critical reading. That is my most original point, as the value of graphs for designing and doing empirical studies has been very well discussed in works on which I build. Thus it seemed right to demonstrate that the use of fatal graphs could be part of a broader close-reading in the tradition pioneered in McCloskey’s Rhetoric. This is not simply a matter of the two techniques (graphical and verbal) belonging to the same class: at several points (although not the example mentioned in the comment) it is the the crucial role played by a particular link in the fatal graph that inspires enhanced scrutiny of the language used to justify that link’s absence. Also, as both reviewers seem to agree, the points made were intrinsically interesting, which I think is a strong reason for keeping them in. If, however, it appears that to most readers some of this would act to distract, it can be trimmed.
Let me add that I quite agree with the remarks of Dimitris Sotiropoulos to the effect that rather a lot has already been granted when we allow something like ‘institutional quality’ to be treated as a ‘variable’, as though it were as straightforward to define as “anual iron ore production’. I don’t think this remark was intended at all to cast doubt on the value of the analysis. In any case, one cannot dismiss such analytical fictions a priori — one must see what people do with them — and in this case, thousands of papers have made use of the fiction, which is surely reason enough to ask whether what they are doing is logically sound. But as I said, I do not think the reviewer disagrees.