Economics and Research Assessment Systems
This paper is closed for comments.
Abstract
This paper seeks to analyse the effects on Economics of Research Assessment Systems, such as the Research Assessment Exercise (or RAE) which was carried out in the UK between 1986 and 2008. The paper begins by pointing out that, in the 2008 RAE, economics turned out to be the research area which was accorded the highest valuation of any subject in the UK, even though economists were then under attack for failing to predict the global financial crash which had occurred a few months earlier. One aim of the paper is to explain this economics anomaly in research assessment. The paper goes on to point out a key difference between economics and the natural sciences. Most areas of the natural sciences are dominated for most of the time by a single, generally accepted, paradigm, whereas there are always in economics different schools of thought which have different and highly conflicting paradigms. Given this situation, it is argued that the effect of research assessment systems in economics is to strengthen the majority school in the subject (whatever that is), and weaken the minority schools. This conclusion is supported by empirical data collected by Frederic Lee for the UK. It is then shown that the greater the dominance of the majority school, the higher the overall valuation of the subject is likely to be, and this is used to explain the anomaly noted earlier. It is argued that research in economics flourishes better in a situation in which there are a number of different schools treated equally, than in one in which a single school dominates. The conclusion is that research assessment systems have a negative effect on research in economics and give misleading results. Instead of such systems, an attempt should be made to encourage pluralism in the subject.
This is a very interesting paper with some strong arguments against the RAE. I am broadly in agreement with the author. However, Donald Gillies is stretching his point somewhat to say that research in economics as a whole must be over-ranked because of the failure to anticipate the global financial crisis. Isn’t that similar to saying that medical research as a whole is over-ranked because medical researchers failed to predict mad cow disease (BSE). Many economists work in areas far removed from those leading to the GFC, and they may be (but not necessarily are) doing excellent research.
It can also be dangerous to praise those economists who did “foresee” the GFC, in that, at any particular time, there are likely to be some economists predicting any of a range of outcomes, with most of them being wrong. If, at some time, one is right, that does not necessarily indicate particular perspicacity. However, Gillies does have a point that approaches that preclude the possibility of a GFC are flawed. Given that no single perspective has a monopoly on the truth, plurality is to be preferred. At the least, this means that a wider range of questions and answers may be aired. Selecting from these, ex ante, may be difficult, however.
On the citation measure and Alberto Baccini’s alternative viewpoint, more members does not necessarily result in more citations per published paper. If all authors include the same number of citations per paper (contrary to Gillies’ presumption), then papers authored by someone from a large group will be cited in a smaller proportion of the group’s papers. I think this is Baccini’s point, although it is described as those in a small group would “cite each other more frequently”. Presumably this can be observed by considering numbers of citations in papers of each group. However, Fredric Lee’s point about all citing mainstream papers (footnote 8) is also relevant, just as in the case of minority groups which have to acknowledge the dominant culture, but not vice-versa.
For the discussion on “Diamond List” journals, note:
“It is arguable that the use of the perceived quality of a journal as a proxy for the quality of a paper is inappropriate. The undeniable quality of a great journal is based on many great papers that have been published in the past. However, a great journal is not an accurate reflection of the quality of a recently published paper, or of a paper that is yet to appear in the journal, especially when the paper has received few if any citations to date.” (Chang, McAleer, & Oxley, 2011, p. 327)
I support the paper’s call for pluralism in economics research. However, I am not sure that pluralist policy advice would necessarily result in better policy decisions. This is not the same as, ex post, being able to point to the advice that should have been followed. Nevertheless, pluralist research does at least lessen the likelihood that any one body of knowledge would be seen as “the truth”.
Chang, C.-L., McAleer, M., & Oxley, L. (2011). WHAT MAKES A GREAT JOURNAL GREAT IN ECONOMICS? THE SINGER NOT THE SONG. Journal of Economic Surveys, 25(2), 326-361.