The trade-off between academic rigour and relevance

I have blogged several times wondering at why the mainstream development economics research is so busy with what are clearly (to any practitioner) marginal concerns and marginalise methodological tools which are most appropriate to explore critical concerns of policy makers. 

A friend recently pointed me to this very good paper by George Akerlof. He makes the distinction between 'hard' and 'soft' research. The former is defined by precision, quantification, causality (as against mere correlation), and empirical analysis. The latter with qualitative, ethnographic, theoretical, and generally less precise. Akerlof writes,

This article.. advances the proposition that economics, as a discipline, gives rewards that are biased in favor of the “Hard” and against the “Soft.” This bias leads to “sins of omission": in which economic research ignores important topics and problems when they are difficult to approach in a “Hard” way... An academic researcher selects from a set of possible research topics. These topics can be characterized along two dimensions: (1) Hardness (i.e., the ease or difficulty of producing precise work on the topic) and (2) Importance. The researcher values both Hardness and Importance; but the weight he places on Hardness leads him to trade off Hardness and Importance in a socially non-optimal way. In this sense, he is biased.

He attributes this bias to economists striving to present their discipline as a 'hard' science; excessive reliance on mathematics in economics; hardness or precision being easier to assess (when compared to 'importance'), such papers are easier to publish and the researchers involved therefore are more likely to be recruited and promoted. The intrinsic value placed on hardness leads to more biased rewards - negative feedback loop. And this has, in turn, led to bias against new ideas, over-specialisation, tenure and promotion evaluations based on top-five publications (which are in turn biased towards the 'hard').

In terms of recommendations to address this problem, Akerlof writes,

At the journals, the norms for what should or should not be published, and the selection of the editors and the referees, and their conduct, should be the subject of examination. Likewise, at the universities, the processes of promotion and tenure should also be examined. Just as medicine in the United States was famously influenced by the Flexner Report of 1910, there is a need for a similar report today on publication and promotion in economics.

Such a report could be divided into two separate parts. The first part would analyze the norms regarding the role of journal editors and referees. As mentioned earlier, times between submission and acceptance are extremely long, as authors and their ideas are strung out with often repeated requests for revise-and-resubmits according to the tastes of the editors and referees. Returning ownership of papers to the authors would not only show greater respect to them. It would also accord with the stated purpose of two of the Top-Five journals: as the AERand REStud both have the word Review in their name. As I understand it, a “Review” is a journal that takes submissions, and decides which to accept/which to reject. That means that the editors and the referees should be viewing themselves as helpmates, rather than dictators holding authors at ransom before accepting their submissions. A second part of the Report would describe appropriate norms regarding criteria and methods of promotion. Special topics for examination would include the appropriate, and inappropriate, criteria based on publication metrics (such as the number in the Top Five), and, internationally, over-dependence on publication in US journals and even on US data.

This framework is nowhere more relevant than in explaining the rise of RCTs as the dominant means of development economics research.