More on the evidence-generation industry in development economics

Lant Pritchett points to the example of this RCT which shows that "reducing proximity to schools increases enrollment for boys and girls, increased enrollment leads to increased learning and the effect was differentially larger for girls" and describes such evidence generation as "feigned ignorance". 

I have blogged here and here about a big problem with academic research and the evidence-based policy making movement in international development. One of marginalisation of priors and the emergence of evidence as an ideology.

Development tourists fly-in, observe a problem, imagine/invent a solution (read this), then try to generate evidence for the same so as to attract funders and scale the solution. Never mind, no such invention has ever reached scale.

As if not happy with such success, there is new direction of emerging research. Again the same tourists find something intriguing (sometimes it is stuff which is commonplace in their own countries, like this), formulate a theory of change or hypothesis, try to generate evidence to attract funding for scale-up. Never mind that the natives have been using the same for centuries or decades, and nobody there seriously dispute the hypothesis.

Both are held-up as examples of evidence-based policy making. It begs the question. Evidence for whom? And for what purpose? And the answers appear to be - for the outsider, and for research publication.  Or to meet the bureaucratic requirements (what is the evidence?) of the donor. Not for those living with the problem or the solution, nor for those practitioner trying to address the problem or scale-up the solution.

One exhibit, forwarded by a friend, is this paper which discovers that threat of disconnections of utility services is effective in enforcing bill payments, and that it is superior to soft-encouragement that merely informs tenants about their delinquency.

Public utilities afraid that service disconnections will have political consequences are reluctant to enforce payment with service cutoff. We test this hypothesis using a field experiment in the slums of Nairobi with two interventions intended to improve repayment for water and sewage services: a soft encouragement that informs tenants about landlord’s payment delinquency and, second, a hard threat of disconnection for nonpayment with enforcement if landlords do not pay. While we find no effect of the soft encouragement intervention, we find very large effects of the disconnection intervention on repayment. Moreover, there seems to be no effect on landlord and tenant perceptions of utility fairness or quality of service delivery, on community activism, on the relationships of tenants with their landlords, or on child health... These results suggest that strict enforcement through disconnections increases payment and the financial position of the utility without incurring political costs.

Did the effectiveness of disconnections and its superiority to soft encouragement really require any evidence at all? Also, can any experiment, howsoever rigorous, convince any practitioner that disconnections don't bring political costs? Leave aside the ethical concerns with "studying" disconnections. 

Another exhibit is this paper on footbridges. What was the need to evaluate the value of footbridges in remote areas? 

Sample these revelations,

Floods decrease labor market income by 18 percent when no bridge is present. Bridges eliminate this effect. The indirect effects on labor market choice, farm investment and profit, and savings are quantitatively important and consistent with the predictions of a general equilibrium model in which farm investment is risky and the labor market can be used to smooth shocks. Improved rural labor market integration increases rural incomes not just through higher wages, but also through these quantitatively important indirect channels.

If you go to the remote interiors anywhere in the world which has a forested terrain and is criss-crossed by rivulets and streams, one of the primary demands of villagers living in isolated small habitations are footbridges to cross the streams. In rainy seasons, when the streams are full, the villages get cut-off from the outside world for weeks/months, and the villagers suffer badly. 

Did we need evidence to show that footbridges are a useful thing? Is qualitative evidence (or self-evident realities) about the suffering of the people not enough to make the case for footbridges, and there is a need for a rigorous quasi-experimental study on labour incomes? Do we need evidence to show that "farm investment is risky and the labor market can be used to smooth shocks"? Or that rural market integration has "indirect channel effects"?

Isn't this all so plain obvious? Clearly not for the two development tourists who were the PIs in this paper.

It is the same naivety or self-centredness that drives people into wanting to test the efficacy of public spending on rural roads and rural electrification! Imagine if Eisenhower had researchers using the logic of value for money (from partial equilibrium analysis) to question building inter-state highway system (as against spending on welfare or even local roads).

Like someone from developing country demanding rigorous evidence to be convinced that Londoners, including the well-off, use public transport, or use bicycles, or normally buy breakfast from Prets (and not make at home).

In case of the footbridges paper, I guess the methodological neatness arising from the naturally available dataset explains the Econometrica publication. But its natural extension to the serious pursuit of international development is a travesty.