Evidence and evaluations in development

One of the most popular narratives in the development world is the idea of evidence-based policy making. Its most salient application is the idea of independent and rigorous evaluation of causality. Is this intervention creating the intended effect?

The development academia and commentators debate about the methodological and other challenges to evaluations and lament the lack of awareness and interest among bureaucrats and politicians. However, there are three very important points to be kept in mind.

One, when academicians, commentators and donors talk about evaluations, they are generally talking about evaluations of small experimental programs or pilots. Pretty much all the literature about "rigorous impact evaluations" involve these. They are rarely talking about evaluating ongoing scaled up programs. This is despite arguably more than 95% of the development world (in terms of budgets, numbers of programs, number of people working etc) being engaged with ongoing large scaled-up programs. 

But there are serious methodological limitations to doing any meaningful enough causal evaluations of such scaled up programs. How do we evaluate an intervention that seeks to improve student learning outcomes  or improve public health or increase nutrition levels or enhance women's empowerment or equip youth with employability skills by isolating the countless known and unknown determinants? How do we insulate the fundamental efficacy of the idea itself from its implementation at scale by a weakly capacitated state? 

Second, it brings us to the point about new ideas. The assumption is that development world needs to embrace the idea of innovation and technology that has come to characterise the private sector. However, instead of blindly adopting them, they need to be evaluated for their efficacy and then implemented. 

But, as I have blogged earlier, apart from procedural tweaks and technological solutions, there are very few new ideas per se in the field of development that are not known to practitioners at large. In the last thirty years, I cannot recollect having seen new development programs with meaningful enough impact on persistent development problems which have emerged as mainstream and have been transformative. In terms of primary interventions in the major development sectors, the choice set of interventions before governments is to only marginally tweak the ongoing programs or adapt interventions which have been effective in developed countries. There are no great unknown or untried ideas which can help leapfrog universal and persistent development problems! 

The paths to improving student learning outcomes, delivering better primary health care, skilling youth, empowering women, improving nutrition levels etc are and will about bringing together certain inputs and combining them effectively. The challenge has been about combining them. While process tweaks, private participation, and technology are useful, effective implementation at scale is mainly about political economy, stakeholder demand, contextual norms, and state capability. 

Third, even if there are such unknown or untried ideas with significant likely impact, their challenge is with getting implementation right. For example, we can think of ideas like self-help groups and microfinance, community health workers, short-term skilling programs, independent quality and social audits, public private partnerships, teaching to the child, and e-governance solutions that have emerged as mainstream in the last three decades. 

But their causal evaluation in a pilot can offer limited insights and guarantees about scale effectiveness. There is the point made earlier about the implementation challenge. It should be borne in mind that even after hundreds of RCTs about microfinance, we are still no more certain about the headline issues. 

The importance of impact evaluations in development discourse is therefore vastly exaggerated. More than any serious evidence, as I have blogged earlier, the impact evaluation movement in international development has been propped up by philanthropic foundations who've been concerned with the purely reductionist approach of finding the greatest value for their small donations.