Two blindspots on evidence and evaluation in development

It's frustrating to hear opinion makers talk about evidence-based policy making and impact evaluations with limited understanding of policy making. They construct straw-man development problems and contexts, draw inferences and rationalisations, and make recommendations about what and how development should be done. 

I want to use this post to highlight two blindspots among development opinion makers on evidence and impact evaluations.

1. There is an important distinction on what constitutes impact evaluation for different stakeholders. When governments talk about impact evaluations, they are talking about evaluating ongoing programs (to improve design if perceived impact is less). However when bilateral/multilateral donors and philanthropists talk about impact evaluations, they are most often talking about rigorously evaluating a (mostly new) idea or project that they plan to fund. 

There is a big difference between the two, especially in their methodological complexity. It's far easier to rigorously evaluate a new idea in a pilot than it's to evaluate to reasonable satisfaction the impact of an ongoing government program. Any evaluation of an ongoing program is immensely complex, and involves a heterodox toolkit that includes procedural compliance, intermediate outputs, and good proxy outcomes. This gets limited attention.

In practice, more than creating evidence on new ideas/innovations, a greater role for evidence generation and impact evaluation would be to limit harm - eliminating or changing elements of an ongoing program or limiting the damage from an ongoing program. It's rare to have a program itself killed off purely because of evidence. 

2. The above distinction segues into an important problem about the conventional wisdom on development. Contrary to what's widely believed in international development which gives disproportionate importance to new ideas, innovation, technology etc, the landscape/space of development interventions is mostly well-known. There is little unknown or to dispute (barring a few) about the WHAT (list of interventions). It's the implementation of the program that's generally the problem - HOW do we effectively execute the WHAT?

In other words, given the nature of development interventions, meaningful impact is less about innovations and new ideas and mostly about doing certain universal and basic interventions right. The limited space for innovations and ideas is also mostly confined to execution strategies. 

Yes, applications of technology have a role to play, but except for a handful of use cases, its impact is mostly marginal and challenging to realise at scale. The overwhelmingly major part of effectively running schools and hospitals, skilling and nutrition programs, and delivery of subsidies and social safety nets is in getting its implementation governance right. 

I think the main source for application of evidence in this endeavour (of getting governance right) comes from analysis of administrative data. In this backdrop, impact evaluations therefore are mainly about administrative data analysis complemented with surveys, ethnographies and other qualitative analysis. Evidence here is required to iterate and get the process and elements of the design right, and less to establish any headline efficacy. 

It's important to be cognisant of these two insights if impact evaluations are to evoke meaningful responses among governments in developing countries.

This, by the way, I think is a good example of the incentive distortions in international development. Nonprofits posit new ideas/innovations to funders, who demand evidence of headline impact, which in turn are captive opportunities for researchers to conduct experimental evaluations. A whole disconnected ecosystem gets perpetuated.