More on the contribution of RCTs

A recent working paper of a randomized control trial (RCT) in Orissa to test the effectiveness of improved/clean cooking stoves in reducing indoor air pollution and fuel consumption found that the intervention had no impact. The study which tracked 2600 randomly selected households who were provided the improved stove, over a four year period, finds that,

While we find a meaningful reduction in smoke inhalation in the first year, there is no effect over longer time horizons. We find no evidence of improvements in lung functioning or health and there is no change in fuel consumption (and presumably greenhouse gas emissions). The difference between the laboratory and field findings appear to result from households’ revealed low valuation of the stoves. Households failed to use the stoves regularly or appropriately, did not make the necessary investments to maintain them properly, and usage rates ultimately declined further over time. 

Personally, I find nothing surprising about the study. India's social policy space is replete with hundreds of such examples of failed interventions. Logically and theoretically sound interventions, end up with disappointing results when subjected to the real world implementation test. A few days back, I posted about the failed experiment with automatic parking meters in Hyderabad.

There are two issues here. The first is a technical/technological question - Do improved or clean cooking stoves optimize on fuel consumption and reduce indoor air pollution? The second is a sociological/socio-economic question - Assuming a positive answer to the first question, how effective will be the cooking stove in realizing the two objectives when used in the real-world? More specifically, how would users and other stakeholders respond to the new stove?

Most often, when public policy is designed, the first question is scrutinized pretty rigorously while the second is assumed away. But, it is here, in the second question, that these interventions end up failing. Rightly, the working paper highlights this issue

More broadly, this study underscores the need to test environmental and health technologies in real-world settings where behavior may temper impacts, and to test them over a long enough horizon to understand how this behavioral effect evolves over time.

It is not just with technology interventions that these issues rise to the fore. Policies involving regulatory restraints, incentives, restructuring/reframing of the environment, computerization, and so on, in all sectors suffer from these implementation challenges. Stakeholders' response patterns to an intervention cannot be fully anticipated. The emergent dynamics often detracts from the desired outcomes.     

Therefore I believe that more than providing a new methodology to evaluate the impact of interventions, the more important contribution of the randomista movement to development economics may be to draw attention to the importance of human and social dynamics in determining the success of those interventions.