Hawthorne Effect in RCTs

Angus points to this study by a group of researchers that appears to point to significant "pseudo-placebo effects" in RCTs. It can also be viewed as an example of the significant role of expectations in shaping the responses to various interventions.

In two separate RCT experiments in rural Tanzania, farmers were supplied better varieties of cowpea seeds. In the first traditional RCT, the control group were informed and supplied traditional seeds while the treatment group were similarly supplied modern seeds. In the second double-blind RCT, both groups (and experimenters) were unaware as to what type of seed they received, though they were aware of the experiment.

The traditional RCT showed a significant, over 20% increase in yields from the modern seed, whereas, the productivity increase due to seed effect in the double-blind RCT was virtually zero. Most interestingly, a comparison of the two control groups revelead that the production increased in the double-blind control group by as much as the two treatment groups. This is a measure of the pseudo-placebo effect - it captures the crop and harvest differential due to beliefs and associated behavioral responses, not to modern seed.

The researchers found that farmers responded to being supplied with modern seeds by planting in their most productive lands, increasing their plantation areas, and also by maintaining optimal spacing. Each of these, especially the last, contributed towards improving yields.

In other words, "the expectation of receiving the treatment can cause people to modify their behaviors in a way that produces a significant "average treatment effect" even if the actual intervention in not particularly effective". The double blind RCT showed that virtually all of that improvement comes from changed behavior, not from any improved effectiveness due to the use of the modern seed. They write,

The double-blind analysis learns us that the increment in output is not due to the quality of the seeds at all, but instead to a behavioral response of the farmer (who plants the – suspected – modern seeds further apart). Positive treatment effects evaporate when controlling for this. The main conclusion is not to shun RCTs — far from it. However, we should interpret the outcomes of RCTs with proper caution, especially when jumping from output to outcomes and impact. The magnitude of the pseudo-placebo effect depends on the participants’ subjective expectations with respect to the degree of complementarity of the intervention and privately supplied inputs.

They caution against drawing inferences from preliminary findings and advocate repeating the experiments to analyze the changes in responses as expectations evolve,

Biased assessments also occur when participants have the wrong set of expectations about complementarities. To attenuate this source of bias it would be advisable to repeat experiments over various production cycles, allowing participants to learn about the nature of the innovation offered to them. As knowledge accumulates, their response becomes better tailored to the innovation, enabling the analyst to obtain an estimate of the total impact of interventions that captures both the innovation effect as well as the optimized behavioral response.

Aside from the learnings about the interpretation of RCT results, this study is an excellent example of the power of shaping behavioural expectations and responses. It raises the possibility of shaping expectations of a target group - farmers, women, teachers, students, patients, poor etc - by nudging them through creative marketing and awareness creation initiatives. Atleast in certain areas, this has the potential to be cheaper and more sustainable than the convetional hard interventions.