More thoughts on evidence generation

I thought it may be useful to analyse this paper which does an RCT (with three treatment arms) to evaluate the efficacy of a mobile App that can track payment releases in the Government of India's National Rural Employment Guarantee Scheme (NREGS).

For those unaware, as part of the program, rural labour are guaranteed a minimum 100 days of work every year by deploying them to build community assets. The program's execution consists of village level officials maintaining muster rolls of workers and making weekly/fortnightly payments directly to the worker's bank accounts. The program is hobbled by delays in payment releases to the bank accounts (due to delays in recording of work, updation of muster rolls, approvals of muster rolls etc). The immediate supervisors of the village level officials are those at the block level, who, in turn, are monitored by those at the district level.

The 3-arm RCT involved providing a payment tracking App, PayDash, to only the block level supervisors, only the district level supervisors, and to officials at both levels. It found that only the last option was effective in significantly reducing payment receipt delays.

Did we at all need an RCT to know this? Isn't it evident from the well accepted theories of management? How has this research moved the body of practically relevant knowledge on the use of such applications to improve state capacity in any way? The authors make this claim,

we have limited empirical evidence on the relative importance of asymmetric information at different levels of the hierarchy in affecting bureaucratic performance.

Do we need "empirical evidence" on this? Don't we have enough evidence from theories of organisational behaviour and management to validate the hypothesis? Is that evidence inferior in a meaningful way?

Instead, shouldn't the focus of evidence generation be on figuring out the most effective deployment dynamics of PayDash - its Dashboard interface, the periodicity and nature of reviews by block and district level officials, nature of integration of this information with performance management of officials etc? All of these are amenable to evidence generation, but not of the kind that international development community is associated with!

We need another type of evidence generation - tight operational feedback loops and close integration of the learnings into the operational processes. Coupled with an eye on capturing and monitoring credible enough proxies of impact.