This (also this) is an RCT study on Aadhaar-based biometric authentication (ABBA) of PDS beneficiaries using electronic Point of Sale (PoS) terminals and the reconciliation of the foodgrain delivered to PDS shops with the data from PoS terminals. The study found "large reductions in leakage, but also significant reductions in benefits received", mostly arising from the reconciliation part.
In an excellent illustration of both the limitations of economic research methodologies and the cavalier manner in which rigorous research is often packaged and disseminated, Jean Dreze, Reetika Khera and Anmol Somanchi write,
Suppose a PDS dealer receives 20 kgs of rice from the government every month, to be distributed to Savitri Devi. If Savitri does not get her rice (for whatever reason) in a particular month, this will show – hopefully – in the ABBA-generated transaction records. In that case, 20 kgs can be deducted from the dealer’s rice allocation in the following month, effectively preventing him (or so it seems) from siphoning off Savitri’s ration. In short, reconciliation helps to curb bogus transactions. Reconciliation sounds like a good idea, especially in a PowerPoint presentation.
In practice, it poses multiple challenges. Just to invoke one, again with an example, consider Olasi, a widow living alone for whom ABBA does not work, perhaps due to rough fingerprints. Before reconciliation, the dealer used to give her rice nevertheless – after all, he was not paying for it. Is he likely to continue giving rice to Olasi after reconciliation? You guessed it. And what if Savitri’s dealer tries to cope with reconciliation by taking a 5-kg cut from Savitri’s monthly ration for a few months? We know that this sort of ‘adaptive corruption’ happened in Jharkhand post-reconciliation, not only blunting the reform but also leading to further exclusion in many cases. As these examples illustrate, reconciliation is not exactly a surefire policy. In particular, it requires an exacting level of preparedness – high performance of ABBA, complete transaction records, an efficient supply chain, cooperation from dealers, and more. We submit that the PDS in Jharkhand was nowhere near the required level of preparedness in July 2017.
Their conclusion is very important,
An RCT is not a guarantee of objectivity. However ‘rigorous’ the evidence may be, it still needs to be interpreted, summarised, and conveyed. It is quite easy for the evidence to get distorted or embellished in this communication process.
To drive home their point, the authors even write down what should have been the actual abstract of the RCT study. The original abstract and that written by Dreze et al conveys it all.
Dreze and Co highlight two things. One, how superficial understanding of context leads to misleading interpretations and conclusions. Two, how researchers spin their experiments to propagate misleading conclusions. This is another similar example I blogged earlier about.
In the context of this debate, I am struck by the proliferation of studies that focus on leakage reductions and efficiency improvements on government programs. The studies add to the narrative of pervasive leakages and the utility of technology in its reduction. The narrative gets more entrenched.
This narrative completely misses the objective of the program itself - ensuring poor people access to food security, through the PDS. Technology is only a means to achieve the objective, though in the process, it can reduce leakages. In other words, the primary role of technology should be to help meet the program objectives, with leakage reduction being only a by-product.
For different reasons, the institutional incentives of bureaucrats have over the years come to be aligned towards efficiency improvements and leakage reductions. Awards and other incentives are aimed at performance that highlight leakage reductions, and not on efforts that maximise coverage of the eligible (minimise exclusion errors). In fact, nobody is even talking about these aspects.
As a research agenda, someone should analyse the Prime Minister's Excellence Awards and see what proportion of them are best practices that involve use of technology and are focused on efficiency, as against those that involve non-technology interventions and focused on quality of service delivery. As another research agenda, researchers should supplement the works of people like Dreze and focus on highlighting the access and quality deficiencies with welfare programs and interventions undertaken to address them.
Undoubtedly things were once very bad. Fortunately, even without (before) RCTs and technologies, things have improved significantly. Egregious forms of corruption and inefficiencies are becoming increasingly marginal and are now confined to a few isolated pockets. This is true of attendance among government officials like teachers or doctors, leakages from in-kind and cash benefit programs, ghost engineering works, and so on.
The egregious nature of the inefficiencies meant that administrative focus was often prioritised on the efficiency dimension - ensuring attendance, limiting inclusion errors and pilferages, and dimensions of roads and irrigation canals or numbers of toilets constructed, and so on. It helped that all these are easily quantifiable and measurable. And technology makes it easier still. Anyways, this was all about getting to the starting line.
Having largely gotten to the starting line, it's now time to refocus attention on the original objectives of programs - whether they are delivering on objectives, the deficiencies, and its reasons. This means a different set of priorities focused more on quality - student learning outcomes and quality of medical care administered; realisation of food security and employment guarantee, especially to the poorest; quality of the buildings, roads, and canals; utilisation of toilets and so on.
This poses a set of problems of assessment - quantification and measurement, and doing them in a non-burdensome and credible enough manner. Unlike the metrics of efficiency, those of quality are not easily measured. In fact, some of them cannot be measured at all, especially for meaningful real-time monitoring at scale, and we need to rely on proxy indicators. In most areas, even all the wonderful technologies fail to contribute anything significant in measuring quality issues.
In fact, I cannot not feel that Aadhaar and IT were perhaps atleast a decade late in arrival in being meaningful in addressing the efficiency objective. And while useful, they are marginal contributors in the larger scheme of things in realising the objectives. Aadhaar and IT have their uses at several other places, but these are not exactly the areas where they add much value.
It's like the difference in approach between moving a system from bad to average, and average to good. The former demands focus on some low-hanging efficiency improvements (even at the cost of objectives and quality) and the latter has to prioritise quality and objectives. And that is much harder.
This applies as much to many other areas of development. For example, consider the narrative on agriculture. It is now excessively focused on the rapacious middleman exploiting the weak farmer. Accordingly, the idea of efficiency enhancing stuff like deregulation and technology innovations (e-NAM) becomes very appealing. But we now know that the maximum gains that can be squeezed out by farmers even by complete elimination of middlemen and direct engagement with buyers is less than 10% of crop value. It has not been technology or innovations, but just better governance and general development that have minimised the egregious rent-seeking and exploitation by middlemen.
The need now is to focus on the basic objective of getting better prices for farmers and connecting them directly to impersonal markets. This in turn demands things like capacity to sort and grade produce, and adherence to contractual obligations by all sides. These are less areas for technology-based reforms and progress measurements.
I think it is important to rebalance the focus of public services delivery and development in general. The pendulum appears to have swung excessively towards improving efficiency and reducing costs and away from quality of service delivery and meeting program objectives (or issues of resilience and fairness). That prioritisation was important for a bygone time when program delivery was very inefficient. I have blogged earlier about how the excessive focus on efficiency distorts perspectives.
For researchers, this reframing of narratives and rebalancing of priorities involves moving away from quantitative and RCT-focused methodologies to mixed methods which combine appropriate quantitative and qualitative approaches.
It is important for researchers to be cognisant that their research priorities are actually creating more damage and distortions than contributing to any improvements. Their headline grabbing inclinations and research are reinforcing an entrenched narrative which is well past its sell-by date.
It is easy and sexy for bureaucrats to pursue leakage reductions. Research on leakage reductions is easily measurable and amenable to RCTs and the likes. In this, both researchers and implementers have become captives of technologies. Technology makes it easier for both to pursue their respective agendas.
There is a much broader lesson here. Narratives, which were relevant for a period, often endure long after the underlying conditions change. There is a very high degree of hysteresis with the process of Bayesian updations among researchers. The persistence then starts to cause damage. One example is that of the enduring narrative that countries should continue with trade liberalisation, despite the very low current baseline of tariffs. But Dani Rodrik and others have shown that we are well into diminishing and even negative returns from any further reductions in tariffs.
Of course, I am not even talking about misleading dissemination of research, a more serious charge laid by Dreze and co-authors.
Update 1 (28.11.2020)
Sonalde Desai and Pallavi Choudhuri write about the extent of exclusion errors from digital transfers,
Recipients of PM KISAN were not among the poorest households, nor were these the households most affected by the Covid 19 lockdown. Data from round-3 of NCAER Delhi Coronovirus Telephone Survey (DCVTS - 3) covering a sample of 3466 households in June in the Delhi NCR region, suggests that 21% of farm households received transfers through PM KISAN. However 42% of those households belonged to the wealthiest one-third of the sample, while another 28.5% belonged to the middle one-third.
No comments:
Post a Comment