The plumbing of third party audits

This study of third party audits of polluting industrial units in Gujarat is highly acclaimed. It claims to provide evidence that independent third-party audits are effective at reducing environmental pollution.

In brief, in response to a High Court directive, certain types of highly polluting industrial units in Gujarat had commissioned and had been filing thrice-a-year third party audit reports since 1996. Instead of being paid by the firms themselves, once the auditor payments were made from a central pool (money raised by researchers) and audits conducted randomly, there was a significant increase in reporting of pollution readings and reduction in actual pollution itself. In order to strengthen their theory of change, the J-PAL researchers also conducted explicitly announced random sample back-check super audits of each auditor and their payments were made contingent on the accuracy of the original audits (compared to the back-check super audits).

Similar third-party audits of polluting industries are being done by some states, either directly or through outsourced management of emission/effluent discharge sensors. In the circumstances, I struggle to understand the likely incremental benefits of an RCT evaluation of third party audits to monitor pollution. Of course, third party audits will reduce pollution. And perceptive environmental protection officials across states are aware of its utility. 

So, did the research provide anything that was valuable for real world life? In the real-world of weak state capacity, effective management of third-party audits itself is a massive task. The back-check super audits make the task even more onerous. Since the back-checks were conducted under the supervision of enthusiastic and committed research associates, were known to the industries, and auditor payments were made contingent on original audits tallying with the super audits, the original audits could not but not have become high stakes and thereby also high quality. The RCT established the efficacy of this particular double-audit design.

Unfortunately, such a two-level audit, which achieves both high stakes and high quality in this manner, while desirable, is too engagement intensive to stand a chance of effective scale up through weak public systems. If instead the research had proved the efficacy of one-level audits, that would have been of at least some practical value (maybe a technical rationale for an interested bureaucrat to push through the reform).

Like with this, I am puzzled by such research endeavours and the attention that it commands in the academic arena. Third party audits or certifications are today common place in monitoring everything from engineering works to the quality of goods procured and services delivered. In countries like India, over the last decade or so, third party quality audits, ostensibly of random samples and carried out unannounced, have come to be embraced for engineering works executed by all departments, big and small, urban and rural. It has undoubtedly contributed to improving the quality of these works, and where done well the benefits are very significant. 

If the authors were interested in showing that how the auditors are paid shape incentives, they need not have taken the trouble since there is a rich body of literature on the problems with ratings shopping by financial institutions. 

The researchers found at least two channels of incentive distortion with the earlier approach. One, an agency problem arising from the auditors being paid by the audited. Two, the auditors being paid significantly less than would have been required to conducted good audits. 

It is difficult to believe that the Gujarat Pollution Control Board (PCB) did not know what was going on. I can think of at least four first-order plumbing reasons why the PCB preferred to go along with the charade than adopt what were obvious (if atleast because they were being done in other sectors) reforms. One, these reports were being generated at the instance of the High Court and being reported to them. As long as the Court was happy, there was no reason nor inherent motivation for the PCB to change the system. Such pro-forma compliance with regulatory requirements is commonplace in bureaucracies. Two, if the government decided to do the audits, there was the question of who would pay for it, not to mention the “headache” of managing this additional administrative responsibility. Three, there are strong vested interests among the polluting industries that prefer the status quo. And regulatory capture, especially in such high stakes sectors (the difference between business as usual and pollution compliance for these firms is in many cases a matter of survival itself), is always imminent. Four, whether we like it or not, pollution control is a marginal concern for most state governments as they chase economic growth and job creation. To this extent, officials posted in such Departments are unlikely to be those with the greatest interest and enthusiasm to pursue painstaking reforms.

The issues of relevance for practitioners with third party audits are more in the plumbing. There at least a few that come to mind immediately. What should be the most cost-effective design of third party audits? What can be done to mitigate the risk of capture of such audits? How can the audits respond to dynamic expectations? And, how can the audits be financed?

In case of an engineering work, the most cost-effective design would focus on the least number of samples with the longest periodicity that would not compromise on deterrence. As regards addressing capture, inspections may not only have to be random but also done by personnel on rotation, and the agencies themselves may have to be shuffled periodically or multiple agencies employed. As to dynamic expectations, it may be necessary to periodically revisit the audit criteria and calibrate for the adaptations. On financing, it may be required for the PCBs to commission and finance the audits and, maybe, recover a part of the cost as a user-fee (or from the fines collected, though it could have perverse incentives) paid into a common pool. 

Ultimately what the practitioner needs is an administratively simple and workable third-party audit design. Or more specifically, he or she would want a tender document that captures these design specifications. The aforesaid research does not provide anything of relevance on this. 

Let me be clear that the purpose of this or an earlier post is not to downplay the importance of field experiments or economics research in such areas. They have undoubted value. Research, even without out a utilitarian dimension, is valuable. At the least, it provides some basis for a government official sitting on the fence to bite the bullet with independent and/or random sampled audits. 

Readers should note that there is nothing Gujarat or even India-specific to the plumbing issues. They are universal to any developing country with weak state capacity. So many of the plumbing features that could have been tested are generalisable. An opportunity has been missed, and more worryingly, we may not even be aware that these are the real challenges. 

These posts are motivated by my strong belief that field research rarely ever start with a felt-need or felt-problem for the primary stakeholder, most often government officials. While there are also practical difficulties, inadequate comprehension of the real plumbing challenges among researchers is the bigger obstacle to engaging directly with the problem. It does appear that, contrary to this, the best plumbers, outside of actual plumbers, are the practitioners themselves. Plumbing knowledge is a lived experience.