The counterfactual problem in public policy

Heads I win, tails you lose! This aphorism could well describe the debate on many intractable public policy issues, those where conclusive answers are difficult to come by. Supporters claim that it would have been worse without the intervention. Critics denounce the intervention as a failure since the problem persists. The challenge with all such issues is the difficulty of establishing the counterfactual. Let me illustrate this dilemma with three examples.

The most famous counterfactual problem of our times is the debate on the impact of expansionary policies implemented in the US in the aftermath of the Great Recession. Conservatives point to the persistent high unemployment rates and weak economic conditions, despite the extraordinary fiscal (more than $ 1 trillion) and monetary expansion (zero bound rates and $2.3 trillion QE), as conclusive proof of the failure of expansionary policies.

They reinforce their argument by pointing to the failure of the now infamous recovery projection, estimating future unemployment rates with and without a stimulus plan, made in January 2009 by Christina Romer and Jared Bernstein, then part of President Barack Obama's team. Their way-off-the-mark estimates suggested that unemployment would approach 9% without a stimulus, but would never exceed 8% with the plan.



In May 2011, using the latest figures available from the BLS, the unemployment rate reached 9.1%. In contrast to the Romer and Bernstein projections which estimated that the unemployment rate would be around 8.1% for May without a recovery plan, or 6.8% with a stimulus plan, the actual rate was 9.1%. The actual unemployment rate has been consistently above Romer and Bernstein’s worse case scenario for the economy – and by a considerable margin. Critics of the stimulus invoke this as proof of its complete failure. After all, though a massive and unprecedented monetary and stimulus was enacted, it appears to have had no impact in terms of improving the economic conditions.

Supporters of the stimulus in turn point to other statistics to put forward their claims about how the stimulus created employment, supported the poorest, propped up aggregate demand, and helped local governments. They argue that in the absence of the stimulus measures, the counterfactual, the economy would have plunged into a full-blown depression.

Further, economists like Paul Krugman have consistently held that the actually enacted stimulus policies have been severely deficient and have been advocating much larger doses of expansion to mitigate the high unemployment rate. In the absence of the required magnitude of expansion, they claim, it is unfair and incorrect to blame the expansionary policies for the economy languishing.

Such counterfactual problems are pervasive in economic policy making. This is especially so given the impossibility of localizing and quantifying the impact of specific policy interventions. In the circumstances, if the intervention fails to yield the desired result, critics will denounce it as a failure. Supporters will find that establishing the counterfactual, the scenario in the absence of the stimulus, is fraught with insurmountable difficulties.

Another example of such analysis is the debate about the benefits of metro-rail in New Delhi. Critics argue that despite the massive investments in the Metro, the Delhi traffic remains as bad as ever, even worse. This argument is made on the assumption that the Delhi Metro was set up with the objective of lowering traffic congestion in the city. Now that the final outcome shows no signs of traffic improvement, they argue, the Metro project has failed.

Supporters naturally point that without the Metro Delhi would been uninhabitable. They argue that the Metro has taken 1.7 million people out of the roads, and thereby ensuring that those many people stay out of city roads. They argue that the success of the Metro is a function of how many people it is able to attract and how fast its network expands. The persistent congestion is only a reflection of the fact that the Delhi traffic has been growing at a pace faster than even the growth in the Delhi Metro traffic.

Such criticisms are commonplace with infrastructure investments. They most often fail to produce tangible and immediate impact, and leaves all stakeholders unsatisfied. When the power deficit is a few gigawatts, the commissioning of a few hundred megawatts of power generation capacity has limited impact on the load-shedding situation. Similar situation arises with even major new water and sewerage treatment capacity expansion, since the requirements are massive. The problem is most acute with transportation, since traffic always appears to worsen. In the absence of any salient impact, municipal councils have no incentive to sanction scarce resources in such sectors.

Finally, the left-wing critics of economic liberalization in India point to the persisting high poverty rates and social deprivation and blame it on the neo-liberal policies of the past two decades. They argue that these policies have exacerbated social tensions, widened economic inequality, dismantled social and economic protections and therefore weakened the nation economically.

This too is a classic counterfactual problem. There are two issues here. One, serious commentators question the nature and extent of liberalization undertaken by successive governments, claiming that they have been too little and limited in scope and piecemeal and stop-start. In the absence of, leave alone the full breadth and scope, atleast even some reasonably acceptable level of liberalization, they argue, how can we blame liberalization for the current state of affairs?

Second, they argue that in the absence of this limited economic liberalization, the economy would have been in doldrums. They point to the undoubted macroeconomic gains of recent years as proof of this. How do we know what would have the state of affairs in the absence of the liberalization policies? See Ananth's excellent take on the critics of economic liberalization, including on other dimensions.

In all three cases - stimulus measures in the US, Metro railways in New Delhi, and economic liberalization in India - there is a classic cognitive bias at work, availability bias. People observe salient outcomes - the poor state of the economy, despite the stimulus spending; poor state of Delhi traffic, despite the Metro; and the persistent high poverty levels, despite economic liberalization - and conclude that these interventions failed to achieve the outcome. However, the reality clearly (albeit less so clearly in case of stimulus) points to all having had considerable effect in mitigating the respective problems, though the exact magnitude of their impacts is difficult to quantify.

Then there is another issue here. In all three cases, the opponents frame the debate by equating the particular intervention with the text-book case of the underlying concept. Accordingly, for example, they define the stimulus as was implemented in the US was the classic Keynesian stimulus, and therefore its apparent failure to get the economy out of the recession is conclusive evidence of the failing of the underlying Keynesian concept itself.

Similarly, critics' definition of the success of metro rail as measured by the resultant reduction in congestion rate, means that an actual increase in congestion is taken as proof of its failure. For neo-liberal critics, Manmohanomics is the embodiment of economic liberalization and since it did not "eliminate poverty", as promised, it has failed!