On the progress Vs stagnation debate

A topic of intense debate in recent times is on the future of technological progress and human development. The optimists argue that thanks to technology we are at the cusp of a world with unlimited possibilities. The pessimists feel that we are in a long period of stagnation where all the low hanging fruits have been harvested. 

A related, but more enduring debate, is one on the apportionment of credit for innovations and inventions. The narrative of modern capitalism, especially that of technology start-up fuelled innovation, credits extraordinary individuals and corporations having created paradigm redefining ideas, products, and services. In contrast, others point to Newton and claim that innovations are inherently accretive, standing on the shoulders of giants. 

In this context, Rohit has a nice essay which examines these issues and argues for a balanced middle-path. 

He starts of with how most of the now-simple ideas are a deceptively complex accumulation of pieces,

Since early gunpowder needed oxygen for a proper flame, arrows in the air were as close to useful as they got. To become good enough for rockets you needed better gunpowder formulae. For that you needed better knowledge of chemistry and materials to build fire lances, and more. It took many more iterations before hand cannons became usable enough in combat.. Similarly, what does it take to make bicycles? Is having a spoke and wheel enough? A padded seat? An axle? Still no. If you want it to be in reasonably continuous operation, the trick is to overcome friction, which like most things to do with Newton is relentless. There needs to be ball bearings, something that would take a hell of a lot more innovation to create. Thinking "oh the bicycle is a simple machine" is to elide the entire world of complexity that's subsumed within each square inch of its frame. And that's without taking into account its materials of construction... technological convergence helped Wright brothers make the first plane fly, because it relied on aluminium, aluminium molds.

Accordingly, he writes about the combinatorial nature of innovation and progress,

I suggest that innovation which leads progress is itself caused by combinatorial growth in its sub-domains, which first naturally creates accelerated growth, and later slowdowns as a result of a slowdown in any individual part or an increase in combinations to be tried... The fact of the matter is that the general purpose technologies that we built, and which had disproportionate impact across all categories, they're few and far between. Once we built steel, it impacted everything. Once we invented electricity and the laws of electromagnetism, it affected everything. But getting to the benefits of both also required advances in chemistry, in manufacturing the right compounds, in thermodynamics, in automobiles and motors, and so much more, not to mention the advances in humanity-organisation-techniques that we were forced to invent along the way. They all reinforced each other in a positive feedback loop. Advances in one is what led to advances in the other.

His conclusion,

1. Progress seems to come primarily from innovation and technology
2. A Kuhnian paradigm shift or the emergence of a new breakthrough innovation is a result of a combinatorial exploration of existing technologies 
1. Technological innovation in any one field shows up in the form of an S curve
2. Several S curves add together to create exponential growth
3. To break through requires the innovation to be above a certain threshold, which itself increases over time as we make more and more discoveries
3. Breakthrough innovation comes from the addition of novelty to existing paradigms - this requires a "search" of the actual technological frontier, which keeps expanding
4. Depending on the level of progress of each S curve, occasionally you'll have periods of stagnation while the search for right combinations is ongoing 
1. If the number of existing fields is too large, the search can take longer, while we wait for the right combinations and the right level of progress amongst the component technologies
2. Bridging that gap requires sufficient progress in more individual technologies, and more collaboration amongst technologies
3. This might feel like a stagnation, but that's only because without the emergence of new paradigms or new breakthrough innovations, we're at the far end of an S curve, where you need exponential inputs for linear outputs.

... The story is that we have moved from a world where ideas are getting harder to find, but we discover new sources of ideas anyway, to a world where ideas are getting harder to find and we discover new seams to mine for ideas more sporadically. Occasionally the progress might cluster when a general purpose technology reaches critical mass, but that is not predictable either. Either you wait until a new GPT reaches maturity, when the steep drop in per unit cost makes it applicable to every other sector. Or you wait until the combinations of existing technologies can be applied to new sectors, which gets more complex as the number of combinations to try increases

This view of progress in turn calls for intellectual humility and balance,

Productivity of a process, whether that's startups trying to expand or scientists inventing the next big thing, are subject to the law of diminishing returns. They will find that initial traction is easy while subsequent traction gets harder and harder. There's a big difference between "work exponentially hard to squeezes the stone and find new ideas" vs "work exponentially hard to find a whole new paradigm". The first is what necessitates exponential increase in researchers to discover the next piece of insight from within a paradigm.

But as Kuhn would point out, the emergence of a new paradigm opens up new vistas. Unfortunately we can't tell how or when it might open up. Maybe banging away at the seams of the existing paradigm is what's needed. Maybe it's trying to find new paradigms directly. Maybe it's throwing smart people from multiple disciplines together and hoping for the best. Regardless, that path of breaking through a paradigm isn't guaranteed, nor is it predictable with any degree of accuracy. We can't tell whether a pause in productivity growth of a few decades is cause for concern or just a blip or a random adjustment period.

My conclusion here is that the dips we see are because the level of specialisation we have achieved increases the level of coordination that's required to get the benefits of one field to another. You could call us victims of our success, though that would be too pessimistic. Knowledge is naturally accretive, though knowledge also requires chiselling out of raw marble to generate. If we need to continue our paths what's concretely needed is tools and norms and institutions to help explore the idea-frontier better. We don't need to jump off a cliff in sadness at the thought that our physicists aren't as productive as those in the mid-century. And we shouldn't jump off a cliff in joy that mRNA vaccines show us the path to a brave new world. This would be the time for some intellectual humility.

This has implications for the extant narrative of modern capitalism. Marianna Mazzucato among others have written extensively about how all the symbols of modern capitalism stand on the "shoulders of giants". Consider a few things:

1. The idea of long and exclusionary patents, derives in no small measure from the underlying assumption of innovation and invention being largely stand-alone phenomena. At the least such innovations attribute a disproportionate share of credit for an new product or service to those who brought it to the market, overlooking the contributions of all those ingredients and their creators. 

2. There is a strong element of path dependency with most innovations. The final product or service is only the last step in a long journey that has been traversed over time involving countless iterations and intermediate stages. In many cases, the previous stage of evolution or iteration critically determines the design or specifics of the present stage. 

3. Nowhere is this evolutionary dependence more relevant than in digital technologies which enjoy network effects. Even their founders admit that they did not plan the creation of the Amazon or Facebook of today. These behemoths emerged opportunistically as new paths opened up with the growth of their original businesses. As a corollary, almost all digital businesses of today are built on the foundations of pre-existing digital landscapes.