Dystopia If We Reach AGI, Utopia If We Do Not
TL;DR
- The AGI race has already moved beyond individual companies and is becoming closer to an ideological competition between the United States and China.
- The first side to reach AGI is likely to use that advantage immediately to eliminate its competitors, not merely hold it.
- The process of reaching AGI and the competitive structure that follows may be more dystopian than AGI itself.
Introduction
At dinner after a customer event yesterday, someone asked me this question.
“Do you see the future of AI as a utopia or a dystopia?”
I answered:
“Dystopia if we reach AGI, utopia if we do not.”
I have occasionally said this to people around me since last year, and it may sound somewhat extreme. Still, I have my reasons.
1. Avoiding Failure Comes First
When I look at businesses and investments, I think preventing failure is more important than finding a way to succeed.
Most causes of failure are already well known. The important thing is therefore to minimize the impact of catastrophic failure factors and survive by removing variables one at a time. In the end, you need to endure long enough to keep repeating the process of forming hypotheses, running experiments, and incorporating feedback.
I think success is closer to the result you obtain after refusing to give up on that process.
2. The AGI Race Is Becoming an Infinite Game
Seen from this perspective, today’s AGI race has already moved beyond what individual companies can bear. It is becoming increasingly similar to an ideological competition between the United States and China, and with enormous funding and sunk costs already committed, it cannot easily stop. I therefore think this competition is increasingly taking on the character of an infinite game in which clear rules are difficult to enforce.
If someone truly reaches AGI first under these conditions, the next move may be more obvious than we think. I am not an expert in this field, but the overall structure feels similar to a nuclear arms race. The first side to arrive is unlikely merely to hold its advantage. It is likely to try to use that advantage immediately to eliminate its competitors.
3. What Would the First Side to Reach AGI Do?
First, the side that reaches AGI first will probably try to deploy it immediately against its competitors. It is likely to attach every available piece of infrastructure and move toward reducing or eliminating the possibility that rival countries can reach AGI at all. From relatively indirect methods such as data poisoning at the level of entire language communities, generating fake data at scale, misdirecting research, and distorting decision-making to far more aggressive measures, AGI could become an actor that finds and executes the optimal means to achieve its objective in near real time. In other words, AGI may be less like an asset that is merely held and more like an actor deployed immediately to cut off every competitor’s chance to catch up.
Second, it will probably try to entrench that advantage by improving the AGI even faster and pushing it toward ASI. In other words, on one side it is likely to deploy the AGI it already has to suppress competitors immediately, while on the other it accelerates the evolution of that AGI into a stronger system. The AGI race could develop into a structure close to winner-takes-all. Even if several players remain, only a small number of overwhelmingly dominant players are likely to survive.
If states move this way, companies may not behave very differently. Companies are also likely to use AGI to weaken competitors and strengthen lock-in around their own services.
The problem is that, by that point, states may also find it difficult to control such behavior. If states themselves are already eliminating competitors in similar ways, a norm that says “let us not do that” will quickly lose its persuasive force.
What comes afterward must be left to each person’s imagination, but at least the picture I currently see is not especially pleasant.
4. Perhaps the Utopia Is Never Reaching It
The utopian scenario I imagine, by contrast, looks different. It is a world in which today’s transformer-based technologies and their applications ultimately cannot reach AGI. That fact, however, would need to remain impossible for anyone to prove conclusively.
Everyone would then keep running because “it might still be possible.” In the end, however, many companies would burn through their capital and collapse. Only the few players able to keep raising capital would continue the research, yet they would never reach AGI.
Ironically, I think this scenario may actually be more utopian.
Even if we never reach AGI, today’s LLMs have already opened far more paths than we had before. In business and technology alike, we will be able to solve much harder problems in far more enjoyable ways.
Conclusion
That is why I think the process of reaching AGI and the competitive structure that follows may be more dystopian than AGI itself. Conversely, even if we never reach AGI, the technology accumulated along the way may allow us to live through a more interesting and productive era for quite a long time.
Even so, I would personally like to see ASI at least once.
So if there is a scenario in which AGI unfolds in a somewhat less dystopian way than total catastrophe, I would like to see the future move in that direction.