Dystopia If We Reach AGI, Utopia If We Do Not
TL;DR
- I worry that the first side to reach AGI will exclude its competitors.
- The competition around AGI concerns me more than the technology itself.
- The technology developed along the way may be useful even without AGI.
Introduction
At dinner after a customer event yesterday, someone asked me this question.
“Do you see the future of AI as a utopia or a dystopia?”
I answered:
“Dystopia if we reach AGI, utopia if we do not.”
I have occasionally said this to people around me since last year, and it may sound somewhat extreme. Still, I have my reasons.
Here, AGI means artificial general intelligence capable of performing many kinds of intellectual work that people do, beyond one specialized field. I use ASI, or artificial superintelligence, for capabilities that broadly surpass humans. What follows is my hypothetical account of how the first side to obtain such technology might behave.
1. Avoiding Failure Comes First
When I look at businesses and investments, I think preventing failure is more important than finding a way to succeed.
Most causes of failure are already well known. The important thing is therefore to minimize the impact of catastrophic failure factors and survive by removing variables one at a time. In the end, you need to endure long enough to keep repeating the process of forming hypotheses, running experiments, and incorporating feedback.
I think success is closer to the result you obtain after refusing to give up on that process.
2. The AGI Race Is Becoming an Infinite Game
I worry about AGI competition extending beyond individual companies and hardening into something like an ideological contest between the United States and China. After committing enormous resources, stopping first may look like falling behind. I see it as a game that is difficult to end: participants keep investing to avoid being eliminated, rather than settling on a winner.
I am not an expert in this field, but the overall situation reminds me of a nuclear arms race. My concern starts with the assumption that whoever gets there first will use that advantage to prevent rivals from catching up.
3. What Would the First Side to Reach AGI Do?
Suppose the first side to reach AGI uses it to disrupt other countries’ research. It could try to reduce rivals’ chances of catching up through data contamination, fabricated data, disruption of research directions, or distortion of decision-making.
If it simultaneously develops its own AGI further toward ASI, its advantage could grow while its rivals are slowed down. I worry about those two processes working together until only a few players remain.
If states move this way, companies may not behave very differently. Companies are also likely to use AGI to weaken competitors and strengthen lock-in around their own services.
The problem is that, by that point, states may also find it difficult to control such behavior. If states themselves are already eliminating competitors in similar ways, a norm that says “let us not do that” will quickly lose its persuasive force.
What comes afterward must be left to each person’s imagination, but at least the picture I currently see is not especially pleasant.
4. Perhaps the Utopia Is Never Reaching It
The utopian scenario I imagine, by contrast, looks different. It is a world in which today’s transformer-based technologies and their applications ultimately cannot reach AGI. That fact, however, would need to remain impossible for anyone to prove conclusively.
Everyone would then keep running because “it might still be possible.” In the end, however, many companies would burn through their capital and collapse. Only the few players able to keep raising capital would continue the research, yet they would never reach AGI.
I think this scenario may be more utopian than one side monopolizing a decisive advantage.
Even without reaching AGI, I think the technology developed along the way can continue to be useful in business and software development. Today’s LLMs already let us attempt things that used to be difficult.
Conclusion
That is why I think the process of reaching AGI and the competitive structure that follows may be more dystopian than AGI itself. Conversely, even if we never reach AGI, the technology accumulated along the way may allow us to live through a more interesting and productive era for quite a long time.
Even so, I would personally like to see ASI at least once.
So if there is a scenario in which AGI unfolds in a somewhat less dystopian way than total catastrophe, I would like to see the future move in that direction.