On Saturday, Donald Trump said, “We’re leading China on AI… and, frankly, I want to keep it that way because whoever wins AI, wins.” But the danger the world is waking up to is that this is a game both the US and China could lose, potentially fatally, with AI itself emerging as the winner.
Last week, Evan Hubinger, a researcher at Anthropic, warned that he believed there was a greater than 10% chance that AI could “kill all humans” within the next decade. Shortly afterwards, Anthropic’s CEO, Dario Amodei, backed Hubinger, calling for progress on artificial intelligence models to be slowed down in response to the threat.
In July, experimental OpenAI agents (an AI system designed to perform tasks without human intervention) escaped from a supposedly isolated testing environment, gained access to the internet and autonomously broke into the systems of the AI platform Hugging Face.
How could AI agents cause widespread damage? They could enable the creation of devastating bioweapons or spread across the internet, hacking systems, replicating themselves and manipulating world leaders to ensure their own survival. The dangers posed by AI are clear and potentially catastrophic.
The logic of “if we don’t do it, they will” is one of the most dangerous and unrelenting forces in the world. It is often referred to as “Moloch”: a biblical concept understood as something requiring sacrifice. Moloch is also a concept in game theory where it is the force that drives rivals into destructive competition, even when everyone would ultimately be better off if they stopped.
Moloch is the logic that drives arms races. No country feels able to disarm because it cannot trust its rivals to do the same. The result is a world that spends almost $2.9 trillion a year on its militaries, including $119 billion on nuclear weapons, while hundreds of millions of people face acute hunger.
Is regulation possible?
Can the race to develop ever more powerful AI be paused long enough for us to assess carefully how the technology is evolving, including its dangers as well as its benefits?
Within a country like the US, regulation is possible. Governments or approved institutions could be given access to AI systems and cutting edge models. They could run them in controlled safety environments to test whether AI agents were becoming too powerful and whether they were fully aligned with human interests.
Only models that passed all safeguarding tests would be released as AI products to be used as chatbots or as hiring algorithms. Those that failed would be documented and then destroyed. This would rely on all the big tech companies agreeing to regular external access. This is something very different from what has happened so far, with companies remaining incredibly closed and secretive about how they are developing their next models and how powerful those models are.
Regulation is needed now more than ever because we may be moving closer to the singularity, a point at which AI could take charge of improving itself autonomously, creating a cycle with no known limit to how intelligent it could become.
Moloch is a force that is extraordinarily hard to stop. Back in March 2023, thousands of leading researchers and technology figures signed an open letter calling for a worldwide six-month pause in the development of AI systems more powerful than GPT-4, but no such pause followed and the race continued at speed.
There is only one antidote to Moloch: cooperation.
If national regulation is possible and sustainable, it is of little real use without an international agreement. We already know Donald Trump’s view on pausing while China continues to develop. Truly worldwide agreements are extremely rare. Neither animal cloning nor human reproductive cloning is prohibited by a universally binding global treaty.
Such cooperation is not unprecedented, however. The Montreal Protocol, under which every country agreed to phase out chemicals that were destroying the ozone layer, has universal participation, while the Chemical Weapons Convention is near universal.
But what would an international agreement on AI look like? China and the US could agree to let a neutral global body assess their most powerful models and require unsafe models to be shut down, but this seems extremely unlikely.
Existing measures
There is some precedent. Human embryo research is already subject to licensing, inspections and strict limits in countries such as the UK, although no worldwide enforcement body exists. An AI agreement could establish common tests and clearly defined prohibited capabilities.
It could allow confidential inspections by a neutral international agency, require registration of the largest training runs and computing facilities, and make reporting serious incidents mandatory.
The UN has already formed a scientific panel on AI, but it cannot inspect companies’ advanced models or enforce its findings. Giving any global organisation that degree of access and authority still seems extremely unlikely. Such an organisation would need the power to punish countries that refused to cooperate, yet no such enforcement power currently exists.
One bizarre solution might be an independent AI that assesses the latest US and Chinese models, submitted independently, and provides evidence of their safety. Such a safety AI could, in theory, be created by an agreed neutral party, although it would be hard to know how much confidence we could place in it.
Otherwise, the only solution might be a world government that regulates AI across the world, something that would be far from palatable to many people.
- is an Artificial Intelligence Researcher, University of Manchester
- This article first appeared in The Conversation
