Over the weekend, Anthropic chief executive Dario Amodei called for artificial intelligence (AI) companies, including his own, to slow down their work. Sam Altman and Elon Musk, heads of rivals OpenAI and xAI respectively, agreed.
The move reflects concern across the AI industry and more broadly about the dangers of new, rapidly improving systems. Recent high-profile incidents such as OpenAI AI agents hacking another company and hijacking a public website have shown current systems can break out of safety confines – and even more capable systems are in development.
Further complicating AI safety is the tension between safety and performance. Companies will be reluctant to limit the performance of their models in the name of safety for fear of losing ground to competitors. US President Donald Trump has also rejected calls for a slowdown, for fear of losing ground to China.
This means any successful effort at “pacing the rate of capabilities advancement so that risk prevention has time to keep up”, as Amodei puts it, will require significant cooperation between rival companies – and nations.
Risk minimisation
New technologies often bring new risks, and new concerns. Often governments, researchers and companies do find ways to manage those risks.
In 1975, the Asilomar conference on then-new DNA technologies did much to ensure research didn’t get ahead of our understanding of safety and risk. Similarly, in the 1990s, the US government attempted to build a consensus on limiting cryptography and computer security.
The AI situation has an extra twist: the industry is in the middle of a gold rush. Scientific rivals may be able to restrain themselves, but commercial rivals rarely hold back.
There are clear precedents for self-regulation failing in the face of competitive pressure. In the Boeing 737 Max disaster in 2018, for example, pressure to catch up to rival Airbus led Boeing to hide the limitations of the Max, which ultimately cost lives.
In the AI race, the scale of the competitive tension is even greater. Anthropic and OpenAI are both competing to establish market dominance before pursuing share market listings that could raise tens or hundreds of billions of dollars.
And at the nation-state level, the stakes are higher again, with the US and China each hoping to use the new technology for geopolitical advantage.
The politics of a pause
Amodei’s slowdown proposal centres around a three-step plan: embedding independent third-party safety reviewers, establishing coordinated industry safety standards within democratic nations, and eventually securing global agreements. This would include strict limits on AI chip exports to companies and countries that do not agree to prioritise AI safety.
Anthropic and OpenAI have already agreed to the first phase of this plan, despite their history of suing, publicly insulting, and undercutting each other in pursuit of market dominance.
While recent high-profile hacks may have forced their hands, the leading AI companies may benefit from a development pause or slowdown. For one thing, it could put off strict legislation such as US Senator Bernie Sanders’ proposed Ban Artificial Superintelligence Act. For another, by creating expensive safety standards and limiting chip exports, it could block smaller competitors – especially Chinese companies such as DeepSeek and Alibaba – from catching up.
A pause would also give the overstretched frontier labs a chance to recoup, recover, and focus on profits over progress.
These AI labs have spent enormous amounts to produce their existing models, which must be repaid. At the same time, progress is hitting speed bumps as new high-quality training data gets harder to find, and building the massive data centre infrastructure needed to sustain development is a challenge in itself.
A coordinated safety pause could be a convenient public reason for a plateau in AI model performance.
What can be done
Making AI safe won’t be easy. The governance of software is notoriously difficult, as past attempts to legislate encryption software have shown.
However, unlike other software, AI is dependent on relatively scarce physical hardware. It needs advanced silicon chips and the massive data centres required to power them.
This is where governments have real leverage. Here, they have ways to monitor and control AI development, if they can find the legislative will.
In the meantime, there are steps businesses and governments can take to minimise how exposed we all are to AI-driven harms. This should include ensuring that critical safety infrastructure – like power grids, water supplies, and military systems – are not only isolated from AI, but potentially even isolated entirely from the internet.
And the question of liability is also important. It can’t just be the users of AI systems who face legal jeopardy for acts assisted by AI, but also the people responsible for making AI systems.
Fundamentally, harm produced by AI – even by “autonomous” AI systems – isn’t an abstract technological byproduct. It is the direct result of decisions made by both those making AI, and those using it.
- is a Senior Research Fellow, School of Computing and Information Systems, The University of Melbourne
- This article first appeared in The Conversation
