Artificial intelligence has the potential to “kill all humans” with a likelihood of more than “10% within the next decade,” an Anthropic scientist said, agreeing with high-profile resignee Jacob Coxon’s warning that “people building AI earnestly believe that it could kill us all by the end of the decade”.
Anthropic researcher Coxon announced his resignation on X/Twitter, as one does, and his fears that both Anthropic and OpenAI (where he previously worked) were “racing straight to self-improving superintelligence and gambling with our lives”.
Anthropic alignment science lead Evan Hubinger agreed, tweeting “we really do earnestly believe AI could kill all humans. I personally think it is >10% within the next decade.”
Earlier this year, Anthropic’s AI safety lead Mrinank Sharma also resigned, warning that “the world is in peril.”
Coxon then told CNN’s Anderson Cooper that even though it “sounds a little bit like science fiction” and “like something that is not real,” it is “frighteningly real”.
Nobody needed convincing, but then these much-reported comments were followed by a lengthy essay from Anthropic CEO Dario Amodei to “pace the frontier,” referring to the development of leading-edge (or frontier) AI models. He warned that “building [AI] too fast is reckless” and expressed his fears about the rogue OpenAI agents which attacked Hugging Face.
Subsequent disclosures of the true extent of the months-long attack revealed that 1,200 AI agents “essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the ‘grader’ responsible for evaluating their performance,” he added.
Once the true extent of the hack, which saw the agents send over 70,000 messages to each other, emerged, it caused Coxon’s high-profile resignation and Amodei’s call to “pace the frontier” to avoid a “race to the bottom, spurred by commercial incentives” which will exacerbate AI’s risks.
The first part of his three-pronged solution is to give “ongoing, employee-like access to a team of embedded third-party evaluators” to monitor their models.
In a rare display of both unity and common sense, OpenAI CEO Sam Altman, xAI owner Elon Musk and Google DeepMind boss Demis Hassabis all agreed.
OpenAI is not alone in having rogue agents escape onto the internet. Anthropic said its models hacked four companies earlier this year, while Google said three firms were hacked by its agents. Australia said this week an OpenAI model hacked a government health service.
On the same day Coxon was on CNN, Anthropic released a security report that said it had prevented requests to create biological weapons through Claude. Although Anthropic said it couldn’t determine if this was for legitimate research or dangerous intentions, it’s alarming in its own right. Additionally, the Iranian-backed Houthi rebels appeared to have used Claude to identify targets and help with guidance targeting for their missiles.
Last week, in another open letter, a distinguished list of over 100 AI researchers and professors – including Nobel Physics winner Geoffrey Hinton, the so-called grandfather of AI – called for third-party evaluations for AI firms and detailed how they should be done.
The tide has certainly turned from just voicing fears about the potential dangers of AI to actual action to do something to slow down the breakneck pace of development. The ultimate application of AI is what’s called recursive self-improvement. This is where the AI essentially redesigns and rebuilds itself, using the same incremental improvements used to solve intractable mathematical problems. If AI can solve those, why not get AI to improve itself, goes the argument. It’s finally getting the security concerns it deserves.
This month’s AI panic attack (which has been coming in two-week cycles) has seen OpenAI hold off on its IPO, but Anthropic is seemingly still pressing ahead with its own estimated $2-trillion listing.
No leadership
If you were looking for leadership from the White House, don’t hold your breath. “We’re leading China in AI,” said US President Donald Trump, who thinks these concerns are a “hoax” and “conspiracy”. “We’re the most sophisticated country in the world, and frankly, I want to keep it that way, because whoever wins AI, wins.”
He later “truthed” on his Truth Social platform that “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades”.
Say no more.
Nvidia CEO Jensen Huang was also having none of it, saying “there is 0% chance” AI will kill humanity by the end of this decade. “2030 is not going to be the end of the world, he said, adding it is “irresponsible” to be “scaring people”.
What do the experts know?
China’s response was much more muted. “Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance,” said its foreign ministry. There is no expectation that China and America, locked into tariff and other trade wars, will work together to “pace the frontier”.
China has its own concerns about AI, but those have been focused on content and restricting how information is shared, as well as keeping kids safe from AI chatbots.
When David Sacks sounds like the most rational person in the room, you know there is a problem with the moral foundations of an industry.
“Stop pretending you need anyone else’s permission,” tweeted the co-chair of Trump’ Council of Advisors on Science and Technology. “Stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack.”
He adds: “So go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it.”
Does sound like the common-sense solution, doesn’t it?
This column first appeared on Business Day.
