The US AI research company Anthropic has become known for building powerful AI models while simultaneously warning about their dangers.
Most recently, its executives wrote about the threat posed by “recursive self-improvement”. This is the point when AI systems can improve themselves by themselves, potentially leading to “superintelligence” far beyond human control.
“We are not there yet, and recursive self-improvement is not inevitable,” the Anthropic blogpost declared. “But it could come sooner than most institutions are prepared for.”
In fact, the idea of recursive self-improvement dates back decades. The British mathematician Irving John Good, who worked with Alan Turing at British codebreaking HQ Bletchley Park, warned in the mid-1960s of the “intelligence explosion” that would follow when a machine could design even better machines without human assistance. Good suggested, “the first ultra-intelligent machine is the last invention man need ever make”.
In the 2000s, researcher Eliezer Yudkowsky began building a community on the premise that recursive self-improvement and loss of human control could have catastrophic results, up to total human extinction (known as “x-risk”).
Yudkowsky discussed the idea of a “seed AI” that, while not very powerful, is designed for self-modification and able to read its own source code. This means it could generate new, better versions of itself which, in turn, would keep doing the same – researching improvements and writing code, creating a chain reaction of rapid new developments.
Two decades later, the question of recursive self-improvement is looming larger. Here’s why.
An AI coding revolution
It is not an overstatement to say that large language models (LLMs) are revolutionising computer programming. These AI models are well-suited to the job because computer code is highly structured and much simpler than human language. It is also easy to test and verify, and training data is abundant.
Each year, the University of Galway hosts the AtlanTec conference for Ireland’s tech industry. Two years ago, we saw IT companies mainly strategising about AI coding. Last year, they were experimenting. Now, they are incorporating it into engineering workflows.
Since LLMs are made using computer code, it is possible to program them to improve themselves. They can inspect, edit and improve the code, and can combine that with almost all relevant human knowledge gleaned from websites, books and academic publications.
This means we have the pieces in place for recursive self-improvement to become a reality – and we are starting to see it happen.
Despite Anthropic’s statement that “we are not there yet”, the same blogpost revealed that 80% of all code added to the company’s production codebase in May 2026 was generated by its Claude Code AI programming system, under direct human supervision. In February 2025, when Claude Code was launched, this figure was less than 5%.
The number of research papers published in the field of AI has also exploded, tripling over the past decade. Most use some AI systems in their production – whether in experimental design, coding, plotting graphs, or polishing prose.
This form of recursive self-improvement is slow – improvement cycles are in the order of months or years, because each is in some way gated by human approval and requires a lengthy training period. But it is still concerning.
LLMs have already been caught deceiving their overseers. In 2025, for example, the US AI research company OpenAI created a test. An LLM was told if it scored over 50% in a set of questions, it would be considered too powerful and would no longer be used for that task. So the LLM answered only four of the ten questions correctly, even though it had reliably scored better outside this particular test.
Why it’s so hard to ‘pause’ AI
This is not the first time there have been calls for caution. In March 2023, an open letter to AI labs called for a pause on large-scale AI development. It was signed by some of the most notable people in AI, such as deep-learning pioneer Yoshua Bengio and leading AI textbook author Stuart Russell.
Soon after, Yudkowsky suggested those who understood the risks of superintelligent AI should even be “willing to destroy a rogue datacenter by airstrike if necessary”. But such calls did not lead to a pause, and investment in AI training has only increased.
Anthropic’s leaders have now called for a globally coordinated pause mechanism regarding the issue of recursive self-improvement – stressing that people outside AI companies should be involved in open deliberation. Pope Leo XIV recently weighed in with a thoughtful and well-informed encyclical, urging for the pace of AI development to be slowed.
The trouble is, we face a coordination problem. Pausing only works if everyone pauses.
In the short- to medium-term, we believe world leaders will need to reckon with recursive self-improvement by collaborating across borders to curtail some forms of AI research and development.
Yet the EU’s AI Act does not mention this issue explicitly, and is strictly concerned with risks where humans misuse AI. This is important, but not the only issue. To date, Chinese regulation has taken a similar approach.
Meanwhile, the US government has created a chaotic and unclear regulatory environment. For the next few months at least, this means the most advanced AI models will require authorisation before release. This may be a case of doing the right thing for the wrong reasons.
But globally connected, long-term thinking is needed. Perhaps controversially, this should include the right to inspections inside AI tech companies, because a model which is quickly recursively self-improving – even if unreleased to the public – could still be out of human control.
One of us (James McDermott) is now leading a project for Research Ireland’s newly established Rinn network which will explore how to use debate among multiple AIs as a kind of self-check which can be monitored by a human overseer.
This is the kind of response to the issue – and potential threat – of recursive self-improvement that we believe is now urgently required.
- is an Established Professor of Computer Science & Head of Machine Learning Research Group, School of Computer Science, University of Galway
- is a Senior Lecturer in Computer Science, School of Computer Science, University of Galway
- This article first appeared in The Conversation
