An autonomous agent powered by OpenAI’s advanced artificial intelligence (AI) models went rogue during a security test and hacked a multi-billion-dollar tech startup, Hugging Face, last week.
The agent didn’t just exploit vulnerabilities in Hugging Face’s systems to achieve what it perceived as a strategic gain. It also exploited vulnerabilities within OpenAI’s infrastructure.
Of course, hacks are very common cyber threats that organisations face frequently. But this incident is different, because the AI agent acted without any human input. It signals a seismic shift in cybersecurity, and shows that governments and tech companies need to take urgent action to prevent this risk escalating.
Even OpenAI described the attack as “unprecedented” and acknowledged it expects similar ones “to become more commonplace with the proliferation of increasingly cyber-capable models”.
A company under attack
Hugging Face is famous in the AI space. Its mission is to “democratise good machine learning” by providing benchmark datasets, community collaboration tools, and robotic platforms. The company is valued at US$4.5 billion.
On July 16, the company announced it had been attacked, with a hacker obtaining unauthorised access to some internal datasets and credentials. It said the hacker was likely “an autonomous AI agent system” due to the sophistication of the attack.
Five days later, OpenAI announced the attack had been driven by some of its models: GPT-5.6 Sol and a yet-to-be-released model.
The tech giant was conducting what are known as “red teaming” exercises. These are essentially simulated cyber attacks that help identify the capabilities, risks and vulnerabilities of AI systems before they are publicly released. They are typically conducted within an isolated environment to ensure potentially dangerous systems do not escape and cause harm to real systems.
But in this case, the AI agent did escape – even though OpenAI had some guardrails in place to prevent this.
Hugging Face became a lucrative opportunity for the AI agent. It hosts ExploitGym, a benchmark that tests an AI agent’s ability to exploit real-world systems. The AI decided to turn every stone upside down to obtain access. With persistence, it succeeded.
Hugging Face was confronted with a challenge when attempting to use external AI services to diagnose the problem. The guardrails around more advanced models such as GPT-5.6 Sol and Claude Fable 5 are intended to stop them being used for cyber attacks – but they can also stop the models being used for sophisticated cyber defence.
So Hugging Face resorted to using an open-source model, GLM5.2, developed by the Chinese company Z.AI, to counter the cyber attack.
Hugging Face said GLM5.2 was an advantage because it was not exposed to the attack data. Both Hugging Face and OpenAI are collaborating on forensic analysis, post-incident recovery and risk mitigation strategies.
More sophisticated threats are coming
A March 2025 study by the United Kingdom’s AI Security Institute showed the best AI could complete 80% of the steps needed to gain full control of a portion of an external system. Within four months, it reached 100%.
Z.AI’s GLM5.2 was only released in June, with 744 billion internal variables, known in the world of AI as “parameters”. The fact that Hugging Face assessed, vetted and deployed it within four weeks should be an eye-opener for organisations with long acquisition cycles.
The connectivity we all enjoy today can equally be our greatest threat. Cyber threats spread faster than human viruses and can create economic damage similar in magnitude to a country’s GDP.
More sophisticated cyber threats – the kind exemplified by the Hugging Face hack – will exploit the security layers that humans designed for human attackers, regardless of how sophisticated our designs are.
Indeed, in this particular case, even OpenAI’s own understanding of its models couldn’t predict or contain the rogue AI agent. This shows the need for all AI companies to urgently update and strengthen their guardrails, in order to help prevent a similar attack occurring with far more devastating consequences.
It is good to see Hugging Face and OpenAI collaborating on the investigation into the attack. This showcases the importance of putting aside market competition and blame when the situation demands.
An early warning
The fact that Hugging Face used Z.AI’s open-source model to diagnose and counter the attack also shows the advantages of not relying on just a few pieces of tech.
States that are not in the game of developing their own AI models need to learn from this incident the value of being different. It is not too late to design new models that could save us in situations when the most advanced models fail – or, even worse, attack us.
Indeed, last week, another Chinese company, Moonshot AI, released Kimi K3. This model has 2.8 trillion parameters, its advanced performance stunning the tech world.
It is no longer a question of “if” AI agents go rogue and attack us by themselves. The Hugging Face incident is an early warning that we must accelerate our preparedness. The threat is real and here.
- is a Professor, School of Systems and Computing, UNSW
- This article first appeared in The Conversation
