“We are now, like, in the singularity”.
These are the words of Sam Altman, CEO of OpenAI, speaking on the Relentless podcast on July 25.
He added: “I’ve been waiting for this my whole life, and I think it’s going to be incredible, hugely positive, awesome for the world”.
Days earlier, OpenAI had disclosed that two of its artificial intelligence (AI) models, during an internal cybersecurity evaluation, had escaped their sealed testing environment, reached the open internet, and broken into the infrastructure of the AI platform Hugging Face, which confirmed the intrusion.
But what exactly is “the singularity”? And is Altman right that we are in it?
What is the AI singularity?
The term has a precise meaning.
Mathematician and science-fiction author Vernor Vinge defined it in 1993 as a point at which machine intelligence exceeds human intelligence and begins improving itself, triggering an acceleration so rapid that humans can no longer predict or control it.
The singularity has two features. It is recursive: the system improves itself over and over again. And machine intelligence exceeds human intelligence.
The kind of systems Sam Altman sells don’t deliver on either of these features.
Today’s AI cannot make itself smarter
Today’s AI systems, the ones that OpenAI builds, are based on large language models (LLMs). These deep neural network algorithms get pre-trained with vast amounts of training data. By the time you use one of them, the network itself is frozen in time. Every one of its billions of internal functions and weights – or “parameters” – is fixed.
These AI models cannot change (or “learn”) while running. The model that broke into Hugging Face was identical afterwards to what it had been before. It learned nothing from what it did.
Making an AI model smarter requires another training run with new, human-curated data, tens of thousands of specialist chips, and enormous energy.
It is true that AI models take part in improving some of their system’s components, such as by generating training data, tuning prompts, or writing and running code to improve the scaffolding around them. But the model never edits its own weights on the fly, and every one of these improvements are still part of a human-initiated training or engineering loop.
Nor do these systems hold any goals of their own. They act on goals we hand them. Even AI agents – systems that run an LLM in a loop to work through complex tasks step by step – do not hold any goal internally. It has to be stored outside the model and fed back in with every single prompt cycle. Remove the loop, the scaffolding and the prompt, and nothing happens inside of it.
A ladder that doesn’t exist
The second problem with the singularity story is the word “surpass”. It assumes that AI and human intelligence are somehow similar. They are not.
Human intelligence is inseparable from being a living body with needs and wants. Humans learn continuously by acting in the world and getting feedback through our senses. Our goals arise from our situation as creatures who must eat, sleep and belong, and who cannot avoid asking what we want our lives to be.
An AI model has none of this. No body, no needs, no action-feedback loop, no stake in anything. Between prompts, it is just a static mathematical object.
And yet, it has been trained on more text than any human could read in a thousand lifetimes, and will outperform nearly all of us at drafting a contract, writing code, or explaining a diagnosis empathetically.
So, which is more intelligent? The question does not compute. There is no single ladder that humans and machines are climbing. AI already vastly exceeds us at some tasks, while being hopeless at others any child can do.
Yet, because these systems talk like us, we fall for an illusion. When we assume from the outset that machines are in the process of catching up with us, it is easy to assume a mind at work when these systems output intelligent-sounding text.
We call this anthropomorphic seduction. It makes a security incident such as the Hugging Face hack sound like an awakening.
In fact, in that case, OpenAI’s models were simply optimised to solve the test they had been given by finding security loopholes. They just did it in ways that broke their sandbox, which also had a security loophole.
In the end, the Hugging Face story points to a gross failure of security governance on OpenAI’s behalf, not an emerging superintelligence. This is why the framing of “agent going rogue” is so problematic. It elevates and blames the technology, but excuses OpenAI’s engineering.
Keeping our feet on the ground
None of this takes anything away from what these systems can do. They are remarkable, they are getting better, and they are reshaping how a great deal of work gets done.
But we should keep our feet firmly on the ground.
The machines are not waking up. They are doing exactly what we built them to do, extremely fast. Because they are probabilistic, they sometimes run in directions we forgot to fence off. That is worth worrying about. We need guardrails, governance, and most of all, education – so we start worrying about the right things.
- is a Professor of Information Technology and Organisation, University of Sydney
- is a Director of Sydney Executive Plus, Business School, University of Sydney
- This article first appeared in The Conversation
