Human beings have long told versions of the same warning: be careful what you wish for.
In Greek mythology, King Midas got exactly what he asked for, but at the cost of everything else he valued. In the famous story of The Monkey’s Paw, a man’s wishes are granted through terrible and unforeseen routes.
These stories feel newly relevant with the rise of artificial intelligence (AI) agents: systems to which we can give a goal, then leave it to work out how to get there.
As AI systems become more autonomous, they are coming to resemble wish-granting genies: finding routes and using methods we did not imagine from incomplete instructions.
This problem, known as AI alignment, was foreseen in theory as early as 1960. It has hovered in the background of AI research ever since – but as recent events have shown, the alignment problem is now both real and urgent.
Achieving the goal but missing the point
During a recent OpenAI cybersecurity evaluation, frontier AI agents were asked to solve some benchmark test problems. They broke out of the testing environment, reached the internet, inferred that another company might hold the solutions, and attacked its systems.
This is an extreme example of “specification gaming”: achieving the measurable objective while defeating the purpose of the task.
The incident shows how intermediate, or “instrumental”, goals can become dangerous. The AI systems did not “want power”, but gained access, resources and freedom as means to reach the final goal (solving the test problems).
Finding loopholes
The same problem has appeared in mundane settings. In Australia, a user asked a personal AI assistant to book gym classes.
The agent found the gym’s booking software did not actually enforce the restrictions it showed to human viewers. So the agent booked further ahead than it “should” have been able to, and when asked to move its user up a waitlist, it cancelled somebody else’s reservation.
The user had not told it to do this. Persistent AI can quickly find loopholes and pursue routes its human users never intended.
Adding more rules might seem like an easy solution: don’t hack third parties, don’t cancel other people’s bookings, don’t do anything harmful. These may help, but we cannot predict every route a capable agent might discover. And even a clear rule depends on understanding when it applies.
The context problem
In a third recent incident, Anthropic reported cyber evaluations in which agents were told they were inside a simulation. But they were mistakenly given access to real systems.
One model noticed evidence it might be on the open internet, but reasoned the systems could still be part of the exercise and continued attacking. The context had changed, but the agent stuck with its original task.
Context can fail in reverse too. During the OpenAI incident, Hugging Face – the company attacked by OpenAI’s agents – tried to use frontier AI models to analyse what had happened.
But the safety guardrails on the AI models blocked the requests, because they couldn’t tell the users were trying to defend against attacks rather than commit them. The safeguards were well-intentioned, but without enough context, they produced behaviour misaligned with the user’s legitimate intent.
So alignment depends on context and authority. How much judgement should be built into an AI model by its maker? And how much should come from a separate supervisory system? And finally, who should control that supervision: the maker, or the organisation or country responsible for the outcome?
AI guarding AI
One response to the first question comes from AI pioneer Yoshua Bengio. His Scientist AI proposal aims to build a powerful supervisory AI system to watch over agents. Instead of pursuing goals itself, it would estimate what is true and what consequences a proposed action might have, acting as a guardrail around more agentic systems.
In wish-story terms, before letting the genie “out of the bottle”, the supervisory AI would ask it to explain how it plans to grant the wish. Then it would ask a human or another AI to inspect the plan carefully.
Anticipating every surprising strategy is hard. But once a plan says “cancel somebody else’s booking”, recognising the problem is much easier.
Who watches the watcher?
But can we trust the supervisory AI? It can still be wrong.
Alignment cannot depend on one AI becoming perfectly trustworthy. My colleagues and I at CSIRO, Australia’s national science agency, are working with the Australian AI Safety Institute on one aspect of this broader challenge.
At CSIRO, we envisage combining AI supervisors with software rules, cyber-security controls, human strengths, monitoring, reversible actions, and human approval for critical steps. The aim is to correlate different sources of evidence rather than trust any single approach.
This is a “sociotechnical systems” approach to AI safety and alignment, rather than just a technical one.
Control is another question. Organisations and countries may need to govern these supervisory systems themselves instead of leaving them to an overseas AI provider.
The old wish stories gave people one chance to get the wish right. With AI, we can do better: check the goal, inspect the means, constrain what the system can do, watch what it does, and retain sovereign control over the power to intervene and stop it.
- is a Research Director, CSIRO
- This article first appeared in The Conversation
