Imagine an artificial intelligence (AI) system that has performed reliably for months. Its recommendations are almost always right, and the employee reviewing them has learned that close scrutiny rarely changes the outcome. Then something unusual happens. The AI makes an error or can’t proceed and needs human intervention. The employee is expected to step in and remains responsible for the outcome.
But does the employee know how to do the task?
I’m a professor of management information systems and the academic director of Drexel University’s Center for Applied AI & Business Analytics. In my research on human-AI relationships, I call this latent risk: the possibility that people may lose capabilities their organisations still depend on them to exercise.
AI can continue to perform well while the people overseeing it become less practised at exercising independent judgment and less prepared to intervene. The risk becomes latent when the organisation still depends on that human capability to step in but has little cause to notice that that capability may be weakening.
The concern is not entirely new. The idea of latent risk builds on decades of research into how people respond to automation. For example, studies of automation bias have shown that people can miss problems or follow automated recommendations even when other evidence suggests they should not.
What is different today is the role AI is beginning to play in organisations. It is beginning to assume larger parts of the analysis and decision-making that people previously performed.
The success paradox
The better AI becomes, the more the nature of human work changes. Work that once required an employee to analyse a problem and reach a decision may instead be performed by AI, leaving the employee to review its recommendation. The employee is no longer doing the same work with AI assistance – rather, their job has shifted from doing the work to overseeing what the AI does.
This shift is already beginning to change how some companies train employees. Accounting giant Deloitte, for example, is redesigning its graduate audit training as AI takes over some of the manual work traditionally done by junior auditors. The firm is introducing simulations so trainees can still develop the judgment they will need as they move into more complex work.
More than four decades ago, cognitive psychologist Lisanne Bainbridge described one of the “ironies of automation”: As systems take over more of the work, instead of simply having their workloads lightened, people can be left facing new challenges in monitoring the systems and handling the unusual situations the systems cannot. Later studies found another problem: As people become less involved in a task, they can become less prepared to step in when needed. Researchers have called this the out-of-the-loop performance problem.
This reduced involvement may affect more than a person’s readiness to intervene. It can change the capabilities available to the organisation.
What work remains for people does not necessarily become easier. It may become harder. These are the situations where experience and independent judgment matter most. The result is a paradox: People may be called upon to exercise independent judgment less often, even as the judgments left to them become more difficult.
The accountability problem
Accountability may not change as quickly as the work does when organisations deploy AI systems. An employee may now oversee far more decisions than they make themselves, while remaining responsible for the outcomes. The organization’s expectations may still reflect how the work was divided before AI took on a larger role. The organisation can therefore end up holding people accountable for decisions over which they have less control.
This creates a mismatch. Research on meaningful human control suggests that responsibility should match a person’s authority and ability to act. An employee may retain the authority to override an AI recommendation and remain responsible for the outcome, yet lose the capability to do so. Authority, responsibility and capability can begin to separate.
Latent risk becomes an organisational design problem, not simply a human performance problem.
New HuggingFace paper argues that increasing agent autonomy can gradually make human oversight ineffective by causing approval fatigue, overreliance, loss of situational awareness, and skill degradation.
As agents do more, users are pushed into approval mode: skimming plans, granting permissions, and reconstructing what happened across steps.
Over time, automation bias, approval fatigue, weaker situational awareness, and skill atrophy can make those approvals less reliable.
Worse, weak approvals can become training or evaluation signals, rewarding systems for being easy to approve rather than easy to scrutinize.
Their answer is cognitive scaffolding at 2 levels: developers add strategic friction, better approval design, behavioral monitoring, and checks that force attention at consequential moments.
– arxiv. org/abs/2608.23642
Title: “AI Agents Push Humans Out of the Loop”
— Rohan Paul (@rohanpaul_ai) September 1, 2026
The measurement problem
Organisations typically judge AI-enabled processes by measures such as accuracy and productivity. Those numbers may look good even as the human role is changing. Suppose employees override AI recommendations less frequently over time. Is that a sign of growing dependence? Perhaps.
But it could also mean that the AI has become more accurate and there is less reason to intervene. The number of overrides alone tells little about whether employees remain ready to exercise their own judgment.
The key is to understand how the division of work between people and AI is changing. Which decisions have migrated from people to AI? What responsibilities remain with people? Looking at the performance of the employee or the AI separately may miss what is changing between them.
The governance problem
Organisational governance – defining roles and responsibilities – is also under pressure from AI.
Someone may remain formally in the loop with an AI system even as their actual role changes. Defining human and AI roles when a system is introduced is only a starting point. As AI improves and people become accustomed to using it, those roles are likely to change over time. There is nothing inherently wrong with that change.
The problem arises when the relationship changes but the assumptions surrounding it do not. It’s important for organisations to periodically ask: What relationship between people and AI did we design, and what relationship do we actually have today?
In short, organisational governance needs to be dynamic.
Maintaining the human backstop
Not every human capability needs to be preserved. If AI can perform a task well, there is little reason to require people to keep doing the same work. Organisations need to decide which capabilities they still expect people to have when human judgment is required. If people may still be called upon to use a capability, it needs to be maintained.
Overseeing AI is not the same as exercising independent judgment. If people rarely make the decision themselves, the organisation cannot assume they will be ready when they need to. One way to test this is to occasionally ask employees to make a decision without first seeing the AI recommendation. The organisation needs to know that the human backstop it relies on will actually work when needed.
As AI takes on more work, some human capabilities will inevitably be used less often. Some may no longer matter. Others may still be essential when AI reaches its limits. The challenge is knowing the difference and making that choice deliberately.
For me, the central question is this: Which human capabilities does an organisation still depend on, and will they be there when needed?
- is a Professor of Decision Sciences and Management Information Systems, Drexel University
- This article first appeared in The Conversation
