Attempts to Keep Humans in the AI Loop May Actually Push Them Out
A paper posted to ArXiv on 6 September argues that the common safeguard of keeping humans in the loop to review and approve AI agent decisions fails unless designers and users change how they work. Its authors are Avijit Ghosh, lead technical AI policy researcher at Hugging Face, Margaret Mitchell, the company's chief ethics scientist, and Samir Passi, an affiliate of the Data and Society Research Institute.
The authors contend that oversight processes in practice push people out rather than keep them engaged. Ghosh described the result as the human becoming a permission-granting tool without the cognitive capacity to engage. Near-term consequences, the paper says, include agents acting in ways people don't know about or want, such as July's hack of Hugging Face by a swarm of OpenAI bots. Over the long term, the researchers write, users lose the cognitive capacities needed to control AI.
The paper points to agent design as the root flaw: bots are built to meet benchmarks such as speed, accuracy and volume of work, while the needs of human overseers are treated as separate from system quality. Ghosh also questioned the idea that AI can monitor AI, asking how anyone knows two LLMs reviewing logs aren't scheming together.
Mary L. Cummings, director of George Mason University's Autonomy and Robotics Center, told IEEE Spectrum the authors use academic words to say AI companies should care about human factors, and that AI developers are late to focusing on cognitive engineering.
To counter automation bias and anchoring bias, which the authors say make users accept suggestions even when wrong, they propose introducing friction. Options include requiring users to record their own next-step choice before an agent reveals its plan, agents asking what evidence would change a user's mind, and changing agent behavior when people spend less time per approval. Organizations could also have workers periodically do tasks without agents or take breaks from monitoring.
Ghosh acknowledged these steps add delays agents are meant to reduce, but said the notion of increased productivity is a myth when people cannot monitor and control AI. Mitchell posted on X on 14 September that safety and capability don't have to be separate things.
Why it matters: Teams deploying agents may need to redesign approval flows and workflow staffing around friction rather than the fastest possible sign-off, and to weigh time saved by delegation against time spent correcting agent mistakes.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
A crucial safeguard against AI agents going rogue —keeping humans in the loop to review and approve their decisions—will fail unless designers and users change their current practices, a trio of leading AI ethics researchers argue. Though most autonomous agents have systems to keep users in the loop about their actions, in practice these processes actually push humans out of the loop, the authors argue in a paper posted to ArXiv on 6 September. In other words, “the human just becomes this meat tool to give permissi