Oversight Has a Capacity: Why More Human Review Can Make Agents Less Safe
The default safety pattern for agents that take real actions — shell commands, file edits, deploys — is a human-in-the-loop approval gate: risky actions pause and wait for a person. The intuition is that more human review means more safety. A 2026 study shows that's wrong past a point: because human attention is finite and fatigues, realized safety follows an inverted-U, and the safety-optimal guard escalates below 'everything.' Oversight isn't a checkbox; it's a finite attention budget the escalation policy spends. Here's why routing every risky action to a human can make your system less safe — and how to design the gate instead.
When an agent starts taking actions that are hard to undo — running a shell command, editing a file, pushing a deploy — the standard safety move is reflexive and feels obviously correct: put a human in the loop. Risky actions pause and wait for a person to approve them. More review, more safety. It’s the pattern nearly every team reaches for first, and it’s the one a 2026 study just complicated in an uncomfortable way: past a point, more human oversight makes the system less safe.
That isn’t a paradox for its own sake. It falls out of a fact everyone knows and most safety designs ignore — the human doing the reviewing is a finite, fatiguing resource. Treat that human as infinite and you build a gate that quietly stops working exactly when you’re relying on it most.
The approval gate and its hidden cost
The human-in-the-loop approval gate is genuinely good design for the right actions. If an agent is about to delete a production database or wire money, you want a person to look first. The logic is sound, and for high-stakes, irreversible actions it’s the correct pattern.
The trouble starts when the same reflex gets applied to everything risky-looking. Each escalation spends a unit of a scarce resource: human attention. And human attention doesn’t degrade gracefully under load — it collapses. A reviewer asked to approve five carefully-chosen actions a day reads each one. The same reviewer asked to approve two hundred starts pattern-matching, then rubber-stamping, then clicking “approve” reflexively to clear the queue. The gate is still there on the org chart. It has stopped being a gate in reality.
So the cost of escalating an action isn’t zero, and it isn’t even constant — every unnecessary escalation degrades the quality of review on every other escalation, including the one that actually mattered.
The inverted-U: more oversight, less safety
This is what the 2026 paper Oversight Has a Capacity (arXiv 2606.08919) set out to measure, in precisely the setting that matters: “LLM agents [beginning] to take real, irreversible actions (shell commands, file edits, deploys),” guarded by “a human-in-the-loop approval gate.” Its central finding is blunt: “more human oversight can make a system less safe, and the safety-optimal guard escalates below full escalation.”
The shape of it is an inverted-U. Escalate too little and risky actions slip through unreviewed — obviously unsafe. But escalate too much and you swamp the reviewer, review quality craters, and dangerous actions slip through approved — also unsafe, and more insidiously, because now they carry a human’s stamp. Realized safety peaks somewhere in the middle, at a moderate escalation rate. “Escalate everything” is not the safe end of the dial; it’s over the top of the curve and down the other side.
It’s worth picturing how the failure actually arrives, because it’s not dramatic. Nobody decides to stop reviewing. The queue just grows: by mid-afternoon there are sixty pending approvals, each one looks like the fifty before it, and the reviewer — under pressure to unblock the agents and the people waiting on them — settles into a rhythm of skimming and approving. The ninety-first approval that day is the one that drops a table, and it sailed through in two seconds with a click, because by then the click was the review. The gate didn’t fail loudly. It eroded, quietly, under exactly the volume that “escalate everything” guarantees. That’s the trap: the policy that feels maximally cautious is the one that manufactures the conditions for a waved-through disaster.
The instinct to maximize oversight, in other words, is optimizing the wrong quantity. You don’t want maximum escalations. You want maximum effective review, and those two diverge the moment the human gets tired.
Oversight is a budget, not a checkbox
The reframe the paper offers is the useful part: “agent oversight … is not only a classification problem but a resource-allocation one: human attention is finite, and the guard’s escalation policy spends it.” That sentence should change how you design the gate.
Treating oversight as classification — is this action risky, yes or no, escalate if yes — implicitly assumes the human can absorb however many “yes” cases the classifier produces. Treating it as resource allocation starts from the opposite premise: you have a fixed attention budget per reviewer per hour, and the escalation policy’s job is to spend that budget where it buys the most safety. Every escalation is a withdrawal. Spend it on the genuinely ambiguous, high-blast-radius, irreversible decisions; do not spend it on the thousand low-stakes actions a classifier flagged just to be safe, because each of those withdrawals leaves less in the account for the decision that counts.
There’s a second finding that makes the brittle-classifier approach even worse: risk is subjective. The paper reports that human reviewers “only moderately agree on what is risky (Fleiss’ kappa = 0.52)” — there’s no single correct label for which actions are dangerous. So a gate built as a crisp risk classifier is not only spending the budget badly; it’s spending it against a target that reasonable people don’t even agree on. The design problem isn’t “build a perfect risk detector.” It’s “allocate a contested, finite resource wisely under uncertainty.”
Designing the gate, not just installing it
If oversight is a budget, the escalation policy is how you spend it, and a few principles follow directly:
- Tier actions by reversibility and blast radius, not by surface risk. An action that’s trivially reversible or sandboxed barely needs a human, however scary it looks; an irreversible, wide-blast action needs one even if it looks routine. This is where sandboxing the agent pays a second dividend: every action whose blast radius you’ve already bounded is an action you don’t have to spend human attention on. Containment and oversight trade off — the more you isolate, the less you must escalate.
- Let the agent self-escalate as difficulty climbs. An agent that raises its own rate of checking in as a task gets harder, ambiguous, or higher-stakes is giving a more honest signal than a single confidence score — and it concentrates the human budget on exactly the moments that warrant it.
- Protect the reviewer from the fatigue that breaks the gate. Cap the escalation rate, rotate reviewers, and treat a flooded queue as a safety incident, not a staffing nuisance — because a fatigued reviewer is a failed gate that still reports “approved.” Instrument it: if your approval rate is 99% and your median review time is two seconds, you don’t have an oversight layer, you have a latency tax that produces a false record of human judgment. Those two numbers are the early warning that the budget is overspent.
The throughline: a human-in-the-loop is not a safety feature you switch on; it’s a system you tune. We’ve made the same argument about the guardrail layer as a real product surface rather than a bolt-on, and it’s what lets you run agent loops unattended while you sleep — those loops only stay safe if the policy for when to wake a human is calibrated to a human who can’t be woken a hundred times a night.
Why this favors Southeast Asia
Here’s the part that matters for this region, and it’s the now-familiar shape. Designing a calibrated escalation policy — tiering actions, setting self-escalation triggers, budgeting reviewer attention — is pure systems judgment. It costs no GPUs, no frontier model, no capital. It’s the work of thinking carefully about which decisions actually need a human and how to keep that human sharp, and it runs entirely on engineering discipline.
That’s an edge, because the reflexive move everywhere else is to “add a human in the loop” and call it safety — which, as the research shows, can produce a fatigued reviewer rubber-stamping a queue, i.e. less safety wearing the costume of more. A team in Phnom Penh or Da Nang that instead treats oversight as a scarce budget — escalating less but reviewing better — builds agents that are genuinely safer to deploy on consequential, regulated work, and does it without spending a dollar more than a competitor who’s drowning their reviewers. As agents take on more irreversible actions, the teams that win trust won’t be the ones with the most oversight. They’ll be the ones who spent a finite human attention budget where it actually mattered.