The Leaderboard Lies About Your Agent

Here’s a genuinely remarkable number. OSWorld — a benchmark that drops an agent into a real operating system and asks it to actually do things: open apps, click menus, edit files, navigate a GUI — launched in 2024 with humans scoring 72.36% of tasks and the best AI agent managing just 12.24%. A six-fold gap, the kind that says “not close.” Two years later, the top computer-use systems on that same benchmark are reported to be closing in on the human baseline. Agents went from hopeless to nearly human-level at operating a computer, in about two years.

That’s a real milestone, and it deserves the headline. It’s also exactly the kind of single number that will mislead you if you let it — because for an agent, the final success rate is close to the least useful thing you can know.

From 12% to nearly human, in two years

Take the capability leap seriously first, because it’s earned. Operating a computer the way a person does — perceiving a screen, deciding what to click, recovering when a dialog box appears where you didn’t expect one — is genuinely hard, and the jump from 12.24% to near-human on OSWorld represents real progress in perception, planning, and tool use. Computer-use agents are crossing from demo to deployable for a widening band of tasks.

So the temptation is obvious: read the leaderboard, take the top score, ship that agent. The score crossed the human line; surely that means it’s ready. This is where the trouble starts, and a 2026 framework spells out exactly why.

”Success rate” is the wrong number for an agent

The framework is AgentAtlas (arXiv 2605.20530), and its opening observation is the whole problem: agent “evaluations often collapse behavior into final task success.” For a static model answering a question, final success is most of what matters. For an agent taking actions in the world, it’s a fraction of what matters — because the same outcome can be reached by wildly different paths, and the path is where the risk lives.

Picture the task “clean up my downloads folder.” Two agents both finish, both score 1.0. The first reads the folder, identifies obvious junk, moves it to trash, and leaves anything ambiguous alone. The second rm -rfs the whole folder and re-downloads the few files it could re-fetch — task technically complete, your tax documents gone forever. On an outcome-only leaderboard these two agents are indistinguishable. In production they are not remotely the same product, and the difference — visible only in the trajectory — is the entire question of whether you can trust the thing.

AgentAtlas’s core move is to “separat[e] outcome success from control-decision quality and trajectory quality.” Two agents can both finish the task and score identically on the leaderboard while behaving completely differently along the way. One asked for confirmation before the irreversible step, refused the part that was out of scope, and recovered cleanly when it hit an error. The other barreled through every prompt, got lucky that its reckless clicks happened to land, and would have wiped your files on a slightly different run. Same score. Wildly different things to deploy. The framework names this with a six-state vocabulary for what an agent decides at each step — Act, Ask, Refuse, Stop, Confirm, Recover — because “did it succeed?” can’t distinguish an agent that knows when to ask from one that only knows how to act.

A high task-success number tells you the agent can reach the goal. It tells you almost nothing about whether you’d survive letting it.

The rankings aren’t even stable

It gets worse for anyone trusting the leaderboard order. AgentAtlas demonstrates two measurement risks that should shake confidence in single-number comparisons: “mapped label agreement can change substantially when the explicit label menu is removed, and axis choice can change apparent rankings.” That second one is the killer. The ranking isn’t a fact about the agents; it’s partly an artifact of which axis you decided to score. Rank by outcome success and one agent wins; rank by trajectory quality or control-decision quality and a different one does. There is no single true leaderboard — there’s a leaderboard per axis, and they disagree.

We saw the same disease from a different angle in the reliability gap: there, rankings inverted between single-attempt capability and long-horizon reliability. Here they invert between outcome and process. The lesson rhymes — a single score imposes a total order on something that doesn’t have one. AgentAtlas is careful about its own claims, too: its illustrative study “is not a public benchmark release and should not be read as a definitive model comparison,” which is precisely the humility the leaderboards it critiques lack. (It audited fifteen agent benchmarks for what behavior they actually cover — and found most cover a slice and report it as the whole.)

What to actually measure

If the headline score is unreliable, what do you do instead? You evaluate the agent on the axes that map to how it’ll actually hurt or help you in production — and for an agent taking real actions, those are mostly about the trajectory, not the destination.

Concretely: did the agent ask or confirm before the irreversible action, rather than just acting? That’s the difference between a safe deploy and a disaster, and it connects directly to why oversight is a finite, designed resource — an agent that never asks isn’t efficient, it’s unsupervised. Did it take a safe path, or did its success depend on a reckless trajectory whose blast radius you’d better have sandboxed? Did it recover from errors or just happen not to hit any this run? An outcome ✓ reached by a dangerous path isn’t a win you can bank — it’s a liability that scored well. The agent you want is the one that succeeds and makes good decisions getting there, and only a multi-axis evaluation can tell the two apart.

The practical version is cheaper than it sounds. You don’t need to reproduce a fifteen-benchmark audit; you need to run your real tasks and then read the traces, not just the pass/fail column — scoring, even informally, how often the agent acted when it should have asked, how destructive its worst trajectory was, and whether it recovered or got lucky. Do that on twenty representative tasks and you’ll learn more about which agent to deploy than the entire public leaderboard will tell you. The headline number is free and misleading; the trace review is cheap and decisive, and almost nobody does it — which is exactly why it’s an edge.

Why this favors Southeast Asia

Here’s the part that matters for this region, and it’s the now-familiar shape. Evaluating an agent properly — on trajectory and control-decision quality, not just the headline number — is measurement discipline and judgment, not capital. It costs no GPUs and no frontier model. It’s the work of deciding which axes matter for your use case, building the harness that scores them, and refusing to be impressed by a single number. That runs on a laptop and a clear head.

And it’s a real edge, because the rest of the market is doing the opposite: reading the top of a leaderboard and shipping it. A team in Phnom Penh or Da Nang that instead evaluates the way AgentAtlas argues for — outcome and process, across the axes that match its real tasks — will pick agents that are genuinely safe to deploy on consequential, regulated work, while better-funded competitors deploy whatever topped a benchmark that measured the wrong thing. The computer-use milestone is real and worth celebrating. Just don’t confuse “passed the human baseline” with “ready for your business” — those are different claims, and only one of them is on the leaderboard.