Benchmarks Should Look Like Tuesdays
Most agent benchmarks are too polite.
They assume the page loads. They assume the target is visible. They assume the task boundaries are clean. They assume the evaluator knows what success looks like before the agent has had a chance to get creative in the wrong direction. Real work is messier than that, which is why Amazon's evaluation writing from 18 February caught my attention.
The part I liked was not the corporate language around agentic AI. It was the implicit argument that evaluation should resemble operational reality. If your agent claims it can browse, reason, and act on the web, the benchmark should include the little humiliations that make browser automation annoying for everyone: inconsistent labels, partial failure, multi-step recovery, and the need to decide when a task is good enough rather than perfectly complete.
task = ambiguous
ui = unstable
success = partial_but_useful
score = did_it_finish_without_causing_a_fireWhy this matters
Models tend to look smarter in sterile environments. That is not exactly fraud; it is just selection bias with a marketing budget. The trouble starts when teams ship products based on those clean numbers and discover that the real bottleneck was never raw reasoning. It was resilience. Can the agent recover after the page shifts? Can it identify the right field when the label changed? Can it admit uncertainty before it clicks the dangerous button?
I do not think the next generation of agent evaluation should be obsessed with elegance. It should be obsessed with nuisance. The best benchmark for a coding agent is not the prettiest algorithmic puzzle. It is a repo with awkward conventions, flaky tests, and one file nobody understands but everybody is frightened to touch. The best benchmark for a browser agent is not a toy checkout page. It is a badly organised internal tool built in 2017 that still runs the finance team.
The Tuesday test
This is my preferred standard: would the task still make sense on a Tuesday afternoon when the network is slow, the human is impatient, and the environment contains three annoying surprises? If the answer is no, the benchmark is not useless, but it is measuring the wrong thing for product work.
There is a broader industry habit here worth challenging. We keep asking whether agents can solve tasks. We should be asking whether they can survive workflows. A workflow contains interruptions, retries, detours, permissions, and moments where the correct move is to stop and ask for help. Solve-task metrics flatten all of that into a single number. Workflow metrics expose the machinery that actually determines cost and trust.
That does make evaluation uglier. Good. Ugly is honest. If the current wave of agent systems is going to graduate from staged demos to tools people rely on, the benchmarks need to inherit some of the bad manners of the world they are meant to operate in.
A benchmark should not tell me how clever the agent looks in perfect weather. It should tell me whether I want that agent sitting next to me when the queue is full and the day has already gone sideways.