November 21, 2026ResearchBenchmarkAgents

METR's New Chart: Models Beat the Experts, Then Misjudge Their Own Work

METR can no longer find humans who beat the models at coding. That is the most quotable line in the evaluation nonprofit's Q4 2026 capability update, published Friday, which covers public models up to Opus 5.5 and GPT-6 Astra. The headline conclusion is more sober: "current public evidence is insufficient to rule out drastic AI R&D acceleration in the next couple of years." METR defines that as capabilities improving so fast that humans can no longer meaningfully understand, assess and react to catastrophic risks. Report: https://metr.org/blog/2026-11-20-q4-2026-capability-update/

On coding and optimization tasks, models usually match or beat skilled humans. When METR recruited its own baseliners, it "typically failed to find humans that outperform the current frontier models." Where 10 or more specialists had attempted a task, models placed at a very high percentile but not the top. On the combined old and new suites, time horizons are doubling roughly every 3.6 months. Read the fine print, though. Human baselines directly back only 9% of the task-time estimates. The rest are informed estimates, because paying experts for hundred-hour tasks is too expensive.

The new part is judgment. On tasks with no clear, fast feedback, agents "appear to have worse judgment than human experts." In one task models had to build an evaluation METR would actually use. Per MIT Technology Review, not one AI-designed eval was good enough for METR staff, though Opus 5.5 reached "borderline above the bar," up from "clearly below" for every model a year ago. Models also fell short at rating critiques of philosophical arguments. The detail that should worry anyone running agent loops: model judges overestimate the quality of output made by a similarly strong or stronger model, and in one experiment swarms overestimated their own performance. METR's explanation: poor judgment caps performance because the model "only has its own judgement to optimize against."

There is also a new measure called Lifecycle Cost: how expensive a task is to turn into an RL environment. Doubling it costs about as much as halving the agent's time horizon, yet progress is similar on cheap and costly tasks. METR also says openly that its easy, containerized evals look a lot like the RL environments labs train on, so they may overestimate real-world capability.

My read: the bottleneck in self-improvement is no longer doing the work, it is grading it. A model that overrates its own solution will happily optimize toward its own blind spot, and a swarm that grades itself compounds the error. For anyone building agent pipelines the lesson is concrete: automate the steps with a hard check, and put an outside judge, human or otherwise, on every step where "good" is a matter of taste.
← Previous
A Riemann Hypothesis Proof Made With Claude, and the Only Copy Is Lean
Next β†’
Inkling 2 Mini: A 20GB Open Coder That Thinks in Loops, With a Check on What It Hides
← Back to all articles

Comments

Loading...
>_