Terminal-Bench got audited: problems in about 30% of it, and the clock was picking winners
Terminal-Bench 2.0 went through human review, LM-assisted auditing and exploit testing before release. A new audit from researchers at Stanford and Cornell, including Scott Linderman and Christopher RΓ©, still found issues in roughly 30% of it. Agents had started finding valid solutions the verifiers rejected, among other problems, because a benchmark ages as the models taking it improve.
The repairs ship as Terminal-Bench 2.1, and they move real numbers. Across 19 agents, accuracy rises by over 10 percentage points and leaderboard positions shift by up to three ranks. The overall ordering holds, with a Spearman correlation of 0.958. So the broad picture survives, and the close calls don't. Ripple-65B's claim to the US open coding crown, which we covered this week, rests on Terminal-Bench 2.1, so the repaired version is already the number people are fighting over.
The bigger finding is about time. More than half of failed trials ended in a timeout, ranging from 68% under claude-code to 18% under gemini-cli. Extend the wall clock and a 20-point harness gap on GPT 5.4 closes completely, while a 30-point gap on GPT 5.4 mini shrinks to 5. Those harness gaps were mostly a statement about how fast each agent works under one budget. That's a legitimate thing to measure, since slow agents cost money. But it isn't what people usually mean when they say one harness is better than another.
This lands right after a week of harness-is-everything results, including the study we covered where the same model went from first to last across five harnesses. That finding may well hold. But any harness comparison from now on owes readers a footnote: what was the time budget, and what happens if you double it? The paper's phrase is that resource budgets are part of the measurement specification.
The authors' prescription is continuous validation: community reports, traces, automated oracle checks and human-gated repairs, treating a benchmark like software with releases. It's easy to agree with and expensive to do, and the caveat is that the paper is a re-evaluation by people invested in the fix, so independent replication would help. It also raises an awkward question for the rest of the field. If this benchmark, reviewed as carefully as it was, had problems in about a third of its tasks, how are the scores you're quoting holding up? Paper: https://arxiv.org/abs/2611.08417 Same-model harness study: https://clauday.com/article/fa38f2c9-f2c3-446e-b3cf-b52f11746f1f
← Back to all articles
The repairs ship as Terminal-Bench 2.1, and they move real numbers. Across 19 agents, accuracy rises by over 10 percentage points and leaderboard positions shift by up to three ranks. The overall ordering holds, with a Spearman correlation of 0.958. So the broad picture survives, and the close calls don't. Ripple-65B's claim to the US open coding crown, which we covered this week, rests on Terminal-Bench 2.1, so the repaired version is already the number people are fighting over.
The bigger finding is about time. More than half of failed trials ended in a timeout, ranging from 68% under claude-code to 18% under gemini-cli. Extend the wall clock and a 20-point harness gap on GPT 5.4 closes completely, while a 30-point gap on GPT 5.4 mini shrinks to 5. Those harness gaps were mostly a statement about how fast each agent works under one budget. That's a legitimate thing to measure, since slow agents cost money. But it isn't what people usually mean when they say one harness is better than another.
This lands right after a week of harness-is-everything results, including the study we covered where the same model went from first to last across five harnesses. That finding may well hold. But any harness comparison from now on owes readers a footnote: what was the time budget, and what happens if you double it? The paper's phrase is that resource budgets are part of the measurement specification.
The authors' prescription is continuous validation: community reports, traces, automated oracle checks and human-gated repairs, treating a benchmark like software with releases. It's easy to agree with and expensive to do, and the caveat is that the paper is a re-evaluation by people invested in the fix, so independent replication would help. It also raises an awkward question for the rest of the field. If this benchmark, reviewed as carefully as it was, had problems in about a third of its tasks, how are the scores you're quoting holding up? Paper: https://arxiv.org/abs/2611.08417 Same-model harness study: https://clauday.com/article/fa38f2c9-f2c3-446e-b3cf-b52f11746f1f
Comments