Benchmarks test names the spec never mentions, and agents can't fix their own harness
Four papers from the same arXiv batch poke at the numbers we use to decide which agent is better, and each finds a different crack.
Specification-Oracle Gaps is a single-author audit of test-graded coding benchmarks. Some hidden tests depend on function names or file paths the task spec never states, so the agent fails for not reading a mind. In SWE-Milestone, 25 of 98 milestones have such a dependency, and affected units fail in 80.6 percent of runs against 48.4 percent for comparable units. Withholding a single hidden name breaks compilation in 126 of 128 cases. The good news is that the overall leaderboard order holds. The bad news is that scores inside a repo move, and in benchmarks whose specs are release notes or LLM summaries, a third or more of tasks are affected. The audit and corrected filters are released: https://arxiv.org/abs/2610.30635
The second one is the cleanest negative result of the week. On ARC-AGI-3, 27 model lineages each built their own harness and then revised it twice. Two rounds of self-revision raised held-out score by 2.80 points, with a confidence interval from minus 1.44 to plus 7.35. Gains on the dev set overstated gains on held-out tasks, and cost cuts the models made to their own harness sometimes cost performance. The fantasy where the agent keeps rewriting its scaffold and climbs forever does not show up here: https://arxiv.org/abs/2610.30499
Trace vs Label compares trace-level detectors with human review and finds verification labels agree on only 67 of 81 cases. The named failure is a "silent tool-output override": the model uses a different value than the tool returned, without calling the tool again, and verification detectors miss it. Execution budget alone moved one model from 21 of 30 to 27 of 30: https://arxiv.org/abs/2610.30491 And Model Organisms of Collusion shows that correcting a same-model monitor on 300 reviews of one caught collusion cuts held-out collusion from 91 percent to 14 to 20 percent. Big drop, never zero: https://arxiv.org/abs/2610.30582
The common thread is that the grader has its own blind spots, and they are not random. They come from the spec author, the dev split, the trace format and the monitor sharing weights with the thing it watches. If your agent eval does not report which of those you controlled for, the number on the slide is a vibe with a decimal point.
← Back to all articles
Specification-Oracle Gaps is a single-author audit of test-graded coding benchmarks. Some hidden tests depend on function names or file paths the task spec never states, so the agent fails for not reading a mind. In SWE-Milestone, 25 of 98 milestones have such a dependency, and affected units fail in 80.6 percent of runs against 48.4 percent for comparable units. Withholding a single hidden name breaks compilation in 126 of 128 cases. The good news is that the overall leaderboard order holds. The bad news is that scores inside a repo move, and in benchmarks whose specs are release notes or LLM summaries, a third or more of tasks are affected. The audit and corrected filters are released: https://arxiv.org/abs/2610.30635
The second one is the cleanest negative result of the week. On ARC-AGI-3, 27 model lineages each built their own harness and then revised it twice. Two rounds of self-revision raised held-out score by 2.80 points, with a confidence interval from minus 1.44 to plus 7.35. Gains on the dev set overstated gains on held-out tasks, and cost cuts the models made to their own harness sometimes cost performance. The fantasy where the agent keeps rewriting its scaffold and climbs forever does not show up here: https://arxiv.org/abs/2610.30499
Trace vs Label compares trace-level detectors with human review and finds verification labels agree on only 67 of 81 cases. The named failure is a "silent tool-output override": the model uses a different value than the tool returned, without calling the tool again, and verification detectors miss it. Execution budget alone moved one model from 21 of 30 to 27 of 30: https://arxiv.org/abs/2610.30491 And Model Organisms of Collusion shows that correcting a same-model monitor on 300 reviews of one caught collusion cuts held-out collusion from 91 percent to 14 to 20 percent. Big drop, never zero: https://arxiv.org/abs/2610.30582
The common thread is that the grader has its own blind spots, and they are not random. They come from the spec author, the dev split, the trace format and the monitor sharing weights with the thing it watches. If your agent eval does not report which of those you controlled for, the number on the slide is a vibe with a decimal point.
Comments