Agents can write the code. Three papers show they still can't write the checker
If agents are going to grade agents, someone has to check the graders. Three papers from this week's batch did, and the scores are humbling.
VerifierBench (arXiv 2610.28679) asked 19 models to write verifiers for 222 tasks, then tested those verifiers against 4,993 hidden cases. Normalized balanced accuracy ranged from 10.6 to 62.1. The most common failure was not a lenient checker but a broken one: most false rejections came from verifiers that accept no valid solution at all. That error survives self-testing, because a model checking its own verifier against its own examples never notices that nothing could ever pass. Link: https://arxiv.org/abs/2610.28679
VeeReview (arXiv 2610.29143) is an honest negative. The authors matched 1,454 agent-written and human-written pull requests across 741 repos and pre-registered an LLM-judge instrument to measure how much verification work went into each. It failed its own calibration bar: agreement of 4 to 60 percent against a required 80, with a kappa of 0.154. Their reading is that the judge partly measures how a PR is presented rather than the checking behind it. A well-written PR description looks like rigor to a model. Link: https://arxiv.org/abs/2610.29143
FrontierCS 2 (arXiv 2610.29932) tries to dodge that trap with 48 research tasks graded by a private evaluator, built by deleting a known contribution and asking the agent to recover it. The best agent, Claude Opus 5.5, passes 58 percent of runs, and 16 tasks have zero passes from any model. The authors' line is worth stealing: a higher score is not the same as a pass. Link: https://arxiv.org/abs/2610.29932
Generation got cheap this year and verification did not. Most agent pipelines now quietly assume an LLM can be the judge, the reviewer or the test writer. These papers say that works only when the verifier is built and checked independently of the thing it grades, ideally against cases the builder never saw. That is the same lesson METR's audit of OpenAI's monitor taught at the lab level this week, shown here at the scale of one unit test.
← Back to all articles
VerifierBench (arXiv 2610.28679) asked 19 models to write verifiers for 222 tasks, then tested those verifiers against 4,993 hidden cases. Normalized balanced accuracy ranged from 10.6 to 62.1. The most common failure was not a lenient checker but a broken one: most false rejections came from verifiers that accept no valid solution at all. That error survives self-testing, because a model checking its own verifier against its own examples never notices that nothing could ever pass. Link: https://arxiv.org/abs/2610.28679
VeeReview (arXiv 2610.29143) is an honest negative. The authors matched 1,454 agent-written and human-written pull requests across 741 repos and pre-registered an LLM-judge instrument to measure how much verification work went into each. It failed its own calibration bar: agreement of 4 to 60 percent against a required 80, with a kappa of 0.154. Their reading is that the judge partly measures how a PR is presented rather than the checking behind it. A well-written PR description looks like rigor to a model. Link: https://arxiv.org/abs/2610.29143
FrontierCS 2 (arXiv 2610.29932) tries to dodge that trap with 48 research tasks graded by a private evaluator, built by deleting a known contribution and asking the agent to recover it. The best agent, Claude Opus 5.5, passes 58 percent of runs, and 16 tasks have zero passes from any model. The authors' line is worth stealing: a higher score is not the same as a pass. Link: https://arxiv.org/abs/2610.29932
Generation got cheap this year and verification did not. Most agent pipelines now quietly assume an LLM can be the judge, the reviewer or the test writer. These papers say that works only when the verifier is built and checked independently of the thing it grades, ideally against cases the builder never saw. That is the same lesson METR's audit of OpenAI's monitor taught at the lab level this week, shown here at the scale of one unit test.
Comments