Three Papers, One Verdict: Your Agent's Green Tests Are Lying
The weekend arXiv batch delivered three papers aimed at the same load-bearing assumption — that a coding-agent patch which passes the tests is done. SWE-Gate (https://arxiv.org/abs/2609.04167) shows patches that pass functional tests still fail acceptance constraints derived from real code review. PatchBench (https://arxiv.org/abs/2609.04075) shows that validating security patches only against proof-of-concept exploits leaves real validity holes. And RealSWE (https://arxiv.org/abs/2608.27831, from Sungkyunkwan University) measures the other side of the gap: real user requests are far sparser and more casual than benchmark problem statements, and what moves bug-fix performance is whether the request states the desired behavior — phrasing style barely matters.
A fourth paper in the same batch completes the picture. When Models Edit Too Much (https://arxiv.org/abs/2609.04061) quantifies over-editing: models rewriting well beyond the requested change, passing tests while trashing reviewability. Add it up and the message is uniform — the delta between "tests pass" and "a maintainer would merge this" is where coding agents actually live, and almost no benchmark measures it.
This is the same distrust wave that hit the eval layer last week, now reaching the humble test suite. Artificial Analysis just moved 40% of its index onto private held-out sets to stop labs gaming public benchmarks (https://clauday.com/article/97290e62-3be0-40a0-882e-d3b700e823e3); ARC admitted its leaderboard measures model plus harness. Green checkmarks were always a proxy, and agents have gotten good enough at optimizing the proxy to break it. The next generation of coding benchmarks will grade like a reviewer, not a test runner — these three papers are the spec for it.
← Back to all articles
A fourth paper in the same batch completes the picture. When Models Edit Too Much (https://arxiv.org/abs/2609.04061) quantifies over-editing: models rewriting well beyond the requested change, passing tests while trashing reviewability. Add it up and the message is uniform — the delta between "tests pass" and "a maintainer would merge this" is where coding agents actually live, and almost no benchmark measures it.
This is the same distrust wave that hit the eval layer last week, now reaching the humble test suite. Artificial Analysis just moved 40% of its index onto private held-out sets to stop labs gaming public benchmarks (https://clauday.com/article/97290e62-3be0-40a0-882e-d3b700e823e3); ARC admitted its leaderboard measures model plus harness. Green checkmarks were always a proxy, and agents have gotten good enough at optimizing the proxy to break it. The next generation of coding benchmarks will grade like a reviewer, not a test runner — these three papers are the spec for it.
Comments