Sixty percent of the SWE-bench tasks nobody solves have broken tests
Buried in the SWE-Bench ProMax paper is a finding that should change how you read every coding-agent leaderboard: nearly 60 percent of the unsolved instances in SWE-bench Verified contain flawed tests. Not hard tests. Flawed ones β too narrow, too broad, or simply wrong. When a model fails those, it isn't failing at software engineering, it's failing at guessing what a bad test wanted.
The other half of the indictment is worse. Frontier models can reproduce the gold patches verbatim. They've seen them. So the ceiling of that benchmark is partly memorization and the floor is partly noise, and the band in between is where everyone has been publishing percentage points for two years.
ProMax is the replacement, from Yuling Shi and fourteen co-authors, published at COLM 2026 and posted August 10. 170 instances across Python, Java, TypeScript, Go, C, C++ and Rust. The tasks are refactoring, not bug-fixing, which kills the memorized-patch problem because there is no canonical single diff. And they are big: 11.4 modified files and 261.6 lines changed per instance on average. Issue descriptions rewritten by hand, test suites manually reviewed.
Best model resolves 41.2 percent. That is a real number on real work, and it is a long way below the numbers that get tweeted. Multi-file, multi-language refactoring is exactly the job people claim agents already do β and the honest measurement says they finish it four times out of ten. Benchmark and data on Hugging Face, CC-BY-4.0.
Paper: https://arxiv.org/abs/2608.09802
← Back to all articles
The other half of the indictment is worse. Frontier models can reproduce the gold patches verbatim. They've seen them. So the ceiling of that benchmark is partly memorization and the floor is partly noise, and the band in between is where everyone has been publishing percentage points for two years.
ProMax is the replacement, from Yuling Shi and fourteen co-authors, published at COLM 2026 and posted August 10. 170 instances across Python, Java, TypeScript, Go, C, C++ and Rust. The tasks are refactoring, not bug-fixing, which kills the memorized-patch problem because there is no canonical single diff. And they are big: 11.4 modified files and 261.6 lines changed per instance on average. Issue descriptions rewritten by hand, test suites manually reviewed.
Best model resolves 41.2 percent. That is a real number on real work, and it is a long way below the numbers that get tweeted. Multi-file, multi-language refactoring is exactly the job people claim agents already do β and the honest measurement says they finish it four times out of ten. Benchmark and data on Hugging Face, CC-BY-4.0.
Paper: https://arxiv.org/abs/2608.09802
Comments