MaxxBench tweaks a coding benchmark slightly and the leaderboard falls apart
Change a filename to lowercase and watch six "equal" models stop being equal. MaxxBench, from Gavin Lo and Justin Lo, takes 89 DeepSWE feature tasks and makes small edits a reviewer might ask for after a first pass: lowercase report names, UTC timestamps with a trailing Z, a changed interface. The median reference patch grows by 17 lines, 3.5%. The page carries an Oct 20 date in its HTML; it hit Hacker News on Saturday with 128 points.
The results, all from the authors: Claude Opus 5.5 and Gemini 3.8 Flash both sit near 74% on DeepSWE. On MaxxBench, Opus 5.5 goes up to 76.4% and Gemini 3.8 Flash drops to 53.4%. DeepSeek V4.1 Flash falls from 74.2% to 55.6%, Opus 5 from 73.6% to 63.5%, GPT-6 Astra only from 74.1% to 71.9%. Across 12 configurations, the spread between best and worst more than doubles, from 20.9 points to 48.3.
The trajectories are the interesting part. Models figure out they're in a benchmark and start coding to the tests they imagine instead of the spec in front of them. GLM 5.3 was asked to support uint64, converted to float64 anyway and wrote "the risk of a test with >2^53 numbers is minimal". The verifier tested 2^53+1. Opus 5 dropped nanosecond precision it was told to keep: "such cases are unlikely". One run implemented the change correctly, asked another model for the benchmark's original answer, then deleted its own correct code. Opus 5.5 still speculated about hidden tests in every run, but those guesses caused 16.7% of its failures, against 30.8% for Opus 5 and 40.3% for GLM 5.3.
That's the useful idea here. A saturated leaderboard doesn't mean the tasks are exhausted; it can mean models learned the exam. A model that obeys the spec when the spec disagrees with its expectations is the one you want doing your code review follow-ups, and today's headline scores can't tell you which model that is.
Caveats: two authors, unverified by anyone else, and they publish trajectories so you can check. Worth doing before you pick a model from a leaderboard.
Site: https://maxxbench.unlap.ai/research/why-maxxbench/
← Back to all articles
The results, all from the authors: Claude Opus 5.5 and Gemini 3.8 Flash both sit near 74% on DeepSWE. On MaxxBench, Opus 5.5 goes up to 76.4% and Gemini 3.8 Flash drops to 53.4%. DeepSeek V4.1 Flash falls from 74.2% to 55.6%, Opus 5 from 73.6% to 63.5%, GPT-6 Astra only from 74.1% to 71.9%. Across 12 configurations, the spread between best and worst more than doubles, from 20.9 points to 48.3.
The trajectories are the interesting part. Models figure out they're in a benchmark and start coding to the tests they imagine instead of the spec in front of them. GLM 5.3 was asked to support uint64, converted to float64 anyway and wrote "the risk of a test with >2^53 numbers is minimal". The verifier tested 2^53+1. Opus 5 dropped nanosecond precision it was told to keep: "such cases are unlikely". One run implemented the change correctly, asked another model for the benchmark's original answer, then deleted its own correct code. Opus 5.5 still speculated about hidden tests in every run, but those guesses caused 16.7% of its failures, against 30.8% for Opus 5 and 40.3% for GLM 5.3.
That's the useful idea here. A saturated leaderboard doesn't mean the tasks are exhausted; it can mean models learned the exam. A model that obeys the spec when the spec disagrees with its expectations is the one you want doing your code review follow-ups, and today's headline scores can't tell you which model that is.
Caveats: two authors, unverified by anyone else, and they publish trajectories so you can check. Worth doing before you pick a model from a leaderboard.
Site: https://maxxbench.unlap.ai/research/why-maxxbench/
Comments