kw-bench: Claude Code and Codex scored zero away from their home models
Take the two flagship lab harnesses, Claude Code and Codex, and run them on someone else's model. On one hard real-world task, both delivered nothing.
That is the result of kw-bench, a single-author paper from last Wednesday's arXiv batch by George Juraj Salapa. The task is a five-year VAT reconstruction across three generations of SAP, graded deterministically with no LLM judge. All four harnesses ran on DeepSeek v4.1 Flash for two hours. OpenCode scored 0.542 and the minimal micro-cc 0.528. Claude Code and Codex, "away from their home model, delivered nothing (0.00 each)." Put a frontier model underneath and the picture flips: micro-cc and Claude Code on Claude Opus 5.5 scored 0.9986 and 1.000, in 22 and 25 minutes. The paper's closing line: "below the frontier, the harness decides how far a model gets."
A second paper, Your Policy Is Secretly a Harness, shows the same thing from the training side. On MiniWoB++, reversing the order of the observation fields changes no optimal decision, yet a trained model's success falls from 85.0% to 19.2%. The authors split deployed performance into the ceiling the harness allows minus the mismatch between harness and policy. On Terminal-Bench Lite, ceiling minus mismatch reproduces the accuracy ranking, and neither term alone does. The model learned the harness as much as the task.
Surge AI's new DAYJOB: Tax benchmark is a reminder of how far professional work still is. Fifty environments across eight areas of tax practice, 1,789 workspace files, up to 123 per assignment. The best pass rate is Claude Sonnet 5.5 at 10.8%, then Fable 5.1 at 9.6% and Opus 5.5 at 9.2%.
Our read: lab harnesses are tuned for their own model, and that tuning becomes a dependency. If you plan to swap in a cheaper or open model, test the harness too, or the leaderboard score you were counting on goes to zero. kw-bench is one study by one author, so treat it as a strong signal, not settled fact.
kw-bench: https://arxiv.org/abs/2610.36405 Your Policy Is Secretly a Harness: https://arxiv.org/abs/2610.37255 DAYJOB: Tax: https://surgehq.ai/blog/dayjob-tax
← Back to all articles
That is the result of kw-bench, a single-author paper from last Wednesday's arXiv batch by George Juraj Salapa. The task is a five-year VAT reconstruction across three generations of SAP, graded deterministically with no LLM judge. All four harnesses ran on DeepSeek v4.1 Flash for two hours. OpenCode scored 0.542 and the minimal micro-cc 0.528. Claude Code and Codex, "away from their home model, delivered nothing (0.00 each)." Put a frontier model underneath and the picture flips: micro-cc and Claude Code on Claude Opus 5.5 scored 0.9986 and 1.000, in 22 and 25 minutes. The paper's closing line: "below the frontier, the harness decides how far a model gets."
A second paper, Your Policy Is Secretly a Harness, shows the same thing from the training side. On MiniWoB++, reversing the order of the observation fields changes no optimal decision, yet a trained model's success falls from 85.0% to 19.2%. The authors split deployed performance into the ceiling the harness allows minus the mismatch between harness and policy. On Terminal-Bench Lite, ceiling minus mismatch reproduces the accuracy ranking, and neither term alone does. The model learned the harness as much as the task.
Surge AI's new DAYJOB: Tax benchmark is a reminder of how far professional work still is. Fifty environments across eight areas of tax practice, 1,789 workspace files, up to 123 per assignment. The best pass rate is Claude Sonnet 5.5 at 10.8%, then Fable 5.1 at 9.6% and Opus 5.5 at 9.2%.
Our read: lab harnesses are tuned for their own model, and that tuning becomes a dependency. If you plan to swap in a cheaper or open model, test the harness too, or the leaderboard score you were counting on goes to zero. kw-bench is one study by one author, so treat it as a strong signal, not settled fact.
kw-bench: https://arxiv.org/abs/2610.36405 Your Policy Is Secretly a Harness: https://arxiv.org/abs/2610.37255 DAYJOB: Tax: https://surgehq.ai/blog/dayjob-tax
Comments