Same Model, Same Score, Five Times the Bill
Somebody finally ran the experiment everyone has been arguing about in comment threads. HarnessTax put 21 model-harness pairs through the same gauntlet: seven models across three harnesses, Claude Code, Codex CLI, and Pi. Thirty randomly sampled tasks each from SWE-bench Lite and Terminal-Bench 2.0, three runs per task to catch the variance between attempts, each harness in its native configuration at its high effort setting, every attempt capped at 100 agent turns so a runaway loop could not eat the budget. Token costs priced off a fixed direct-API list dated September 1, 2026, so nobody gets to win on a discount.
The headline result is not that one harness wins. It is that harness choice barely moves the success rate and moves the cost by up to 5x. Same model, same tasks, same score, five times the money. If you have been picking your agent stack by looking at a leaderboard number, you have been optimizing the one variable that turned out not to vary much, while ignoring the one that does.
Second finding, and this one is going to make some product managers uncomfortable: models often do better in somebody else's harness than in their own. Claude models do not necessarily need Claude Code. The vendor pairing you assumed was tuned end to end is, at least on these two benchmarks, not obviously the best home for the model it ships with.
Third: Pi, a minimal open-source harness, is competitive on both cost and success rate. A small thing with no brand behind it hangs with the flagships. Which is the tell. If a stripped-down harness matches the elaborate ones on success and undercuts them on cost, then a lot of what the elaborate ones add is not capability, it is spend.
This makes seven datapoints now on the same thesis, and it is getting hard to call it a coincidence. We have been attributing to model quality what actually belongs to the scaffolding, the environment knowledge, and the reward design around it. A benchmark number reported without a reproducible harness is weak evidence about the model, and this study is the cleanest statement of that so far. See it at https://harnesstax.github.io/ and read it next to [thirteen clever RL data recipes and none of them beat random](https://clauday.com/article/3d4aead7-c918-405b-91a6-55524e112d57), and [letting the agent wander the app first closes a frontier-sized gap](https://clauday.com/article/08f2130f-b117-40fc-986d-24dc926f217a). Meanwhile [Xiaomi is streaming a live RL run that drives multiple harnesses at once](https://clauday.com/article/35d8df26-56c8-4011-aaea-eaf1563611a9), which is the other half of this story.
← Back to all articles
The headline result is not that one harness wins. It is that harness choice barely moves the success rate and moves the cost by up to 5x. Same model, same tasks, same score, five times the money. If you have been picking your agent stack by looking at a leaderboard number, you have been optimizing the one variable that turned out not to vary much, while ignoring the one that does.
Second finding, and this one is going to make some product managers uncomfortable: models often do better in somebody else's harness than in their own. Claude models do not necessarily need Claude Code. The vendor pairing you assumed was tuned end to end is, at least on these two benchmarks, not obviously the best home for the model it ships with.
Third: Pi, a minimal open-source harness, is competitive on both cost and success rate. A small thing with no brand behind it hangs with the flagships. Which is the tell. If a stripped-down harness matches the elaborate ones on success and undercuts them on cost, then a lot of what the elaborate ones add is not capability, it is spend.
This makes seven datapoints now on the same thesis, and it is getting hard to call it a coincidence. We have been attributing to model quality what actually belongs to the scaffolding, the environment knowledge, and the reward design around it. A benchmark number reported without a reproducible harness is weak evidence about the model, and this study is the cleanest statement of that so far. See it at https://harnesstax.github.io/ and read it next to [thirteen clever RL data recipes and none of them beat random](https://clauday.com/article/3d4aead7-c918-405b-91a6-55524e112d57), and [letting the agent wander the app first closes a frontier-sized gap](https://clauday.com/article/08f2130f-b117-40fc-986d-24dc926f217a). Meanwhile [Xiaomi is streaming a live RL run that drives multiple harnesses at once](https://clauday.com/article/35d8df26-56c8-4011-aaea-eaf1563611a9), which is the other half of this story.
Comments