Same Model, Two Bills: The Week the Harness Finally Got Measured
Anthropic deleted 80% of the system prompt that Claude Code reads before every task, re-ran the coding tests, and found no measurable drop. That line sat in the State of AI report this week while five unrelated groups published numbers that say the same thing from the other direction: hold the model fixed, change the plumbing around it, and the result moves more than a model generation does.
Start with the biggest spread. LobeHub let its agent rewrite its own harness for seven days and then ran the result on FrontierHarness with Kimi K3 underneath. It passed 21 of 30 tasks, first among 13 harnesses, at $1.68 per successful task. The same benchmark put Codex at $3.47 and Claude Code at $18.34. LobeHub itself says the comparison includes selected retries and different execution environments and should not be read as a leaderboard. Fine. Even if you cut the gap in half twice, it is still a 2.7x difference in the cost of a solved task with the model held constant. Nobody shipped a model this year that was 2.7x cheaper per solved task than the one before it.
Alibaba open-sourced the code reviewer it ran internally for two years, and the benchmark table is the clearest single-variable test of the week. Same Claude model on both sides, 200 real pull requests. Through Open Code Review: precision 37.8%, about 352,000 tokens per review, one minute six seconds. Through Claude Code asked to review the PR: precision 15.9%, about 2.06 million tokens, five minutes thirty-eight. The difference is that plain code picks which files get reviewed and which rules apply, bundles related files, and sends them to parallel sub-agents, while the model only reads and judges. Recall did not improve, 11.7% against 12.7%. The harness bought fewer false alarms at a sixth of the tokens, not more bugs found. That trade is a design decision, and it was made in the pipeline, not in the weights.
Prime Intellect's story is the extreme case. Prime Agent ported itself from TypeScript to Rust: 2,209 agents, 228.7 billion tokens, two weeks, 14x faster cold start, memory under a fifth. The part worth copying is not the scale. It is that the humans mostly wrote verification, not code: screen-by-screen comparison of the two versions on the same script, matching session logs and model requests from the same input, protocol message types checked against the old build, and a feature audit sorted into match, partial and missing. Every task went planner, implementer, reviewer, verifier, with the reviewer on a different model and a fresh context because, in their words, the agent that wrote the code grades its own code softly. GLM-5.3 did the inference. The orchestrator ran on two 8-core CPU boxes. The model was a commodity in this story. The checks were the product.
NVIDIA's SoL-Pi paper ran the same experiment on the harness itself: a research agent watched coding agents work, proposed changes, and kept only the ones that lowered cost without hurting scores. Four mechanisms survived, action fusion, online context compaction, observation packing, and an evidence-preserving reducer for delegated reading. Result: 54.3% cheaper than Claude Code on Opus 5, 50% cheaper than Codex on GPT-5.6 Sol, scores in the same range. Same model, half the bill, and the four things that worked are all about what the model is shown and when, not what the model knows.
Then there is the smallest example, which may be the most honest. A Ubuntu operator had a maintenance program that passed 38 simulated tests and then failed on the real box, returning ACTIVE_SKIP every time. OpenClaw could do the same operation on the same machine. The cause was not the model. DeepSeek Harness ran commands inside a bwrap sandbox with its own PID namespace and no path to the user systemd bus, so the agent literally could not see host processes or service state. Two harnesses on one Linux box were looking at two different operating systems. The fix was not more permissions; it was letting the agent write the program and letting the host's own systemd run it.
Put those five next to each other and the pattern is hard to argue with. For two years the public conversation about agents has had one knob, which model, and one metric, the benchmark score. This week the other knob got turned on purpose, with the model held still, and in every case the harness moved cost by a factor of two or more and moved quality by whatever the designer chose to trade for. Think of it as engine versus car. The engine suppliers spent two years telling you horsepower. Then five teams put the same engine in different chassis and one took the lap record while another halved the fuel bill, and it turns out the chassis was never a rounding error.
Why is the harness suddenly measurable when it was folklore a year ago? Because harnesses fail deterministically, and deterministic failures have diffs. Flowness, a one-person multi-agent system with 140 billion Claude tokens on record, published a postmortem in which one task was dispatched to nine background Claude Code sessions in a row. The automatic redispatch cap was 3. The retry counter reached 8. The cap never fired. One acceptance threshold of 1,500 milliseconds came from a cost model in a report and had never been measured; the real number was 244. Every check in the loop asked "can we dispatch now?" and none asked "is this different from last time?" That is not a model problem. You cannot prompt your way out of it and you do not need to. You fix the code and the bug is gone. Model failures are statistical and you manage them. Harness failures are bugs and you close them. That asymmetry is why harness improvements compound and prompt tricks expire.
The cost side has the same shape. One Claude Code user worked out that a late model switch at 300K tokens of context costs about $1.50 just to rebuild cache, while a clean 20K handoff costs about $0.10, and that the Opus-to-Sonnet price ratio shrinks from 1.88x at 20K context to 1.26x at 400K, so the question "which model is cheaper" has no answer until you know how long your loop runs. Another measured that a single tool definition, Claude Code's Artifacts, weighs about 17,800 tokens, nearly 2% of a 1M window and almost a fifth of the whole bootstrap, loaded on every fresh session. A third found that Claude Code's subagents default to a five-minute cache while the main agent gets an hour, and the day split in two over whether to change it: three posts said set it to an hour, one said that burns weekly limits up to 15% faster because subagents rarely idle, and the careful answer was "depends on whether the subagent waits on builds." None of that is about intelligence. It is about who gets to see what, when, and how long the cache stays warm. That is the harness, and it now has a price list.
There is a catch, and this week supplied it too. Pine Computer, a cloud computer built for agents, posted the week's most-circulated number: GPT-5.6 Luna on Pine scored 78.3% on SaaS-Bench checkpoints against 74.3% for Opus 5 with Claude Code, at $1.02 per task versus $26.50. A smaller model on a better environment beat a bigger model on a worse one, at a 26x cost difference. Read the next line of the same post: end-to-end task completion was 27.4% on Pine against 31.1% for Claude Code. The harness that wins the checkpoint metric loses the finish-the-job metric. Which number you optimize is a choice the harness designer makes for you, and most benchmark posts only show the one that flatters.
The second catch comes from the self-improvement crowd itself. A builder reviewing MIT and Sakana's SIFT, which lets an agent propose harness changes and uses an LLM judge to rank them before paying for full benchmark runs, named the obvious risk: a change that improves SWE-bench can make the agent worse at real development, because the harness overfits to the benchmark faster than a model does. LobeHub's own caveats say the same thing more quietly. The 17-researcher paper that took an agent apart into six harness components, observation, context, control loop, action, state, verification, is useful precisely because it gives you six places to look for the overfit. The number to demand from any harness claim is not pass rate. It is cost per verified task across projects the harness has never seen. Nobody is publishing that yet. The first team that does will have the only honest leaderboard in the category.
And the counter-case deserves a seat. Someone who has used Claude Code for a long time asked why anyone needs subagents at all: a single agent loop works in almost every case, keeps the harness simple and limits token use, and nobody has actually measured several single-session Claudes against one session with subagents. That is a fair challenge to the whole week. More harness is not better harness. Alibaba's reviewer won by removing decisions from the model, not adding agents around it. The Rust port won on verification, not on 2,209 workers. The Ubuntu fix was to let the agent do less. Harness engineering, done well, mostly looks like subtraction.
So what moved this week? The prompt stopped being the moat. Anthropic could delete 80% of its own instruction list because the smarter the model gets, the less you need to nag it, and the State of AI report says the part that lasts is the plumbing: the tools the agent can reach, the sandbox it runs in, the monitoring that shows what it did. If your product is a clever pile of prompts around someone else's model, the next model release deletes your edge. If it is the pipeline that decides what the model sees, checks what it claims, and pays for only the turns that mattered, every model release makes it more valuable, because the same chassis gets a better engine for free. Five groups measured that this week. The number was never a rounding error. It just took until now for anyone to hold the engine still long enough to read it.
← Back to all articles
Start with the biggest spread. LobeHub let its agent rewrite its own harness for seven days and then ran the result on FrontierHarness with Kimi K3 underneath. It passed 21 of 30 tasks, first among 13 harnesses, at $1.68 per successful task. The same benchmark put Codex at $3.47 and Claude Code at $18.34. LobeHub itself says the comparison includes selected retries and different execution environments and should not be read as a leaderboard. Fine. Even if you cut the gap in half twice, it is still a 2.7x difference in the cost of a solved task with the model held constant. Nobody shipped a model this year that was 2.7x cheaper per solved task than the one before it.
Alibaba open-sourced the code reviewer it ran internally for two years, and the benchmark table is the clearest single-variable test of the week. Same Claude model on both sides, 200 real pull requests. Through Open Code Review: precision 37.8%, about 352,000 tokens per review, one minute six seconds. Through Claude Code asked to review the PR: precision 15.9%, about 2.06 million tokens, five minutes thirty-eight. The difference is that plain code picks which files get reviewed and which rules apply, bundles related files, and sends them to parallel sub-agents, while the model only reads and judges. Recall did not improve, 11.7% against 12.7%. The harness bought fewer false alarms at a sixth of the tokens, not more bugs found. That trade is a design decision, and it was made in the pipeline, not in the weights.
Prime Intellect's story is the extreme case. Prime Agent ported itself from TypeScript to Rust: 2,209 agents, 228.7 billion tokens, two weeks, 14x faster cold start, memory under a fifth. The part worth copying is not the scale. It is that the humans mostly wrote verification, not code: screen-by-screen comparison of the two versions on the same script, matching session logs and model requests from the same input, protocol message types checked against the old build, and a feature audit sorted into match, partial and missing. Every task went planner, implementer, reviewer, verifier, with the reviewer on a different model and a fresh context because, in their words, the agent that wrote the code grades its own code softly. GLM-5.3 did the inference. The orchestrator ran on two 8-core CPU boxes. The model was a commodity in this story. The checks were the product.
NVIDIA's SoL-Pi paper ran the same experiment on the harness itself: a research agent watched coding agents work, proposed changes, and kept only the ones that lowered cost without hurting scores. Four mechanisms survived, action fusion, online context compaction, observation packing, and an evidence-preserving reducer for delegated reading. Result: 54.3% cheaper than Claude Code on Opus 5, 50% cheaper than Codex on GPT-5.6 Sol, scores in the same range. Same model, half the bill, and the four things that worked are all about what the model is shown and when, not what the model knows.
Then there is the smallest example, which may be the most honest. A Ubuntu operator had a maintenance program that passed 38 simulated tests and then failed on the real box, returning ACTIVE_SKIP every time. OpenClaw could do the same operation on the same machine. The cause was not the model. DeepSeek Harness ran commands inside a bwrap sandbox with its own PID namespace and no path to the user systemd bus, so the agent literally could not see host processes or service state. Two harnesses on one Linux box were looking at two different operating systems. The fix was not more permissions; it was letting the agent write the program and letting the host's own systemd run it.
Put those five next to each other and the pattern is hard to argue with. For two years the public conversation about agents has had one knob, which model, and one metric, the benchmark score. This week the other knob got turned on purpose, with the model held still, and in every case the harness moved cost by a factor of two or more and moved quality by whatever the designer chose to trade for. Think of it as engine versus car. The engine suppliers spent two years telling you horsepower. Then five teams put the same engine in different chassis and one took the lap record while another halved the fuel bill, and it turns out the chassis was never a rounding error.
Why is the harness suddenly measurable when it was folklore a year ago? Because harnesses fail deterministically, and deterministic failures have diffs. Flowness, a one-person multi-agent system with 140 billion Claude tokens on record, published a postmortem in which one task was dispatched to nine background Claude Code sessions in a row. The automatic redispatch cap was 3. The retry counter reached 8. The cap never fired. One acceptance threshold of 1,500 milliseconds came from a cost model in a report and had never been measured; the real number was 244. Every check in the loop asked "can we dispatch now?" and none asked "is this different from last time?" That is not a model problem. You cannot prompt your way out of it and you do not need to. You fix the code and the bug is gone. Model failures are statistical and you manage them. Harness failures are bugs and you close them. That asymmetry is why harness improvements compound and prompt tricks expire.
The cost side has the same shape. One Claude Code user worked out that a late model switch at 300K tokens of context costs about $1.50 just to rebuild cache, while a clean 20K handoff costs about $0.10, and that the Opus-to-Sonnet price ratio shrinks from 1.88x at 20K context to 1.26x at 400K, so the question "which model is cheaper" has no answer until you know how long your loop runs. Another measured that a single tool definition, Claude Code's Artifacts, weighs about 17,800 tokens, nearly 2% of a 1M window and almost a fifth of the whole bootstrap, loaded on every fresh session. A third found that Claude Code's subagents default to a five-minute cache while the main agent gets an hour, and the day split in two over whether to change it: three posts said set it to an hour, one said that burns weekly limits up to 15% faster because subagents rarely idle, and the careful answer was "depends on whether the subagent waits on builds." None of that is about intelligence. It is about who gets to see what, when, and how long the cache stays warm. That is the harness, and it now has a price list.
There is a catch, and this week supplied it too. Pine Computer, a cloud computer built for agents, posted the week's most-circulated number: GPT-5.6 Luna on Pine scored 78.3% on SaaS-Bench checkpoints against 74.3% for Opus 5 with Claude Code, at $1.02 per task versus $26.50. A smaller model on a better environment beat a bigger model on a worse one, at a 26x cost difference. Read the next line of the same post: end-to-end task completion was 27.4% on Pine against 31.1% for Claude Code. The harness that wins the checkpoint metric loses the finish-the-job metric. Which number you optimize is a choice the harness designer makes for you, and most benchmark posts only show the one that flatters.
The second catch comes from the self-improvement crowd itself. A builder reviewing MIT and Sakana's SIFT, which lets an agent propose harness changes and uses an LLM judge to rank them before paying for full benchmark runs, named the obvious risk: a change that improves SWE-bench can make the agent worse at real development, because the harness overfits to the benchmark faster than a model does. LobeHub's own caveats say the same thing more quietly. The 17-researcher paper that took an agent apart into six harness components, observation, context, control loop, action, state, verification, is useful precisely because it gives you six places to look for the overfit. The number to demand from any harness claim is not pass rate. It is cost per verified task across projects the harness has never seen. Nobody is publishing that yet. The first team that does will have the only honest leaderboard in the category.
And the counter-case deserves a seat. Someone who has used Claude Code for a long time asked why anyone needs subagents at all: a single agent loop works in almost every case, keeps the harness simple and limits token use, and nobody has actually measured several single-session Claudes against one session with subagents. That is a fair challenge to the whole week. More harness is not better harness. Alibaba's reviewer won by removing decisions from the model, not adding agents around it. The Rust port won on verification, not on 2,209 workers. The Ubuntu fix was to let the agent do less. Harness engineering, done well, mostly looks like subtraction.
So what moved this week? The prompt stopped being the moat. Anthropic could delete 80% of its own instruction list because the smarter the model gets, the less you need to nag it, and the State of AI report says the part that lasts is the plumbing: the tools the agent can reach, the sandbox it runs in, the monitoring that shows what it did. If your product is a clever pile of prompts around someone else's model, the next model release deletes your edge. If it is the pipeline that decides what the model sees, checks what it claims, and pays for only the turns that mattered, every model release makes it more valuable, because the same chassis gets a better engine for free. Five groups measured that this week. The number was never a rounding error. It just took until now for anyone to hold the engine still long enough to read it.
Comments