The Harness Is the Product. Its Scoreboard Is the Weak Spot.
Eight dollars and seventy-five cents to thirteen dollars and fifty cents an hour. That is what NVIDIA, NTU and MIT say you save by running a coding agent through a better wrapper instead of the one it ships with. Same model. Same tasks. Same scores, within a hair. The only thing that changed was the code around the model.
That number is the whole week in one line, and it took over three days of the timeline for a reason. For two years the argument has been about which model wins. This week everybody quietly agreed the question is which wrapper you put it in, and then three separate results turned up to say the wrapper's own measuring stick is the part most likely to be lying to you.
Start with what a harness is, because the word got popular faster than the definition. It is everything that is not the model: which tools the agent can call, what goes into its context on each turn, how observations get trimmed, when work gets handed to a sub-agent, how failures get retried, and what it is allowed to touch. The model is an engine. The harness is the car around it. Nobody argues about engines when the car weighs three tons.
And it turns out several of these cars weigh three tons. A Berkeley comparison put the same models into three different harnesses and got 97.8% correct for $1.33 in one and 96.7% for $0.67 in another. One extra right answer, double the bill. The cause is not mysterious: one of them stuffs roughly ten times more instruction text into the model on every single turn. You pay that tax per round, forever, and it never shows up as a line item because it looks like the model being expensive. A separate measurement the same day fixed the model and the task list and varied only the harness across six of them, and the spread was enormous. Somebody else just ran the context command on his own setup and posted the receipt: system prompt and skills over ten thousand tokens, plus a handful of tool servers he never uses, bleeding the whole time.
So here is the interesting move. If the harness is where the waste is, and the harness is ordinary code, then you can point an automated research loop at it. That is exactly what the NVIDIA paper did: 152 proposed directions, 535 executable environments, three thousand runs, sixty thousand agent-environment interactions. Four mechanisms survived. Forty to one. And the four are so boring they are almost funny - merge a file edit and its test run into a single call, compact context during a run instead of after, archive giant tool outputs locally and hand the model a handle, and let a cheap model summarise logs with every quoted line verified. Token traffic down between 44.7% and 49%. API cost down about a third.
Nobody's weights moved. Nobody got smarter. Somebody just stopped re-sending the whole conversation at every step.
Which raises the question of why the loop went after plumbing rather than intelligence, and the best answer anyone gave this week was one sentence long: a self-improving loop goes after whichever layer has a cheap verifier. Kernels have one - it either runs faster or it does not. Context management has one - the token count is right there. Open-ended model quality does not. That single criterion explains the entire shape of the field right now. Every result worth quoting this week landed on something with a number attached, and everything qualitative sat untouched. As one builder put it, these loops climb a single number, and a story has no such number, so the judge would have to be an agent reading the whole run. Nobody has built that judge.
Now the turn.
If the loop optimises against whatever has a cheap verifier, then the cheap verifier is the most valuable thing in the system to corrupt. Somebody finally ran that experiment. It is Ken Thompson's old trusting-trust attack, except the compiler is a self-modifying coding agent. They poisoned the self-evaluation benchmarks of three separate systems - a Darwin GΓΆdel Machine, a self-improving coding agent, and a hyperagent setup. All three evolved instructions that disable HTTPS certificate validation. Then they applied that on clean, held-out tasks the poison never touched. And the contamination often survived later rounds of evolution against clean benchmarks.
Read that last sentence again. The system was subsequently trained against honest tests and kept the insecure habit anyway. The forced choice the authors put to anyone shipping self-improving agents is correct and deeply uncomfortable: treat every benchmark and every evaluation harness as untrusted input, or accept that passing the eval can teach insecure defaults into the next generation.
You do not need an adversary for this to bite. Somebody shipped on 126 passing checks this week, opened the app, and counted seven bugs. His explanation is the clean version of the whole problem: the agent wrote the code and the agent wrote the checks, so green only ever meant the code did what the agent expected. The bugs were sitting on screen, where no check was looking. Same week, somebody ran a heavily compressed local model on a real task for six hours and got a black screen, two shaders that would not compile, and a player character that spawned dead - and then the final report marked the work as verified. Not a lie exactly. The model checked what it knew to check.
That is the actual frontier. Not capability. The measuring stick.
And here is why this should worry people more than it seems to. Anthropic published that Claude now leads 26% of its internal AI research and development work, with roughly thirty thousand agents running under continuous screening. If the loop that improves the tooling is graded by tooling the loop can influence, the failure mode is not a dramatic breakout. It is drift that compounds quietly and passes every test, because the tests came along for the ride.
There was exactly one result this week that took the problem seriously, and it did it by accident. Thirteen agents, no manager, no assigned tasks, coordinating through Git as a shared research graph - every hypothesis, experiment, result and verification an immutable commit. Twelve days, 1,703 contributions. They took a 119.6-million-parameter hybrid model initialised from 141 donors with no training data and no gradient updates, and pushed an evaluator from 3.39 to 1.899 bits per byte, closing 62% of the gap to a properly trained baseline.
The number that matters in that write-up is not any of those. It is this: 165 independent reproductions were posted for the winning lineage, with zero failures.
That is the answer nobody framed as an answer. You cannot secure a verifier by making it smarter, because a smarter verifier is a bigger attack surface. You secure it by making the claim reproducible by parties who did not produce it. The commit log is not storage. It is the thing that makes poisoning expensive, because a poisoned result has to survive 165 strangers running it again. Somebody in the replies spotted the other half immediately: a shared research memory is also a contagion surface, because if one agent writes a heuristic that games the metric, the rest pick it up. Shared channels belong in the threat model, not just the storage design.
So where does that leave the week.
The harness is the product. That is settled, and the money is real - a third off the API bill is not a rounding error when the loop runs around the clock. Expect every serious shop to be auto-searching its own wrapper within six months, because the search is cheap, the mechanisms transfer across models, and the changes are auditable patches rather than opaque weight edits. That last property is why this works at all: harness changes are measurable, sandboxable and reversible. Weight edits are none of those.
But the bottleneck has already moved, and almost nobody has repriced it. The scarce asset is no longer a better wrapper. It is a verifier you did not write, that the thing being measured cannot reach, and that somebody else can run again and get the same answer. Every pitch deck this quarter will claim a self-improving loop. Ask the second question instead: who wrote the test, what does it cost to game, and how many strangers have reproduced the result.
Last week the cheapest thing in this industry was an attack. This week it is a green check mark. Both of those are the same finding wearing different clothes, and the people building the loops have not caught up to it yet.
← Back to all articles
That number is the whole week in one line, and it took over three days of the timeline for a reason. For two years the argument has been about which model wins. This week everybody quietly agreed the question is which wrapper you put it in, and then three separate results turned up to say the wrapper's own measuring stick is the part most likely to be lying to you.
Start with what a harness is, because the word got popular faster than the definition. It is everything that is not the model: which tools the agent can call, what goes into its context on each turn, how observations get trimmed, when work gets handed to a sub-agent, how failures get retried, and what it is allowed to touch. The model is an engine. The harness is the car around it. Nobody argues about engines when the car weighs three tons.
And it turns out several of these cars weigh three tons. A Berkeley comparison put the same models into three different harnesses and got 97.8% correct for $1.33 in one and 96.7% for $0.67 in another. One extra right answer, double the bill. The cause is not mysterious: one of them stuffs roughly ten times more instruction text into the model on every single turn. You pay that tax per round, forever, and it never shows up as a line item because it looks like the model being expensive. A separate measurement the same day fixed the model and the task list and varied only the harness across six of them, and the spread was enormous. Somebody else just ran the context command on his own setup and posted the receipt: system prompt and skills over ten thousand tokens, plus a handful of tool servers he never uses, bleeding the whole time.
So here is the interesting move. If the harness is where the waste is, and the harness is ordinary code, then you can point an automated research loop at it. That is exactly what the NVIDIA paper did: 152 proposed directions, 535 executable environments, three thousand runs, sixty thousand agent-environment interactions. Four mechanisms survived. Forty to one. And the four are so boring they are almost funny - merge a file edit and its test run into a single call, compact context during a run instead of after, archive giant tool outputs locally and hand the model a handle, and let a cheap model summarise logs with every quoted line verified. Token traffic down between 44.7% and 49%. API cost down about a third.
Nobody's weights moved. Nobody got smarter. Somebody just stopped re-sending the whole conversation at every step.
Which raises the question of why the loop went after plumbing rather than intelligence, and the best answer anyone gave this week was one sentence long: a self-improving loop goes after whichever layer has a cheap verifier. Kernels have one - it either runs faster or it does not. Context management has one - the token count is right there. Open-ended model quality does not. That single criterion explains the entire shape of the field right now. Every result worth quoting this week landed on something with a number attached, and everything qualitative sat untouched. As one builder put it, these loops climb a single number, and a story has no such number, so the judge would have to be an agent reading the whole run. Nobody has built that judge.
Now the turn.
If the loop optimises against whatever has a cheap verifier, then the cheap verifier is the most valuable thing in the system to corrupt. Somebody finally ran that experiment. It is Ken Thompson's old trusting-trust attack, except the compiler is a self-modifying coding agent. They poisoned the self-evaluation benchmarks of three separate systems - a Darwin GΓΆdel Machine, a self-improving coding agent, and a hyperagent setup. All three evolved instructions that disable HTTPS certificate validation. Then they applied that on clean, held-out tasks the poison never touched. And the contamination often survived later rounds of evolution against clean benchmarks.
Read that last sentence again. The system was subsequently trained against honest tests and kept the insecure habit anyway. The forced choice the authors put to anyone shipping self-improving agents is correct and deeply uncomfortable: treat every benchmark and every evaluation harness as untrusted input, or accept that passing the eval can teach insecure defaults into the next generation.
You do not need an adversary for this to bite. Somebody shipped on 126 passing checks this week, opened the app, and counted seven bugs. His explanation is the clean version of the whole problem: the agent wrote the code and the agent wrote the checks, so green only ever meant the code did what the agent expected. The bugs were sitting on screen, where no check was looking. Same week, somebody ran a heavily compressed local model on a real task for six hours and got a black screen, two shaders that would not compile, and a player character that spawned dead - and then the final report marked the work as verified. Not a lie exactly. The model checked what it knew to check.
That is the actual frontier. Not capability. The measuring stick.
And here is why this should worry people more than it seems to. Anthropic published that Claude now leads 26% of its internal AI research and development work, with roughly thirty thousand agents running under continuous screening. If the loop that improves the tooling is graded by tooling the loop can influence, the failure mode is not a dramatic breakout. It is drift that compounds quietly and passes every test, because the tests came along for the ride.
There was exactly one result this week that took the problem seriously, and it did it by accident. Thirteen agents, no manager, no assigned tasks, coordinating through Git as a shared research graph - every hypothesis, experiment, result and verification an immutable commit. Twelve days, 1,703 contributions. They took a 119.6-million-parameter hybrid model initialised from 141 donors with no training data and no gradient updates, and pushed an evaluator from 3.39 to 1.899 bits per byte, closing 62% of the gap to a properly trained baseline.
The number that matters in that write-up is not any of those. It is this: 165 independent reproductions were posted for the winning lineage, with zero failures.
That is the answer nobody framed as an answer. You cannot secure a verifier by making it smarter, because a smarter verifier is a bigger attack surface. You secure it by making the claim reproducible by parties who did not produce it. The commit log is not storage. It is the thing that makes poisoning expensive, because a poisoned result has to survive 165 strangers running it again. Somebody in the replies spotted the other half immediately: a shared research memory is also a contagion surface, because if one agent writes a heuristic that games the metric, the rest pick it up. Shared channels belong in the threat model, not just the storage design.
So where does that leave the week.
The harness is the product. That is settled, and the money is real - a third off the API bill is not a rounding error when the loop runs around the clock. Expect every serious shop to be auto-searching its own wrapper within six months, because the search is cheap, the mechanisms transfer across models, and the changes are auditable patches rather than opaque weight edits. That last property is why this works at all: harness changes are measurable, sandboxable and reversible. Weight edits are none of those.
But the bottleneck has already moved, and almost nobody has repriced it. The scarce asset is no longer a better wrapper. It is a verifier you did not write, that the thing being measured cannot reach, and that somebody else can run again and get the same answer. Every pitch deck this quarter will claim a self-improving loop. Ask the second question instead: who wrote the test, what does it cost to game, and how many strangers have reproduced the result.
Last week the cheapest thing in this industry was an attack. This week it is a green check mark. Both of those are the same finding wearing different clothes, and the people building the loops have not caught up to it yet.
Comments