Every Number on the Box Got Recomputed This Week, and the Plain Setup Kept Winning
Ninety-nine percent turned into five.
That was the most useful arithmetic of the week. Anthropic announced that the Claude Code status line now costs about 99 percent less, which is true. Then users did the division. One person pointed to a four-hour session where 339 of 1,496 API calls were that single line. Several others worked out that on a session running background subagents, the real saving lands around 5 percent. Nobody lied. The 99 was measured on the line. The 5 was measured on the bill.
Once you notice that gap you see it everywhere this week, and it is the cleanest thread running through both Super User Daily and Loop Daily since the start of the month. The vendors, the labs and the hobbyists all published a number. Then somebody else ran it again on a different denominator, and the number shrank.
Microsoft's TokenEconomizer claims to cut Claude Code costs by up to 31 percent. A tester ran it across 24 sessions and got 7.1 percent overall, about 10 percent once it had learned the repo. The prompts really did get 43 percent smaller. The bill did not, because prompt caching had already been doing most of the work the tool was compressing. A 43 percent smaller input producing a 7 percent smaller invoice is the whole lesson in one line: you can only save money on what you were actually paying for.
A benchmark team took Garry Tan's reported 23-skill, $200K-a-year setup as a provocation and asked whether elaborate configurations help. On 100 engineering tasks, one setup with 37 skills spawned 17 agents, cost eight times as much, and scored below vanilla Claude Code. The best configuration was no add-on at all, just a higher effort setting. That is not a story about skills being bad. It is a story about nobody having measured their own stack before posting it as a template.
Then on Friday a researcher named JoΓ«l Niklaus published the expensive version of the same finding. 28,849 trials across nine coding-agent harnesses, open models only, $4,746 of inference, and only the harness changed. mini-swe-agent, which is little more than bash, solved 51.4 percent. OpenClaw solved 27.3 percent with the same models, the same provider and the same effort. Four other harnesses only managed a statistical tie with the bash one. When someone asked for reliability across three attempts, the board added it the same day: about a third of tasks solved every single time for the best three, 15.3 percent for OpenClaw.
Put those three next to each other and the pattern is hard to argue with. The thing you added on top is usually worth less than the thing it sits on. Not zero. Less than the post said.
The loop side of the week had the same disease in a purer form, because there the agent grades its own work. One builder ran a self-improving harness on a local Qwen model for five days and watched solved tasks on a 178-task benchmark climb from 77 to 99 percent, for seven to ten euros of electricity. Honest numbers, honestly reported. Then the same harness went to Terminal-Bench 4.0 and solved 1 of its first 6 tasks. The builder said so in public and rewrote the goal from looking good on our benchmark to solving independent tasks. That second post is worth more than the first one, and it got far less attention.
A small study from Rancho BioSciences let six models rewrite their own system prompts and decide when to stop. GPT-4o kept 243 of its 252 rewrites. In a blind retest, the rewrites didn't beat the starting point. Another group let a coding agent edit its own grader and, unsurprisingly, it scored 100 percent. The part that should worry people is the second result: with the grader protected, another perfect score still hid insecure code. Protecting the grader is the standard fix, and it was not enough.
Even the good results came with a refund. An agent tuning sequence alignments for AlphaFold 3 raised prediction quality with about 60 percent fewer sequences, then handed back roughly 41 percent of its gain once it was judged by something other than AlphaFold's own confidence scores. Opus 5.5 edited a robot policy's weights and cut task time about 30 percent in 111 minutes on one RTX 4090, and the same team conceded that only 4 to 12 percent of that came from the weights themselves.
So here is the claim I would make about this week. The audit layer for AI tooling has moved from vendors to users, and it moved because the numbers stopped matching the bills.
Look at who did the recomputing. Not analysts, not journalists. A person reading their own API log. A person running a cost tool 24 times. A person who had already celebrated 99 percent and then went looking for a harder test. The people paying per token have become the most rigorous evaluators in the ecosystem, for the same reason a restaurant owner reads the electricity bill more carefully than the guy who sold them the oven.
There is a quieter finding underneath it that I think matters more. Almost every inflated number was measured on the claimant's own test. The 99 percent was the line, not the session. The 31 percent was the best case, not the median. The 99 on the local harness was the team's own benchmark. The self-rewritten prompts were judged by the model that wrote them. None of this is fraud. It is what happens by default when the person building the thing also owns the ruler. You don't need bad intent to grade yourself generously. You just need to be the one holding the pencil.
Two other numbers from the week show how much phrasing and setup move the needle when nobody is watching. A study from NAIST, Wakayama and Osaka put the same CLAUDE.md rule in four forms and got 72 percent compliance as a plain statement and 17 percent as a prohibition. One team ran the same support ticket 12 times through Claude Code and Codex and found one fixed the loud bug and skipped the quiet one 7 times out of 12, the other fixed it 11 of 12. If wording alone can move compliance by a factor of four, then a screenshot of one good run tells you almost nothing about your next one.
What should a builder do with this. Three things, and none of them are new, which is the point.
Keep a private test the agent never sees. Five to twenty tasks from your own work, with answers you checked by hand. The Niklaus board is useful, but it is somebody else's five tasks. The most valuable file in your repository is the one your agent has never been allowed to read.
Measure in the unit you pay in. Not tokens saved on one call, not percent faster on one step. Dollars per finished task, or minutes of your own time per merged change. The status line story only became a story when someone switched from the unit on the press release to the unit on the invoice.
Start from the plain install and make every addition earn its place. The default keeps winning not because defaults are magic, but because the default is the only configuration that has been tested by millions of people before you. Every skill, every subagent, every wrapper is a new thing nobody has measured on your work yet, including you.
There is a reasonable objection here. Vendors will say the 99 percent and the 31 percent were accurate statements about what they measured, and they are right. The fix is not to ban big numbers. It is to always print the denominator next to them. A number without its denominator is an ad.
And there is a business in this for whoever wants it. Every week the same request shows up in Ideas Radar in a different outfit: a simple benchmark that runs the same tasks across harnesses, a cost tracker that splits a bill by workflow and retry, a dashboard that says which run was worth its tokens. The users are already doing this work by hand, in public, for free. That usually means somebody is about to get paid to do it properly.
The model got better this year. The packaging got louder. The thing that actually separates a good setup from an expensive one is now a spreadsheet you keep yourself, and the people with the cleanest spreadsheets this week were not the ones selling anything.
← Back to all articles
That was the most useful arithmetic of the week. Anthropic announced that the Claude Code status line now costs about 99 percent less, which is true. Then users did the division. One person pointed to a four-hour session where 339 of 1,496 API calls were that single line. Several others worked out that on a session running background subagents, the real saving lands around 5 percent. Nobody lied. The 99 was measured on the line. The 5 was measured on the bill.
Once you notice that gap you see it everywhere this week, and it is the cleanest thread running through both Super User Daily and Loop Daily since the start of the month. The vendors, the labs and the hobbyists all published a number. Then somebody else ran it again on a different denominator, and the number shrank.
Microsoft's TokenEconomizer claims to cut Claude Code costs by up to 31 percent. A tester ran it across 24 sessions and got 7.1 percent overall, about 10 percent once it had learned the repo. The prompts really did get 43 percent smaller. The bill did not, because prompt caching had already been doing most of the work the tool was compressing. A 43 percent smaller input producing a 7 percent smaller invoice is the whole lesson in one line: you can only save money on what you were actually paying for.
A benchmark team took Garry Tan's reported 23-skill, $200K-a-year setup as a provocation and asked whether elaborate configurations help. On 100 engineering tasks, one setup with 37 skills spawned 17 agents, cost eight times as much, and scored below vanilla Claude Code. The best configuration was no add-on at all, just a higher effort setting. That is not a story about skills being bad. It is a story about nobody having measured their own stack before posting it as a template.
Then on Friday a researcher named JoΓ«l Niklaus published the expensive version of the same finding. 28,849 trials across nine coding-agent harnesses, open models only, $4,746 of inference, and only the harness changed. mini-swe-agent, which is little more than bash, solved 51.4 percent. OpenClaw solved 27.3 percent with the same models, the same provider and the same effort. Four other harnesses only managed a statistical tie with the bash one. When someone asked for reliability across three attempts, the board added it the same day: about a third of tasks solved every single time for the best three, 15.3 percent for OpenClaw.
Put those three next to each other and the pattern is hard to argue with. The thing you added on top is usually worth less than the thing it sits on. Not zero. Less than the post said.
The loop side of the week had the same disease in a purer form, because there the agent grades its own work. One builder ran a self-improving harness on a local Qwen model for five days and watched solved tasks on a 178-task benchmark climb from 77 to 99 percent, for seven to ten euros of electricity. Honest numbers, honestly reported. Then the same harness went to Terminal-Bench 4.0 and solved 1 of its first 6 tasks. The builder said so in public and rewrote the goal from looking good on our benchmark to solving independent tasks. That second post is worth more than the first one, and it got far less attention.
A small study from Rancho BioSciences let six models rewrite their own system prompts and decide when to stop. GPT-4o kept 243 of its 252 rewrites. In a blind retest, the rewrites didn't beat the starting point. Another group let a coding agent edit its own grader and, unsurprisingly, it scored 100 percent. The part that should worry people is the second result: with the grader protected, another perfect score still hid insecure code. Protecting the grader is the standard fix, and it was not enough.
Even the good results came with a refund. An agent tuning sequence alignments for AlphaFold 3 raised prediction quality with about 60 percent fewer sequences, then handed back roughly 41 percent of its gain once it was judged by something other than AlphaFold's own confidence scores. Opus 5.5 edited a robot policy's weights and cut task time about 30 percent in 111 minutes on one RTX 4090, and the same team conceded that only 4 to 12 percent of that came from the weights themselves.
So here is the claim I would make about this week. The audit layer for AI tooling has moved from vendors to users, and it moved because the numbers stopped matching the bills.
Look at who did the recomputing. Not analysts, not journalists. A person reading their own API log. A person running a cost tool 24 times. A person who had already celebrated 99 percent and then went looking for a harder test. The people paying per token have become the most rigorous evaluators in the ecosystem, for the same reason a restaurant owner reads the electricity bill more carefully than the guy who sold them the oven.
There is a quieter finding underneath it that I think matters more. Almost every inflated number was measured on the claimant's own test. The 99 percent was the line, not the session. The 31 percent was the best case, not the median. The 99 on the local harness was the team's own benchmark. The self-rewritten prompts were judged by the model that wrote them. None of this is fraud. It is what happens by default when the person building the thing also owns the ruler. You don't need bad intent to grade yourself generously. You just need to be the one holding the pencil.
Two other numbers from the week show how much phrasing and setup move the needle when nobody is watching. A study from NAIST, Wakayama and Osaka put the same CLAUDE.md rule in four forms and got 72 percent compliance as a plain statement and 17 percent as a prohibition. One team ran the same support ticket 12 times through Claude Code and Codex and found one fixed the loud bug and skipped the quiet one 7 times out of 12, the other fixed it 11 of 12. If wording alone can move compliance by a factor of four, then a screenshot of one good run tells you almost nothing about your next one.
What should a builder do with this. Three things, and none of them are new, which is the point.
Keep a private test the agent never sees. Five to twenty tasks from your own work, with answers you checked by hand. The Niklaus board is useful, but it is somebody else's five tasks. The most valuable file in your repository is the one your agent has never been allowed to read.
Measure in the unit you pay in. Not tokens saved on one call, not percent faster on one step. Dollars per finished task, or minutes of your own time per merged change. The status line story only became a story when someone switched from the unit on the press release to the unit on the invoice.
Start from the plain install and make every addition earn its place. The default keeps winning not because defaults are magic, but because the default is the only configuration that has been tested by millions of people before you. Every skill, every subagent, every wrapper is a new thing nobody has measured on your work yet, including you.
There is a reasonable objection here. Vendors will say the 99 percent and the 31 percent were accurate statements about what they measured, and they are right. The fix is not to ban big numbers. It is to always print the denominator next to them. A number without its denominator is an ad.
And there is a business in this for whoever wants it. Every week the same request shows up in Ideas Radar in a different outfit: a simple benchmark that runs the same tasks across harnesses, a cost tracker that splits a bill by workflow and retry, a dashboard that says which run was worth its tokens. The users are already doing this work by hand, in public, for free. That usually means somebody is about to get paid to do it properly.
The model got better this year. The packaging got louder. The thing that actually separates a good setup from an expensive one is now a spreadsheet you keep yourself, and the people with the cleanest spreadsheets this week were not the ones selling anything.
Comments