Self-improving agents pay a tax, and benchmarks can't see it
A 2,048-token note beat three expensive self-improvement systems. That is the uncomfortable headline of The RSI Tax (https://arxiv.org/abs/2611.13524, Zhang, Li and Wang, Nov 10). The paper scores agents on continual task streams and, unlike most self-improvement papers, adds the tokens spent improving to the tokens spent working. Across six methods and two models, a bounded-note baseline called Simple Reflection with Notes matched or beat Meta-Harness, EvoSkill and RRSI on accuracy, with 65.6 to 71.0% lower total token use.
Where does the money go? Verification eats 74.8 to 93.6% of the self-improvement bill for those three methods. And a large share of the tax pays for states and artifacts the agent never uses again later in the stream. Put plainly: the agents are spending most of their improvement budget checking work, and a lot of the rest building notes nobody reads. If your self-improvement loop has no cost line, it is not an engineering result yet.
The companion paper makes the same point about the scoreboard. Agent Benchmarks Stop Ranking Before They Run Out of Tasks (https://arxiv.org/abs/2611.13541, Dong, Zhan and Hu, Nov 10) re-scores released runs from eight benchmarks, including MLE-bench, ALE-Bench, SWE-fficiency, AlgoTune, SWE-bench Verified and Terminal-Bench. Its finding: much of what looks like saturation is in the measurement, not the tasks. Neighbouring top-ten agents on SWE-bench Verified differ by less than the error in its own tests; correcting those errors reorders neighbours no more than deleting a random sixth of each agent's passes. On ALE-Bench, the top ten tie when graded against the median human, although nearly half the runs beat that human.
The fix they propose is graded credit: a clipped log-ratio score with dated anchors, and cost reported as its own axis. In a Terminal-Bench re-run, models tied on pass/fail differed more than tenfold on the quantity the tests threw away. Building a harder benchmark only moves the ceiling.
Read together, the two papers say the same thing from two ends. The RSI paper shows the bill is hidden in the improvement loop, the benchmark paper shows it is hidden in the scoring bit. Both point to the same habit: report what it cost, and report how much better, not just whether it passed. Anyone selling a self-improving agent this quarter should be asked for both numbers.
← Back to all articles
Where does the money go? Verification eats 74.8 to 93.6% of the self-improvement bill for those three methods. And a large share of the tax pays for states and artifacts the agent never uses again later in the stream. Put plainly: the agents are spending most of their improvement budget checking work, and a lot of the rest building notes nobody reads. If your self-improvement loop has no cost line, it is not an engineering result yet.
The companion paper makes the same point about the scoreboard. Agent Benchmarks Stop Ranking Before They Run Out of Tasks (https://arxiv.org/abs/2611.13541, Dong, Zhan and Hu, Nov 10) re-scores released runs from eight benchmarks, including MLE-bench, ALE-Bench, SWE-fficiency, AlgoTune, SWE-bench Verified and Terminal-Bench. Its finding: much of what looks like saturation is in the measurement, not the tasks. Neighbouring top-ten agents on SWE-bench Verified differ by less than the error in its own tests; correcting those errors reorders neighbours no more than deleting a random sixth of each agent's passes. On ALE-Bench, the top ten tie when graded against the median human, although nearly half the runs beat that human.
The fix they propose is graded credit: a clipped log-ratio score with dated anchors, and cost reported as its own axis. In a Terminal-Bench re-run, models tied on pass/fail differed more than tenfold on the quantity the tests threw away. Building a harder benchmark only moves the ceiling.
Read together, the two papers say the same thing from two ends. The RSI paper shows the bill is hidden in the improvement loop, the benchmark paper shows it is hidden in the scoring bit. Both point to the same habit: report what it cost, and report how much better, not just whether it passed. Anyone selling a self-improving agent this quarter should be asked for both numbers.
Comments