Somebody Spent $1,500 Proving the 90% Token Savings Aren't Real
RTK has been going around as the free lunch of coding agents: pipe your terminal output through it, get 60 to 90 percent token savings. Quesma ran 1,740 benchmark attempts to check, and the answer is no. Fable 5.0 costs went down 1 to 5 percent depending on how you measure. DeepSeek V4 Pro costs went up 5 to 17 percent. The full writeup is at https://quesma.com/blog/does-rtk-make-ai-coding-cheaper/ and it hit Hacker News on September 11.
The methodology is the point. 85 tasks on Claude Code with Fable 5.0 and 89 on OpenCode with DeepSeek V4 Pro, run on Terminal-Bench 2.1, tracking pass rate, tokens and dollars rather than just tokens. That last distinction is where the whole illusion lives. RTK's own metric, rtk gain, counts bytes removed from command output divided by four. It does not count the turns those removals cause. In the DeepSeek runs RTK reported 349 million tokens saved while the tasks got more expensive.
Why compression backfires is worth internalizing because it generalizes past this one tool. Terminal output is only about 11 percent of Fable's input tokens and 40 percent of DeepSeek's, so the ceiling is low to start. Frontier models already write careful commands, they pipe through head, they do not cat the whole file. And the moment you truncate something the model wanted, it issues another tool call, which means a full uncached prefill of the entire conversation. One extra turn costs more than a lot of saved bytes. JetBrains found the same thing independently in July, measuring rtk at 7.6 percent more expensive at low effort.
The honest conclusion Quesma lands on is that RTK is a niche optimization for older models, not a general cost cut. The broader lesson is that every context-compression tool should be required to publish a dollar benchmark, not a bytes-removed number, and almost none of them do. If you are running one of these in production, measure your own bill before and after. The vendor's savings metric is measuring something that is not your invoice.
Related reading: https://clauday.com/article/6a998ccf-b793-4da4-9f00-97b27af81532
← Back to all articles
The methodology is the point. 85 tasks on Claude Code with Fable 5.0 and 89 on OpenCode with DeepSeek V4 Pro, run on Terminal-Bench 2.1, tracking pass rate, tokens and dollars rather than just tokens. That last distinction is where the whole illusion lives. RTK's own metric, rtk gain, counts bytes removed from command output divided by four. It does not count the turns those removals cause. In the DeepSeek runs RTK reported 349 million tokens saved while the tasks got more expensive.
Why compression backfires is worth internalizing because it generalizes past this one tool. Terminal output is only about 11 percent of Fable's input tokens and 40 percent of DeepSeek's, so the ceiling is low to start. Frontier models already write careful commands, they pipe through head, they do not cat the whole file. And the moment you truncate something the model wanted, it issues another tool call, which means a full uncached prefill of the entire conversation. One extra turn costs more than a lot of saved bytes. JetBrains found the same thing independently in July, measuring rtk at 7.6 percent more expensive at low effort.
The honest conclusion Quesma lands on is that RTK is a niche optimization for older models, not a general cost cut. The broader lesson is that every context-compression tool should be required to publish a dollar benchmark, not a bytes-removed number, and almost none of them do. If you are running one of these in production, measure your own bill before and after. The vendor's savings metric is measuring something that is not your invoice.
Related reading: https://clauday.com/article/6a998ccf-b793-4da4-9f00-97b27af81532
Comments