176 harness configs later: give a strong model bash and get out of its way
A 43-page study swept 176 harness configurations across four models on SWE-Bench Verified and Terminal-Bench 2.1, isolating three knobs that everybody tunes by vibes: context management, planning, and action space. arXiv 2609.20804, submitted September 17, Run-Ze Fan and seven co-authors with Zoom Communications on the byline. It hit 194 points on Hacker News in a day, which tells you how starved this question is for actual measurement.
The action space finding is the one to act on. Predefined tools help models with weak bash skills. Capable models do just as well with bash only, at substantially lower cost. So the elaborate tool surface most harnesses ship is a crutch for the model you are not using, and you are paying for it in tokens on every turn. Planning splits the same way: it scaffolds weaker models into working, and for stronger models it reduces cost without changing accuracy at all. Two knobs, same shape, and in both cases the right setting depends on which model you pointed at the problem rather than on which is better in the abstract.
Context management is the one that behaves differently. It matters more as available context shrinks, and what it is mostly buying you is not smarter behavior, it is the absence of overflow failures. The authors find that staging rule-based elision before LLM-based summarization gives the strongest overall efficiency, which is to say: throw away what you can throw away deterministically first, and only pay a model to compress what survives.
The trajectory analysis is the part I would quote to anyone designing a harness from scratch. Context management extends how long the agent runs, planning determines where it stops, and action space controls how granular its edits are. Three separate levers on three separate properties, which is the closest thing to a mental model anybody has published for this. Paper at https://arxiv.org/abs/2609.20804, no code repo linked, which is the one real gap in it.
Related: NVIDIA's SoL-Pi https://clauday.com/article/132a6ee0-9d6b-401c-81b0-3d92deca839b
← Back to all articles
The action space finding is the one to act on. Predefined tools help models with weak bash skills. Capable models do just as well with bash only, at substantially lower cost. So the elaborate tool surface most harnesses ship is a crutch for the model you are not using, and you are paying for it in tokens on every turn. Planning splits the same way: it scaffolds weaker models into working, and for stronger models it reduces cost without changing accuracy at all. Two knobs, same shape, and in both cases the right setting depends on which model you pointed at the problem rather than on which is better in the abstract.
Context management is the one that behaves differently. It matters more as available context shrinks, and what it is mostly buying you is not smarter behavior, it is the absence of overflow failures. The authors find that staging rule-based elision before LLM-based summarization gives the strongest overall efficiency, which is to say: throw away what you can throw away deterministically first, and only pay a model to compress what survives.
The trajectory analysis is the part I would quote to anyone designing a harness from scratch. Context management extends how long the agent runs, planning determines where it stops, and action space controls how granular its edits are. Three separate levers on three separate properties, which is the closest thing to a mental model anybody has published for this. Paper at https://arxiv.org/abs/2609.20804, no code repo linked, which is the one real gap in it.
Related: NVIDIA's SoL-Pi https://clauday.com/article/132a6ee0-9d6b-401c-81b0-3d92deca839b
Comments