Four agent papers: borrow a better harness, stop reading tool output as text
No new arXiv batch lands on weekends, so this is the best of Friday's pile, four papers with one shared theme: the stuff around the model.
Harness-S (arXiv 2611.04277, 63 upvotes on Hugging Face, the top agent paper of the batch) takes the "harness matters" idea into training. Versions of a harness evolved from a common base fix different failures, and which one helps changes as the policy learns. So instead of hunting for one best harness, the authors keep a fixed pool of sibling harnesses and call on it only when every rollout under the base harness fails. The same policy retries under the siblings, and those trajectories replace the failed group in RL. The trained policy then goes back to the original harness and, per the authors, substantially beats policy-only training on AppWorld, SWE-Gym and DBBench. The abstract gives no headline number, so read the paper before quoting a gain.
Coca-Cola (2611.06500, from THU-Agent) questions why agents read tool output as text at all. It compresses observations into latent tokens fed straight into the agent's embedding space, trained so the next action matches what the agent would do with the full text. Authors report 70% to 89% fewer tokens and 24% to 56% less runtime on four web-search QA benchmarks across two model families, with better accuracy. Code is on GitHub. Only QA tasks so far, so whether it holds for code or browsing is open.
VisSearch-Bench (2611.04894) is the reality check: 204 human-written tasks where an agent has to tie image details, objects and spatial relations to web search. Across 16 multimodal models, the best fully solves 26.47%, even with search tools.
TB-MathModel (2611.08237) turns real math-modeling contest problems into terminal tasks with a hard feasibility gate, including 624 Lean 4 checks, that zeroes physically impossible answers. Best score 64.58%; an open model costing $2.54 gets 59.81% at about 1/100 the cost. Gating answers on feasibility instead of an LLM judge is the part worth copying.
All numbers are from the papers' own abstracts, not replicated.
Harness-S: https://arxiv.org/abs/2611.04277
Coca-Cola: https://arxiv.org/abs/2611.06500
VisSearch-Bench: https://arxiv.org/abs/2611.04894
TB-MathModel: https://arxiv.org/abs/2611.08237
← Back to all articles
Harness-S (arXiv 2611.04277, 63 upvotes on Hugging Face, the top agent paper of the batch) takes the "harness matters" idea into training. Versions of a harness evolved from a common base fix different failures, and which one helps changes as the policy learns. So instead of hunting for one best harness, the authors keep a fixed pool of sibling harnesses and call on it only when every rollout under the base harness fails. The same policy retries under the siblings, and those trajectories replace the failed group in RL. The trained policy then goes back to the original harness and, per the authors, substantially beats policy-only training on AppWorld, SWE-Gym and DBBench. The abstract gives no headline number, so read the paper before quoting a gain.
Coca-Cola (2611.06500, from THU-Agent) questions why agents read tool output as text at all. It compresses observations into latent tokens fed straight into the agent's embedding space, trained so the next action matches what the agent would do with the full text. Authors report 70% to 89% fewer tokens and 24% to 56% less runtime on four web-search QA benchmarks across two model families, with better accuracy. Code is on GitHub. Only QA tasks so far, so whether it holds for code or browsing is open.
VisSearch-Bench (2611.04894) is the reality check: 204 human-written tasks where an agent has to tie image details, objects and spatial relations to web search. Across 16 multimodal models, the best fully solves 26.47%, even with search tools.
TB-MathModel (2611.08237) turns real math-modeling contest problems into terminal tasks with a hard feasibility gate, including 624 Lean 4 checks, that zeroes physically impossible answers. Best score 64.58%; an open model costing $2.54 gets 59.81% at about 1/100 the cost. Gating answers on feasibility instead of an LLM judge is the part worth copying.
All numbers are from the papers' own abstracts, not replicated.
Harness-S: https://arxiv.org/abs/2611.04277
Coca-Cola: https://arxiv.org/abs/2611.06500
VisSearch-Bench: https://arxiv.org/abs/2611.04894
TB-MathModel: https://arxiv.org/abs/2611.08237
Comments