Simple harnesses, independent critics, low effort first: this week's harness papers agree
If you build agent harnesses, three papers from Tuesday's arXiv batch are worth an afternoon, because they push back on the instinct to add more structure.
The first tests root-cause analysis for production incidents on a 90-case set called RCAtlas. Simple general-purpose harnesses, one of them just the Codex coding agent, hit 65-77% top-1 accuracy on three mid-tier models. Published RCA agents with hand-written procedures scored 16-53%, and non-LLM methods 6-39%. Turning up reasoning effort moved accuracy by a few points while making runs up to seven times slower. The scripted procedure was getting in the model's way.
The second, Rethinking Verification in Agent Harnesses, compares 17 verification designs across seven coding models on Terminal-Bench and SWE-bench Verified. An independent critic reviewing the work added 5.9-9.5 points on Terminal-Bench at 1.13-1.25 times baseline cost, and 3.9-6.6 points on SWE-bench at almost no extra cost. Completion gates can win bigger but can blow up the bill. The third paper calls reasoning effort "a credence good": you pay for it without being able to see if you needed it. On easy tasks the top effort level costs up to 42x the lowest, and much of the extra goes into the agent re-checking itself. The authors' policy is to start at the lowest effort and escalate only after a verified failure.
Two smaller results fill in the picture. A matched MCP versus CLI study found the same tools produce structurally different behavior in 94.3% of 2,004 episodes, so the interface is part of the harness. AOS-Bench found that when agents manage their own task queues on shared machines, serial dispatch leaves 73% of heavy-workload tasks unclaimed while median CPU stays under 30%, and a fixed concurrency of two comes within four points of the best agent-managed policy.
Our read: the money in a harness is not in the script, it is in who checks the work and when you pay for more thinking. Give the model room, put a separate reviewer on the output, and spend effort only on proven failures.
RCA harnesses: https://arxiv.org/abs/2610.35767 Rethinking Verification: https://arxiv.org/abs/2610.34577 Effort economics: https://arxiv.org/abs/2610.34491 MCP vs CLI: https://arxiv.org/abs/2610.34298 AOS-Bench: https://arxiv.org/abs/2610.33178
← Back to all articles
The first tests root-cause analysis for production incidents on a 90-case set called RCAtlas. Simple general-purpose harnesses, one of them just the Codex coding agent, hit 65-77% top-1 accuracy on three mid-tier models. Published RCA agents with hand-written procedures scored 16-53%, and non-LLM methods 6-39%. Turning up reasoning effort moved accuracy by a few points while making runs up to seven times slower. The scripted procedure was getting in the model's way.
The second, Rethinking Verification in Agent Harnesses, compares 17 verification designs across seven coding models on Terminal-Bench and SWE-bench Verified. An independent critic reviewing the work added 5.9-9.5 points on Terminal-Bench at 1.13-1.25 times baseline cost, and 3.9-6.6 points on SWE-bench at almost no extra cost. Completion gates can win bigger but can blow up the bill. The third paper calls reasoning effort "a credence good": you pay for it without being able to see if you needed it. On easy tasks the top effort level costs up to 42x the lowest, and much of the extra goes into the agent re-checking itself. The authors' policy is to start at the lowest effort and escalate only after a verified failure.
Two smaller results fill in the picture. A matched MCP versus CLI study found the same tools produce structurally different behavior in 94.3% of 2,004 episodes, so the interface is part of the harness. AOS-Bench found that when agents manage their own task queues on shared machines, serial dispatch leaves 73% of heavy-workload tasks unclaimed while median CPU stays under 30%, and a fixed concurrency of two comes within four points of the best agent-managed policy.
Our read: the money in a harness is not in the script, it is in who checks the work and when you pay for more thinking. Give the model room, put a separate reviewer on the output, and spend effort only on proven failures.
RCA harnesses: https://arxiv.org/abs/2610.35767 Rethinking Verification: https://arxiv.org/abs/2610.34577 Effort economics: https://arxiv.org/abs/2610.34491 MCP vs CLI: https://arxiv.org/abs/2610.34298 AOS-Bench: https://arxiv.org/abs/2610.33178
Comments