Three papers grade the harness, and model scale doesn't save you
For two years agent evals scored the model and treated the harness as plumbing. Three papers in Monday's arXiv batch score the plumbing, and the plumbing turns out to decide a lot.
TraceTrial-Bench breaks a harness into seven parts: context, execution, governance, lifecycle, observability, tools and verification. Across 4 harnesses, 9 models and 95 tasks, 3,420 runs, harness scores range from 59.8% to 72.0%. The number to remember is this one: "31.1% of runs achieve task success despite failing at least one harness criterion." Almost a third of the green checkmarks came from runs that broke a rule along the way, and a success-only benchmark would have counted them all as wins.
"When Agents Fail: Model or Harness, Who's to Blame?" ran OpenCode, Codex CLI, Pi CLI and DeepAgents across 111 settings and 52,168 trajectories. "Harness rankings vary across models and benchmarks", and task performance and valid completion can favor different pairings. OpenCode ranked first across models, while DeepSeek V4 Flash Preview held a consistently high valid completion rate whichever harness it sat in. There is no single best harness, only best pairings.
The third, accepted as a NeurIPS 2026 poster, mined 57,240 issue and PR threads from 1,774 open-source MCP repos to build realistic tool faults. Under those faults gpt-oss scores fell 60%, from 8.08 to 3.23 out of 10, and the 120B model kept no bigger share of its score than the 20B (p = 0.96). The authors' line: "framework-level error handling matters more than model scale."
Read together, the message for builders is direct. When you switch models, re-run your evals in your own harness, because last month's ranking may not survive the move. Measure valid completion, not just success. And if your MCP tools time out, return partial results or change schemas, the fix is retries and error handling in the framework, not a bigger model.
TraceTrial-Bench: https://arxiv.org/abs/2610.32285 Model or Harness: https://arxiv.org/abs/2610.32483 MCP-Injector: https://arxiv.org/abs/2610.32555
← Back to all articles
TraceTrial-Bench breaks a harness into seven parts: context, execution, governance, lifecycle, observability, tools and verification. Across 4 harnesses, 9 models and 95 tasks, 3,420 runs, harness scores range from 59.8% to 72.0%. The number to remember is this one: "31.1% of runs achieve task success despite failing at least one harness criterion." Almost a third of the green checkmarks came from runs that broke a rule along the way, and a success-only benchmark would have counted them all as wins.
"When Agents Fail: Model or Harness, Who's to Blame?" ran OpenCode, Codex CLI, Pi CLI and DeepAgents across 111 settings and 52,168 trajectories. "Harness rankings vary across models and benchmarks", and task performance and valid completion can favor different pairings. OpenCode ranked first across models, while DeepSeek V4 Flash Preview held a consistently high valid completion rate whichever harness it sat in. There is no single best harness, only best pairings.
The third, accepted as a NeurIPS 2026 poster, mined 57,240 issue and PR threads from 1,774 open-source MCP repos to build realistic tool faults. Under those faults gpt-oss scores fell 60%, from 8.08 to 3.23 out of 10, and the 120B model kept no bigger share of its score than the 20B (p = 0.96). The authors' line: "framework-level error handling matters more than model scale."
Read together, the message for builders is direct. When you switch models, re-run your evals in your own harness, because last month's ranking may not survive the move. Measure valid completion, not just success. And if your MCP tools time out, return partial results or change schemas, the fix is retries and error handling in the framework, not a bigger model.
TraceTrial-Bench: https://arxiv.org/abs/2610.32285 Model or Harness: https://arxiv.org/abs/2610.32483 MCP-Injector: https://arxiv.org/abs/2610.32555
Comments