HarnessLens: Evolving the Harness Without Burning the Eval Budget
HarnessLens attacks the least glamorous bottleneck in agent development: verification cost. If you're automatically evolving an agent's harness — its prompts, tools, configs — every candidate change needs testing, and testing everything against every task burns absurd eval budget. Most teams either overpay or under-verify, and aggregate scores happily hide the specific regressions that will bite you in production.
The fix is behavior-aware selective verification. HarnessLens jointly explores the task space and the configurable components, then uses what it calls an attributable-evidence gate: only verify a modification against tasks where that specific behavior actually matters. Across three agent harnesses and four benchmarks it gained 7.6 to 13.6 percent on held-out tasks while spending substantially less eval budget than competing approaches. Code is on GitHub.
The harness is the loudest thread in agent-land right now — DeepSeek shipped one as a product, TaoLive trains models jointly with theirs, and every serious team hand-tunes one. Hand-tuning doesn't scale, and naive auto-tuning drowns in eval cost. Verify-what-matters is the obvious-in-retrospect move, and 'behavior-aware' likely becomes a standard qualifier in this line of work.
Paper: https://arxiv.org/abs/2608.27311
Code: https://github.com/jhxu5214/HarnessLens
← Back to all articles
The fix is behavior-aware selective verification. HarnessLens jointly explores the task space and the configurable components, then uses what it calls an attributable-evidence gate: only verify a modification against tasks where that specific behavior actually matters. Across three agent harnesses and four benchmarks it gained 7.6 to 13.6 percent on held-out tasks while spending substantially less eval budget than competing approaches. Code is on GitHub.
The harness is the loudest thread in agent-land right now — DeepSeek shipped one as a product, TaoLive trains models jointly with theirs, and every serious team hand-tunes one. Hand-tuning doesn't scale, and naive auto-tuning drowns in eval cost. Verify-what-matters is the obvious-in-retrospect move, and 'behavior-aware' likely becomes a standard qualifier in this line of work.
Paper: https://arxiv.org/abs/2608.27311
Code: https://github.com/jhxu5214/HarnessLens
Comments