UniClawBench: An Agent Benchmark That Hides the Grader
Benchmark overfitting is the chronic disease of agent evals: once the grading criteria are public, everyone teaches to the test and the numbers stop meaning anything. UniClawBench, a new benchmark from HKU's MMLab posted July 9, has the cleanest fix so far — it hides the grader.
The setup: 400 real-world tasks, bilingual, executed in live Docker containers instead of pre-recorded sandboxes. Every run is a closed loop between three parties — an executor agent doing the work, a user agent simulating realistic multi-turn interaction, and a hidden supervisor agent that grades without ever revealing its criteria. The agent under test cannot see the rubric, so it cannot game it.
The other useful move is disentanglement. UniClawBench separates what comes from the base model — skill usage, exploration, long-context reasoning, multimodal understanding, cross-platform coordination — from what comes from the framework wrapped around it. That is the question every team building on top of frontier models actually has: is my harness adding anything, or is it all the model?
This lands in the middle of a benchmark arms race — Agents' Last Exam showed top agents passing barely a quarter of real economic tasks, FrontierCode moved the goalposts to mergeability. UniClawBench's hidden-supervisor design is the one most likely to survive contact with the leaderboard chasers.
https://arxiv.org/abs/2607.08768
← Back to all articles
The setup: 400 real-world tasks, bilingual, executed in live Docker containers instead of pre-recorded sandboxes. Every run is a closed loop between three parties — an executor agent doing the work, a user agent simulating realistic multi-turn interaction, and a hidden supervisor agent that grades without ever revealing its criteria. The agent under test cannot see the rubric, so it cannot game it.
The other useful move is disentanglement. UniClawBench separates what comes from the base model — skill usage, exploration, long-context reasoning, multimodal understanding, cross-platform coordination — from what comes from the framework wrapped around it. That is the question every team building on top of frontier models actually has: is my harness adding anything, or is it all the model?
This lands in the middle of a benchmark arms race — Agents' Last Exam showed top agents passing barely a quarter of real economic tasks, FrontierCode moved the goalposts to mergeability. UniClawBench's hidden-supervisor design is the one most likely to survive contact with the leaderboard chasers.
https://arxiv.org/abs/2607.08768
Comments