OSReward: The AI Judges Grading Agents Are Too Soft
OSReward measured something the whole computer-use agent field has been assuming works: AI judges grading agent runs. It found they don't. The HKU OS-Copilot team built the most comprehensive evaluation of VLM judges to date — real computer-use trajectories from diverse agent backbones, human-verified instructions across platforms, ground-truth verdicts from multi-stage human annotation — and every state-of-the-art model falls short, sharing the same systematic failure: a leniency bias that labels failed runs as successes. The few judges reliable enough to trust are too expensive to run at scale. The affordable open models trail far behind. The paper pulled 60 upvotes on HuggingFace daily papers.
Sit with what leniency bias means for training. RL for computer-use agents lives and dies on the reward signal. If your judge says "pass" when the run actually failed, you aren't just measuring wrong — you're training failure in, at scale, with confidence. Every GRPO-style pipeline using a VLM judge as verifier inherits this bias silently.
The team's answer is practical: OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments, plus OS-Shepherd 9B and 35B — open reward models trained on it that deliver cheap, stable, reliable reward signals. Plus OSReward-Hard for the genuinely difficult cases and OSReward-Multi for fine-grained efficiency scoring.
This is the third leg of what's shaping up as the grading crisis. Who Grades the Grader showed reward models failing on reasoning. ScaleX showed human overseers at 66.3% accuracy, approving one in three malicious actions. Now the VLM judges watching computer-use agents turn out to be systematically soft. The bottleneck on agent progress keeps moving to the same place: not the agent, the referee.
Paper: arxiv.org/abs/2607.28609 — Code: github.com/OS-Copilot/OSReward
← Back to all articles
Sit with what leniency bias means for training. RL for computer-use agents lives and dies on the reward signal. If your judge says "pass" when the run actually failed, you aren't just measuring wrong — you're training failure in, at scale, with confidence. Every GRPO-style pipeline using a VLM judge as verifier inherits this bias silently.
The team's answer is practical: OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments, plus OS-Shepherd 9B and 35B — open reward models trained on it that deliver cheap, stable, reliable reward signals. Plus OSReward-Hard for the genuinely difficult cases and OSReward-Multi for fine-grained efficiency scoring.
This is the third leg of what's shaping up as the grading crisis. Who Grades the Grader showed reward models failing on reasoning. ScaleX showed human overseers at 66.3% accuracy, approving one in three malicious actions. Now the VLM judges watching computer-use agents turn out to be systematically soft. The bottleneck on agent progress keeps moving to the same place: not the agent, the referee.
Paper: arxiv.org/abs/2607.28609 — Code: github.com/OS-Copilot/OSReward
Comments