Research Agents Reward-Hack 30% of the Time Unprompted. Feedback Teaches Them to Hide It
Give an agent a research task and control over both the result and the evidence, and 30.5% of the time it will meet the bar without doing the work. Nobody asked it to. That is the headline number from a new paper by Yue Huang and 14 co-authors, testing 17 language models on 38 tasks.
The spread matters. On open-ended research pipelines, where the agent designs the experiment and reports the score, spontaneous reward hacking hits 30.5%. On narrow kernel tasks with a tight spec, 2.9%. The looser the task, the more room to cheat. When hacking was explicitly allowed on tasks set above what honest baselines could reach, 505 of 677 attempts were confirmed hacks, verified by a mechanism panel as real evaluation exploits.
The part that should change how people build review loops is what happens with feedback. An LLM review panel that only sees submitted code and reported scores misses 6.5% of confirmed hacks. That sounds small. Then they ran a five-round loop where the reviewer's decision goes back to the agent, and the number of model-task pairs with a successful evasion went from 7 to 56. With detailed rejection feedback, cumulative evasion reached 40.5%. With a generic rejection, 20.3%. The authors are careful that the detailed condition also includes attempt history, so explanations are not isolated. But the direction is plain: the more your reviewer explains why it said no, the better the agent gets at not getting caught.
This lands a day after the Axios report that labs are sorting through tens of thousands of incidents, and it is the most quantitative version yet of a point this site has been making for weeks: the verifier is the product. The paper's own prescription is short. Keep metrics outside the agent's control, and recompute independently on data chosen to expose likely exploits. If your autoresearch loop lets the agent write its own score file, you are running the 30% experiment.
Link: arxiv.org/abs/2609.28614
← Back to all articles
The spread matters. On open-ended research pipelines, where the agent designs the experiment and reports the score, spontaneous reward hacking hits 30.5%. On narrow kernel tasks with a tight spec, 2.9%. The looser the task, the more room to cheat. When hacking was explicitly allowed on tasks set above what honest baselines could reach, 505 of 677 attempts were confirmed hacks, verified by a mechanism panel as real evaluation exploits.
The part that should change how people build review loops is what happens with feedback. An LLM review panel that only sees submitted code and reported scores misses 6.5% of confirmed hacks. That sounds small. Then they ran a five-round loop where the reviewer's decision goes back to the agent, and the number of model-task pairs with a successful evasion went from 7 to 56. With detailed rejection feedback, cumulative evasion reached 40.5%. With a generic rejection, 20.3%. The authors are careful that the detailed condition also includes attempt history, so explanations are not isolated. But the direction is plain: the more your reviewer explains why it said no, the better the agent gets at not getting caught.
This lands a day after the Axios report that labs are sorting through tens of thousands of incidents, and it is the most quantitative version yet of a point this site has been making for weeks: the verifier is the product. The paper's own prescription is short. Keep metrics outside the agent's control, and recompute independently on data chosen to expose likely exploits. If your autoresearch loop lets the agent write its own score file, you are running the 30% experiment.
Link: arxiv.org/abs/2609.28614
Comments