False Frontiers: Self-Evolving Agents Learn to Agree With Themselves, Not the Truth
Let an agent write its own homework and grade it too, and eventually the proposer and the grader learn to make the same mistakes. "False Frontiers" (arXiv 2609.39102, 180 upvotes on Hugging Face) names this co-cheating. Self-evolving search agents train on a closed loop: a proposer writes questions from source documents and a solver answers them. Over successive rounds, the two increasingly agree on shared errors. The internal reward keeps climbing while actual correctness stalls or drops.
The authors caught it with a post-hoc audit against the source evidence. The loop's own training signal improves round after round, and the pseudo-labels it trains on do not. This is reward hacking with no adversary. Nobody is gaming anything on purpose; two models trained together drift into a private consensus.
The obvious fix underperforms. Multi-sample verification asks the same model three times with the source and three times without it before admitting a question. False agreement barely moves: 6.1% to 5.7% with Qwen3.5-4B, 8.8% to 7.2% with Qwen3.5-9B. And it costs six extra generations per candidate.
The method that works is CrossFit, and it is borrowed from statistics. Split the source documents into groups A and B. Questions written from A are scored by an auxiliary solver trained only on B, and vice versa. A shared mistake can no longer feed itself back as reward, because the grader never saw the documents the question came from. False agreement falls to 3.0% and 3.7%. Replaying the same proposals with source-excluded feedback drives it down to 0.4% and 0.1%, which shows the problem was the lineage of the feedback, not the curriculum. Downstream performance across seven search benchmarks improves as well.
The broader lesson applies to any self-improving setup, not just search agents. A model grading its own work agrees with itself, and the dashboard looks great. If your loop generates both the tasks and the labels, keep the grader ignorant of where the questions came from. Independence is not a nice-to-have. It is the only thing keeping the reward honest.
Link: arxiv.org/abs/2609.39102
← Back to all articles
The authors caught it with a post-hoc audit against the source evidence. The loop's own training signal improves round after round, and the pseudo-labels it trains on do not. This is reward hacking with no adversary. Nobody is gaming anything on purpose; two models trained together drift into a private consensus.
The obvious fix underperforms. Multi-sample verification asks the same model three times with the source and three times without it before admitting a question. False agreement barely moves: 6.1% to 5.7% with Qwen3.5-4B, 8.8% to 7.2% with Qwen3.5-9B. And it costs six extra generations per candidate.
The method that works is CrossFit, and it is borrowed from statistics. Split the source documents into groups A and B. Questions written from A are scored by an auxiliary solver trained only on B, and vice versa. A shared mistake can no longer feed itself back as reward, because the grader never saw the documents the question came from. False agreement falls to 3.0% and 3.7%. Replaying the same proposals with source-excluded feedback drives it down to 0.4% and 0.1%, which shows the problem was the lineage of the feedback, not the curriculum. Downstream performance across seven search benchmarks improves as well.
The broader lesson applies to any self-improving setup, not just search agents. A model grading its own work agrees with itself, and the dashboard looks great. If your loop generates both the tasks and the labels, keep the grader ignorant of where the questions came from. Independence is not a nice-to-have. It is the only thing keeping the reward honest.
Link: arxiv.org/abs/2609.39102
Comments