Co-RL: Models Grading Themselves Collapse. A Cohort Doesn't
Self-rewarding RL has an incest problem. Let a model generate its own training reward and it amplifies its own biases until responses homogenize and training collapses, the failure mode stalking every self-improvement pipeline. Co-RL, at 82 upvotes on HuggingFace's daily board, proposes the fix evolution found first: diversity. Train a cohort of models simultaneously, sharing no parameters, drawn from different model families at different sizes, and derive each one's reward from its peers instead of itself.
The mechanism works because errors that survive self-grading are exactly the ones a different model, with different biases, catches. The cohort's disagreement is the training signal, no ground-truth labels, no human annotation, no judge model standing outside the loop. On seven text benchmarks Co-RL delivers 3.0 to 8.6 percent average gains, on four multimodal benchmarks 2.3 to 7.2 percent, matching or beating supervised methods that needed labels, while maintaining the behavioral diversity that keeps training from collapsing.
The result lands on the same conclusion as this week's other supervision-free work, SPADE, which has one model generate its own training environments: the bottleneck in agent training is no longer data or reward engineering, it is escaping the gravity of a single model's blind spots. Co-RL's answer is peer review, structurally, not metaphorically. A homogeneous committee rubber-stamps; a diverse one catches errors.
For the agent world the implication is concrete: multi-agent systems already run heterogeneous cohorts in production. Co-RL says that same heterogeneity is a training resource, the disagreement between your agents is free supervision, if you harvest it.
https://arxiv.org/abs/2608.17253
← Back to all articles
The mechanism works because errors that survive self-grading are exactly the ones a different model, with different biases, catches. The cohort's disagreement is the training signal, no ground-truth labels, no human annotation, no judge model standing outside the loop. On seven text benchmarks Co-RL delivers 3.0 to 8.6 percent average gains, on four multimodal benchmarks 2.3 to 7.2 percent, matching or beating supervised methods that needed labels, while maintaining the behavioral diversity that keeps training from collapsing.
The result lands on the same conclusion as this week's other supervision-free work, SPADE, which has one model generate its own training environments: the bottleneck in agent training is no longer data or reward engineering, it is escaping the gravity of a single model's blind spots. Co-RL's answer is peer review, structurally, not metaphorically. A homogeneous committee rubber-stamps; a diverse one catches errors.
For the agent world the implication is concrete: multi-agent systems already run heterogeneous cohorts in production. Co-RL says that same heterogeneity is a training resource, the disagreement between your agents is free supervision, if you harvest it.
https://arxiv.org/abs/2608.17253
Comments