AgentOPSD Finds the Three Turns That Actually Mattered
The top agent paper on Hugging Face today (66 upvotes) is AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning, from a Tsinghua-led team (arxiv.org/abs/2608.05987, code at github.com/ZethWang/AgentOPSD). It attacks the problem that quietly bounds all agent RL: credit assignment. A thirty-turn task fails or succeeds as a whole, the reward arrives at the end, and trajectory-level methods like GRPO smear that single bit evenly across every step β even though usually two or three decisions determined the outcome.
The mechanism is critic-free, which matters for anyone who has tried training a value model on long horizons. Instead of a critic, AgentOPSD compares token-level probability divergences between a teacher and student model and aggregates them into turn-level evidence, maintained as a recursive Bayesian belief state in log-odds space. Where beliefs revise sharply, that turn mattered; the sparse end-of-episode reward gets reweighted accordingly. The sparse signal becomes dense without ever learning a separate critic.
Results: 89.1% success on ALFWorld with a Qwen2.5-7B base, beating GRPO and self-distillation baselines, with the pattern holding on WebShop and Search-QA. Ablations confirm the turn-level aggregation is what drives it. This slots into the clearest trend in agent training right now β SEED's on-policy distillation, ABSeeker's answer-backtracked dense rewards last week, now this: everyone is converging on the same diagnosis, that agent RL is not compute-limited but signal-limited, and the wins come from squeezing more information out of each expensive rollout.
← Back to all articles
The mechanism is critic-free, which matters for anyone who has tried training a value model on long horizons. Instead of a critic, AgentOPSD compares token-level probability divergences between a teacher and student model and aggregates them into turn-level evidence, maintained as a recursive Bayesian belief state in log-odds space. Where beliefs revise sharply, that turn mattered; the sparse end-of-episode reward gets reweighted accordingly. The sparse signal becomes dense without ever learning a separate critic.
Results: 89.1% success on ALFWorld with a Qwen2.5-7B base, beating GRPO and self-distillation baselines, with the pattern holding on WebShop and Search-QA. Ablations confirm the turn-level aggregation is what drives it. This slots into the clearest trend in agent training right now β SEED's on-policy distillation, ABSeeker's answer-backtracked dense rewards last week, now this: everyone is converging on the same diagnosis, that agent RL is not compute-limited but signal-limited, and the wins come from squeezing more information out of each expensive rollout.
Comments