SAO: the RL trick that keeps long-horizon agents from blowing up
Everyone wants agents that run for hundreds of steps. Almost nobody can train them without the reinforcement learning going unstable. A new paper from the Zhipu/GLM team — Single-Rollout Asynchronous Optimization, arXiv 2607.07508 — is a direct attack on that wall, and it is not a toy: they used it to train GLM-5.2, a 750-billion-parameter model.
The problem it solves is specific and nasty. Async RL is fast because you generate rollouts and update in parallel, but that speed introduces policy lag — the model you sampled from drifts away from the one you are updating, and long agent trajectories amplify the damage. SAO's answer is three moves: a double-sided importance-sampling clip that handles the lag using rollout log-probabilities directly, without dragging around historical checkpoints; a single rollout per prompt instead of GRPO's group sampling, which cuts the off-policy effect; and a skip-observation advantage estimator that ignores environment-feedback tokens and links action values directly.
The numbers are the kind that make you look twice: AIME2025 jumps from 84.2% to 97.3% over GRPO, SWE-Bench Verified from 27.0% to 29.8%, with stable training past a thousand steps. The SWE-Bench gain is modest; the math and stability gains are not, and stability is the point — a training recipe that does not collapse at long horizons is worth more than a few benchmark points.
This is the unglamorous layer the agent boom actually rests on. Products like ChatGPT Work and models like Muse Spark grab the headlines, but they only work because someone figured out how to train a policy that survives hundreds of tool calls without the optimizer eating itself. That someone, increasingly, is a Chinese lab publishing the method in the open. Paper at https://arxiv.org/abs/2607.07508
← Back to all articles
The problem it solves is specific and nasty. Async RL is fast because you generate rollouts and update in parallel, but that speed introduces policy lag — the model you sampled from drifts away from the one you are updating, and long agent trajectories amplify the damage. SAO's answer is three moves: a double-sided importance-sampling clip that handles the lag using rollout log-probabilities directly, without dragging around historical checkpoints; a single rollout per prompt instead of GRPO's group sampling, which cuts the off-policy effect; and a skip-observation advantage estimator that ignores environment-feedback tokens and links action values directly.
The numbers are the kind that make you look twice: AIME2025 jumps from 84.2% to 97.3% over GRPO, SWE-Bench Verified from 27.0% to 29.8%, with stable training past a thousand steps. The SWE-Bench gain is modest; the math and stability gains are not, and stability is the point — a training recipe that does not collapse at long horizons is worth more than a few benchmark points.
This is the unglamorous layer the agent boom actually rests on. Products like ChatGPT Work and models like Muse Spark grab the headlines, but they only work because someone figured out how to train a policy that survives hundreds of tool calls without the optimizer eating itself. That someone, increasingly, is a Chinese lab publishing the method in the open. Paper at https://arxiv.org/abs/2607.07508
Comments