This paper says everyone's been optimizing the wrong policy in LLM RL
A team out of Alibaba's Taobao group put out a paper with a blunt title: The Mirage of Optimizing Training Policies. The claim is that in RL for LLMs, we've been optimizing the wrong thing. During training, the model's probabilities come from the training engine; at deployment, they come from a different inference engine, and those two disagree. So you can push the training policy up and up while the thing you actually ship, the inference policy, barely moves or even gets worse. You optimized a number that isn't the number.
Their fix is a method called MIPU, Monotonic Inference Policy Update. Instead of trusting the training-side gradient, it builds candidate updates referenced to the sampler and only accepts the ones a cheap inference-side proxy says will actually improve the deployed model. The point is monotonic improvement of the policy you deploy, not the one you train, and they show it stabilizes training and lifts reasoning across model scales. It's the top paper on HuggingFace's daily list right now with 140 upvotes.
Why care if you're not training models. This training-inference mismatch is one of the quiet reasons RL runs blow up or plateau in ways nobody can explain, and it's mostly been treated as an engineering annoyance to patch over. Reframing it as the actual objective, saying the mismatch is the whole game and here's how to optimize through it, is the kind of move that changes how the next generation of agent-RL recipes get written. If it holds up, a lot of teams are chasing the wrong loss. Link: arxiv.org/abs/2606.29526
← Back to all articles
Their fix is a method called MIPU, Monotonic Inference Policy Update. Instead of trusting the training-side gradient, it builds candidate updates referenced to the sampler and only accepts the ones a cheap inference-side proxy says will actually improve the deployed model. The point is monotonic improvement of the policy you deploy, not the one you train, and they show it stabilizes training and lifts reasoning across model scales. It's the top paper on HuggingFace's daily list right now with 140 upvotes.
Why care if you're not training models. This training-inference mismatch is one of the quiet reasons RL runs blow up or plateau in ways nobody can explain, and it's mostly been treated as an engineering annoyance to patch over. Reframing it as the actual objective, saying the mismatch is the whole game and here's how to optimize through it, is the kind of move that changes how the next generation of agent-RL recipes get written. If it holds up, a lot of teams are chasing the wrong loss. Link: arxiv.org/abs/2606.29526
Comments