Why RL trains many skills at once and SFT can't
Top paper on Hugging Face today, and it answers a question anyone training an agent on more than one task has hit face-first: why does supervised fine-tuning on a mixed task set make everything slightly worse, while RL on the same mix mostly doesn't?
The answer the authors give is geometric. RL produces sparse, approximately orthogonal parameter updates across tasks β the tasks touch different directions and largely leave each other alone. SFT produces updates that collide. They trace it to the type of interference: RL is variance-limited, SFT is norm-limited. Same data, same tasks, completely different interference structure, and it falls out of how the two objectives generate gradients rather than from anything about the tasks themselves.
Then they build on it. Parallel-RL is a training setup that exploits the orthogonality directly β train tasks in parallel, compose them, add capabilities without a full remix and retrain of everything. Code is at github.com/GaryStack/Parallel-RL. arXiv 2608.03573, submitted August 4, revised August 6, out of a Tsinghua and CAS group.
Why it matters for agents specifically: an agent is a multi-task model whether you designed it that way or not. Browse, code, call tools, follow formats, recover from errors. Everybody has been mixing that soup by hand and watching one capability sag when another improves. If RL updates really are near-orthogonal, capability composition stops being a dark art and starts being an operation you can plan.
https://arxiv.org/abs/2608.03573
← Back to all articles
The answer the authors give is geometric. RL produces sparse, approximately orthogonal parameter updates across tasks β the tasks touch different directions and largely leave each other alone. SFT produces updates that collide. They trace it to the type of interference: RL is variance-limited, SFT is norm-limited. Same data, same tasks, completely different interference structure, and it falls out of how the two objectives generate gradients rather than from anything about the tasks themselves.
Then they build on it. Parallel-RL is a training setup that exploits the orthogonality directly β train tasks in parallel, compose them, add capabilities without a full remix and retrain of everything. Code is at github.com/GaryStack/Parallel-RL. arXiv 2608.03573, submitted August 4, revised August 6, out of a Tsinghua and CAS group.
Why it matters for agents specifically: an agent is a multi-task model whether you designed it that way or not. Browse, code, call tools, follow formats, recover from errors. Everybody has been mixing that soup by hand and watching one capability sag when another improves. If RL updates really are near-orthogonal, capability composition stops being a dark art and starts being an operation you can plan.
https://arxiv.org/abs/2608.03573
Comments