Your Distilled Model Will Not Shut Up, and Now We Know Why
If you have run on-policy distillation you have probably hit this: the student starts producing answers that are far longer than the teacher's, sometimes long enough to burn the entire generation budget and never terminate. Everyone has a folk explanation. A paper from Microsoft Research and collaborators, arXiv 2609.20511, went and found the actual cause, and it is dumber and more fixable than the folk explanations.
The problem is the stop token. A base student and a post-trained teacher can put their stopping probability on different EOS tokens even when their declared stopping sets are identical. The student wants to stop using one token, the teacher's distribution says that token is not where the mass goes, and the distillation objective duly pushes the student off its own termination action without reliably handing it the teacher's alternative. The student ends up with no confident way to stop. So it does not.
They show this holds across Qwen3, Llama and Gemma, which is what makes it worth reading rather than filing as a quirk of one tokenizer. Three families, three vocabularies, same failure. And the obvious fix does not work: aligning the decoding stopping set alone is not enough, because the sets were already the same in the failing cases. What works is treating functionally equivalent EOS tokens as one shared semantic stopping action, so the objective stops penalizing the student for wanting to end the sentence in its own dialect. Code is released.
Why this belongs on an agents site rather than a training-methods site: length inflation is a bill. Every distilled model in an agent loop that runs twenty percent long is twenty percent more expensive per step, and agent loops have a lot of steps. The whole point of distilling is to get a cheaper model, and a silent termination bug can eat the margin you distilled for in the first place. Worse, it is invisible on accuracy benchmarks. The model gets the answer right, eventually, at length, and your eval is green while your inference spend is not.
It also fits a pattern that has been showing up all month. HarnessTax found a five-times cost spread across harnesses on identical tasks. Zoom's harness design study found per-component effects nobody had measured. Now a token-level accounting bug explains a chunk of distillation cost. The expensive part of running agents keeps turning out to be plumbing rather than intelligence. Paper at arxiv.org/abs/2609.20511.
← Back to all articles
The problem is the stop token. A base student and a post-trained teacher can put their stopping probability on different EOS tokens even when their declared stopping sets are identical. The student wants to stop using one token, the teacher's distribution says that token is not where the mass goes, and the distillation objective duly pushes the student off its own termination action without reliably handing it the teacher's alternative. The student ends up with no confident way to stop. So it does not.
They show this holds across Qwen3, Llama and Gemma, which is what makes it worth reading rather than filing as a quirk of one tokenizer. Three families, three vocabularies, same failure. And the obvious fix does not work: aligning the decoding stopping set alone is not enough, because the sets were already the same in the failing cases. What works is treating functionally equivalent EOS tokens as one shared semantic stopping action, so the objective stops penalizing the student for wanting to end the sentence in its own dialect. Code is released.
Why this belongs on an agents site rather than a training-methods site: length inflation is a bill. Every distilled model in an agent loop that runs twenty percent long is twenty percent more expensive per step, and agent loops have a lot of steps. The whole point of distilling is to get a cheaper model, and a silent termination bug can eat the margin you distilled for in the first place. Worse, it is invisible on accuracy benchmarks. The model gets the answer right, eventually, at length, and your eval is green while your inference spend is not.
It also fits a pattern that has been showing up all month. HarnessTax found a five-times cost spread across harnesses on identical tasks. Zoom's harness design study found per-component effects nobody had measured. Now a token-level accounting bug explains a chunk of distillation cost. The expensive part of running agents keeps turning out to be plumbing rather than intelligence. Paper at arxiv.org/abs/2609.20511.
Comments