Compaction keeps the spec and drops the rules
Two papers in Monday's arXiv batch measured what happens to a coding agent's context when it gets compacted, and they agree on the uncomfortable part. The summary keeps what you asked for and loses how you told the agent to behave.
Compact-Bench tested 14 summarizer backbones on 35 SWE-bench repairs, 146 agentic sessions and 106 production chains. "LLM summaries retain at most 16.0% of recurring details, compared with 79.1% for mechanical extraction." Production summaries keep 15.7% at the first boundary. Damage ranges from 5% to 64% of the runs that succeeded with full history, and one repair broke under 13 of 14 backbones. The harness prompt matters as much as the model: with Kimi-K3 summarizing, the Kimi Code prompt preserved the most task continuations (19 of 22), while the Claude Code prompt kept the most recurring details.
The second paper, "Benchmarks Are Surprisingly Compaction-Friendly", ran five compaction strategies in OpenCode on PaperBench, ProgramBench and PostTrainBench. The headline sounds reassuring: for the two 1M-context models the threshold did not move scores, and nothing beat default self-summarization. The details are not. On ProgramBench "the first compaction summary already drops about one third of the task's anticheating rules", and it gets worse with every compaction. On real SWE-chat sessions, instructions about how the agent should work and what it must not do are dropped "around 6 times more" than specs of what to build.
Put together, that explains a failure every heavy agent user has seen: three hours in, the agent still knows the feature but has forgotten "don't touch the migrations" and "run the tests before you commit". Benchmarks miss it because they score the output, not the rules. The practical fix is boring and works: keep constraints in a file the harness re-reads on every turn instead of trusting them to survive a summary, and extract recurring details mechanically instead of asking a model to paraphrase them. A third paper in the same batch, a census of AGENTS.md and CLAUDE.md across 22,854 active GitHub repos, found such files in 26.3% of them and found that 74.0% of their directives give no reason. A rule with no reason is exactly the kind of line a summarizer decides is safe to drop.
Compact-Bench: https://arxiv.org/abs/2610.31710 Compaction-Friendly: https://arxiv.org/abs/2610.31778 AGENTS.md census: https://arxiv.org/abs/2610.31715
← Back to all articles
Compact-Bench tested 14 summarizer backbones on 35 SWE-bench repairs, 146 agentic sessions and 106 production chains. "LLM summaries retain at most 16.0% of recurring details, compared with 79.1% for mechanical extraction." Production summaries keep 15.7% at the first boundary. Damage ranges from 5% to 64% of the runs that succeeded with full history, and one repair broke under 13 of 14 backbones. The harness prompt matters as much as the model: with Kimi-K3 summarizing, the Kimi Code prompt preserved the most task continuations (19 of 22), while the Claude Code prompt kept the most recurring details.
The second paper, "Benchmarks Are Surprisingly Compaction-Friendly", ran five compaction strategies in OpenCode on PaperBench, ProgramBench and PostTrainBench. The headline sounds reassuring: for the two 1M-context models the threshold did not move scores, and nothing beat default self-summarization. The details are not. On ProgramBench "the first compaction summary already drops about one third of the task's anticheating rules", and it gets worse with every compaction. On real SWE-chat sessions, instructions about how the agent should work and what it must not do are dropped "around 6 times more" than specs of what to build.
Put together, that explains a failure every heavy agent user has seen: three hours in, the agent still knows the feature but has forgotten "don't touch the migrations" and "run the tests before you commit". Benchmarks miss it because they score the output, not the rules. The practical fix is boring and works: keep constraints in a file the harness re-reads on every turn instead of trusting them to survive a summary, and extract recurring details mechanically instead of asking a model to paraphrase them. A third paper in the same batch, a census of AGENTS.md and CLAUDE.md across 22,854 active GitHub repos, found such files in 26.3% of them and found that 74.0% of their directives give no reason. A rule with no reason is exactly the kind of line a summarizer decides is safe to drop.
Compact-Bench: https://arxiv.org/abs/2610.31710 Compaction-Friendly: https://arxiv.org/abs/2610.31778 AGENTS.md census: https://arxiv.org/abs/2610.31715
Comments