8,000 SWE-bench runs: agent memory moved the resolve rate by less than half a point
Memory startups have a new awkward number. MemUse, a paper titled "Delivered, Not Used", ran 8,000 SWE-bench Verified episodes through three memory systems, and resolve rates stayed within 0.4 points of a no-memory baseline.
The breakdown is the interesting part. Most delivered entries were irrelevant. Most relevant ones went unused. In two of the systems, most deliveries just restated what the agent could still see. And the outcomes barely track the memory anyway: 23.2% of episodes with beneficial memory use still failed, and 5 of 16 with harmful use still succeeded. Retrieval quality is what the industry benchmarks, and retrieval looks like the wrong place to measure. The bottleneck is whether the model does anything with what it gets. SWE-bench is one narrow domain with mostly one-shot tasks, so this does not prove memory is useless, only that this common pitch has yet to show up in this common test.
ARCBench finds something uglier about compaction. You would assume the danger is that summarizing deletes a rule. It rarely does. Most post-compaction violations happen with the rule still sitting in the history, and compacted runs that retain the rule violate more than twice as often as runs with the full history. The agent can see the rule. It has lost the weight of it.
A third paper explains why state beats logs. "Agent Memory as Rate-Limited Filtering" has models read a full access log: 7B to 32B open models fall from 72 to 86% at 8 to 16 events down to 48 to 55% at 256, while reasoning-trained 32B models reading the current state hit 96 and 98%. A plain parse-then-apply filter scores 80 to 89% at 256 and beats retrieval, model-kept ledgers and Mem0. Boring code wins again.
The takeaway for anyone building this layer: stop reporting what you retrieved and start reporting what changed. If you cannot show the agent acted differently because of the memory, what you built is an expensive way to add tokens to the prompt.
MemUse: https://arxiv.org/abs/2611.01718 ARCBench: https://arxiv.org/abs/2611.03223 Rate-Limited Filtering: https://arxiv.org/abs/2611.02361
← Back to all articles
The breakdown is the interesting part. Most delivered entries were irrelevant. Most relevant ones went unused. In two of the systems, most deliveries just restated what the agent could still see. And the outcomes barely track the memory anyway: 23.2% of episodes with beneficial memory use still failed, and 5 of 16 with harmful use still succeeded. Retrieval quality is what the industry benchmarks, and retrieval looks like the wrong place to measure. The bottleneck is whether the model does anything with what it gets. SWE-bench is one narrow domain with mostly one-shot tasks, so this does not prove memory is useless, only that this common pitch has yet to show up in this common test.
ARCBench finds something uglier about compaction. You would assume the danger is that summarizing deletes a rule. It rarely does. Most post-compaction violations happen with the rule still sitting in the history, and compacted runs that retain the rule violate more than twice as often as runs with the full history. The agent can see the rule. It has lost the weight of it.
A third paper explains why state beats logs. "Agent Memory as Rate-Limited Filtering" has models read a full access log: 7B to 32B open models fall from 72 to 86% at 8 to 16 events down to 48 to 55% at 256, while reasoning-trained 32B models reading the current state hit 96 and 98%. A plain parse-then-apply filter scores 80 to 89% at 256 and beats retrieval, model-kept ledgers and Mem0. Boring code wins again.
The takeaway for anyone building this layer: stop reporting what you retrieved and start reporting what changed. If you cannot show the agent acted differently because of the memory, what you built is an expensive way to add tokens to the prompt.
MemUse: https://arxiv.org/abs/2611.01718 ARCBench: https://arxiv.org/abs/2611.03223 Rate-Limited Filtering: https://arxiv.org/abs/2611.02361
Comments