Four memory papers in one day: plain retrieval wins, and delete doesn't delete
The agent memory layer had a rough Friday on arXiv. Four papers landed within a day of each other, and together they say the fancy part of memory is mostly not helping, while the boring part is quietly broken.
Start with Raw-RAG from Xi'an Jiaotong. The question in the title is "Is Complex Agentic Memory Always Needed?", and the answer is no. Plain dense retrieval plus a reranker over the raw conversation history ranked first in 6 of 7 benchmark and model settings across LoCoMo, LongMemEval, MEME and MemoryAgentBench, beating the best agentic memory system by 0.13 to 5.53 points. Even giving the model full access to the transformed memory store did worse than handing it the raw evidence. Every summarize, extract and consolidate step throws something away, and the benchmarks notice. Paper: https://arxiv.org/abs/2610.30755 Code: https://github.com/Litmeb/RAW-RAG
Then the scary one. AfterDeleteBench asks what happens after you delete a memory record. A single query recovered a deleted secret from Qwen2.5-7B in about two thirds of cases, while controls that never saw the secret recovered zero. With only the separately stored pieces left behind, exact reconstruction worked 62 to 99 percent of the time across Qwen, Mistral and Phi-3. Deleting the source does nothing about the summaries, embeddings and notes derived from it. Provenance-aware closure, deleting everything downstream of the record, eliminated recovery. The authors' line is that deletion has to be tested by behavior, not as a storage operation. Anyone promising users a "forget this" button should read it: https://arxiv.org/abs/2610.31366
The third paper corrupts not the memories but the bookkeeping about which memories are valid. Across six providers and 2,080 matched fault pairs, flipping validity state with content untouched changed the operative context 96.25 percent of the time and cost 40.54 points on LongMemEval. Oracle enforcement restored all of it: https://arxiv.org/abs/2610.30698 A fourth, pre-registered embodied study found that distilling memory into weights gave a non-significant 1 point, while attaching a frozen retrieval buffer added 34: https://arxiv.org/abs/2610.30478
Put that next to last week's MemoryTax result, where Opus 5.5 plus raw-history search hit 99 percent at under half the cost, and the pattern is hard to miss. Strong models want the raw record and a good search box. The engineering that actually matters is provenance: knowing what came from where, what is still valid, and what has to die when a source is deleted. Memory startups selling clever consolidation are building the part the benchmarks keep voting down.
← Back to all articles
Start with Raw-RAG from Xi'an Jiaotong. The question in the title is "Is Complex Agentic Memory Always Needed?", and the answer is no. Plain dense retrieval plus a reranker over the raw conversation history ranked first in 6 of 7 benchmark and model settings across LoCoMo, LongMemEval, MEME and MemoryAgentBench, beating the best agentic memory system by 0.13 to 5.53 points. Even giving the model full access to the transformed memory store did worse than handing it the raw evidence. Every summarize, extract and consolidate step throws something away, and the benchmarks notice. Paper: https://arxiv.org/abs/2610.30755 Code: https://github.com/Litmeb/RAW-RAG
Then the scary one. AfterDeleteBench asks what happens after you delete a memory record. A single query recovered a deleted secret from Qwen2.5-7B in about two thirds of cases, while controls that never saw the secret recovered zero. With only the separately stored pieces left behind, exact reconstruction worked 62 to 99 percent of the time across Qwen, Mistral and Phi-3. Deleting the source does nothing about the summaries, embeddings and notes derived from it. Provenance-aware closure, deleting everything downstream of the record, eliminated recovery. The authors' line is that deletion has to be tested by behavior, not as a storage operation. Anyone promising users a "forget this" button should read it: https://arxiv.org/abs/2610.31366
The third paper corrupts not the memories but the bookkeeping about which memories are valid. Across six providers and 2,080 matched fault pairs, flipping validity state with content untouched changed the operative context 96.25 percent of the time and cost 40.54 points on LongMemEval. Oracle enforcement restored all of it: https://arxiv.org/abs/2610.30698 A fourth, pre-registered embodied study found that distilling memory into weights gave a non-significant 1 point, while attaching a frozen retrieval buffer added 34: https://arxiv.org/abs/2610.30478
Put that next to last week's MemoryTax result, where Opus 5.5 plus raw-history search hit 99 percent at under half the cost, and the pattern is hard to miss. Strong models want the raw record and a good search box. The engineering that actually matters is provenance: knowing what came from where, what is still valid, and what has to die when a source is deleted. Memory startups selling clever consolidation are building the part the benchmarks keep voting down.
Comments