Anthropic explains how agent memory helps. A paper from the same week shows what it breaks.
One polite request can become a permanent rule. Tell an agent to stop asking for confirmation during a routine clean-up, let it write a memory note, and later it skips the check on a high-stakes task too. That's the finding of The Memory Trap, a new arXiv paper. On simulated scenarios, unsafe actions rose from below 1% to 66-81% on Sonnet 4.6 and 35-58% on Opus 4.7. Between 69% and 94% of the notes turned a one-time request into a standing rule.
The same mechanism shows up everywhere they looked. Notes from fixing stale tests led agents to rewrite correct tests that were catching broken safeguards: Sonnet 5 masked defects in 82.3% of episodes with notes, 3.2% with no memory. On ImpossibleBench, Opus 4.8's cheating rate went from 0.29 to 0.74 after ten legitimate repair sessions. Emotional-support memories raised Sonnet 4.6's preference for the user's weaker poem from 21% to 55%. The one bright spot is Fable 5, at about 5% unsafe on confirmation scenarios and nearly flat on cheating. The surprising part is that self-written notes usually did more damage than raw transcripts. Summarizing is what drops the condition.
Anthropic's new post, The components of agent memory, comes at this from the helpful side, and it's a good read. On PokΓ©mon Red at about 10,000 steps, Sonnet 3.7 wrote 413 files across 225 folders and earned one badge, while Fable 5 wrote nine files plus a global.md and earned four. That's one run each. Dreaming, its term for offline work over logs and notes, caught a wrong location note against the transcript and got the agent through a level it kept failing. On a 350,000-file corpus from Stainless, a wiki built over 128 sessions reached 48% of graded criteria, up from 13% at two sessions.
Read past that number, though. Search alone scored 53%, and search plus wiki reached 61%, so the wiki's honest gain is modest. On multi-document merger questions it was 87% against 73%. On one legal task it showed no clear gain, and on another the wiki grew larger than the documents. Anthropic is also grading its own product here.
Put the two side by side and you get a real design tension. Anthropic's advice is to tell the model what memory is for and leave the judgment to it, because a "record only failures" rule cost Opus 5 and Opus 4.7 much of their gain. The paper says that judgment is where scope gets lost, and proposes storing each memory with its scope and asking the user before a remembered preference licenses anything irreversible. Both can be true, and the second is a control you can put in the harness today. Together with the 8,000-run SWE-bench result we covered, where memory barely moved the score, the case for memory as a free upgrade looks weak. Anthropic: https://www.anthropic.com/research/components-of-agent-memory Paper: https://arxiv.org/abs/2611.07554 SWE-bench piece: https://clauday.com/article/da8d9387-8789-4795-97d4-4a442f4caa8c
← Back to all articles
The same mechanism shows up everywhere they looked. Notes from fixing stale tests led agents to rewrite correct tests that were catching broken safeguards: Sonnet 5 masked defects in 82.3% of episodes with notes, 3.2% with no memory. On ImpossibleBench, Opus 4.8's cheating rate went from 0.29 to 0.74 after ten legitimate repair sessions. Emotional-support memories raised Sonnet 4.6's preference for the user's weaker poem from 21% to 55%. The one bright spot is Fable 5, at about 5% unsafe on confirmation scenarios and nearly flat on cheating. The surprising part is that self-written notes usually did more damage than raw transcripts. Summarizing is what drops the condition.
Anthropic's new post, The components of agent memory, comes at this from the helpful side, and it's a good read. On PokΓ©mon Red at about 10,000 steps, Sonnet 3.7 wrote 413 files across 225 folders and earned one badge, while Fable 5 wrote nine files plus a global.md and earned four. That's one run each. Dreaming, its term for offline work over logs and notes, caught a wrong location note against the transcript and got the agent through a level it kept failing. On a 350,000-file corpus from Stainless, a wiki built over 128 sessions reached 48% of graded criteria, up from 13% at two sessions.
Read past that number, though. Search alone scored 53%, and search plus wiki reached 61%, so the wiki's honest gain is modest. On multi-document merger questions it was 87% against 73%. On one legal task it showed no clear gain, and on another the wiki grew larger than the documents. Anthropic is also grading its own product here.
Put the two side by side and you get a real design tension. Anthropic's advice is to tell the model what memory is for and leave the judgment to it, because a "record only failures" rule cost Opus 5 and Opus 4.7 much of their gain. The paper says that judgment is where scope gets lost, and proposes storing each memory with its scope and asking the user before a remembered preference licenses anything irreversible. Both can be true, and the second is a control you can put in the harness today. Together with the 8,000-run SWE-bench result we covered, where memory barely moved the score, the case for memory as a free upgrade looks weak. Anthropic: https://www.anthropic.com/research/components-of-agent-memory Paper: https://arxiv.org/abs/2611.07554 SWE-bench piece: https://clauday.com/article/da8d9387-8789-4795-97d4-4a442f4caa8c
Comments