Your Agent's Memory May Not Survive the Next Model Upgrade
Here's a failure mode almost nobody is testing for: you upgrade the model, keep the memory store, and the agent quietly forgets. A new controlled study (arXiv 2609.05339) shows the same stored history reads differently to a new model — old notes get reinterpreted, mixed embedding versions break retrieval, and there's often no original evidence left to repair from.
The numbers are stark. Memory kept as a fixed-schema knowledge graph survived a model swap essentially untouched — accuracy moved 0.0004. Memory compressed into natural-language notes by the old model swung by 10 to 13 points, asymmetrically, depending on which direction you migrated. The notes weren't knowledge; they were knowledge in the old model's dialect, and the new model half-understands it. RAG with a 50/50 mixed embedding index sat in between. Verbatim history was safest but blows up your context.
Which makes the timing of OKF Agent Memory funny — 78 points on HN, 452 stars. It's the Open Knowledge Format bet: agent memory as plain markdown and YAML living in your git repo, searched by in-memory BM25 in under 300 microseconds, exposed over MCP, pure Go, MIT, zero embedding API costs. Explicitly positioned against Mem0 and Letta.
Put the two together and you get a design rule: memory that lives as readable text under version control is model-portable by construction; memory compressed into a model's own summaries is coupled to that model's mind. Choose your store like you'll change models — because you will, roughly every quarter. https://arxiv.org/abs/2609.05339 and https://github.com/okf-memory/okf-agent-memory — for the memory-as-product angle, see https://clauday.com/article/79b72c72-af35-4272-84d3-c76c62b689e0.
← Back to all articles
The numbers are stark. Memory kept as a fixed-schema knowledge graph survived a model swap essentially untouched — accuracy moved 0.0004. Memory compressed into natural-language notes by the old model swung by 10 to 13 points, asymmetrically, depending on which direction you migrated. The notes weren't knowledge; they were knowledge in the old model's dialect, and the new model half-understands it. RAG with a 50/50 mixed embedding index sat in between. Verbatim history was safest but blows up your context.
Which makes the timing of OKF Agent Memory funny — 78 points on HN, 452 stars. It's the Open Knowledge Format bet: agent memory as plain markdown and YAML living in your git repo, searched by in-memory BM25 in under 300 microseconds, exposed over MCP, pure Go, MIT, zero embedding API costs. Explicitly positioned against Mem0 and Letta.
Put the two together and you get a design rule: memory that lives as readable text under version control is model-portable by construction; memory compressed into a model's own summaries is coupled to that model's mind. Choose your store like you'll change models — because you will, roughly every quarter. https://arxiv.org/abs/2609.05339 and https://github.com/okf-memory/okf-agent-memory — for the memory-as-product angle, see https://clauday.com/article/79b72c72-af35-4272-84d3-c76c62b689e0.
Comments