DM Explains · No. 019 · Agents
Agent memory: why more recall can make an AI agent worse at acting
Agents that remember past sessions feel smarter, but the machinery behind that memory trades accuracy against cost, and sometimes against the very task it's meant to help with.
· 14 min read
Read me first
Agent memory genuinely does let an agent carry facts across separate sessions, but giving it more memory to retrieve can make it worse at the task in front of it, because recall and action pull attention in different directions.
Papa says "aisle seat, no early flights" → 1 extracted as a preference fact → 2 added without overwriting the earlier hotel note → 3 linked to Papa's trip thread across sessions → 4 semantic, keyword and entity signals fuse the top candidates → 5 strongest facts assembled into about 6,787 tokens of context → 6 agent retrieves fewer, sharper memories to protect task attention → 7 stale hotel note pruned locally after the plan changes → booked flight honors the aisle-seat rule weeks later
An agent that helped plan a family trip last week is asked a follow-up question today, in a brand-new conversation, and has no idea the trip ever existed unless something was built to carry that fact forward. Teams fix this by bolting on a vector database and calling it memory, then watch the agent either forget a stated preference or, worse, drown in retrieved history and stall on the task actually in front of it.
Everyday picture. Dadi keeps a small notebook by the phone noting what each grandchild likes and who called about what, rather than replaying every past phone call from memory. When Meera asks Amma what Papa wants for his birthday, Amma checks her own memory, the fridge shopping list and what Dadi mentioned, combining the clues into one answer. Bruno, meanwhile, remembers every treat-giver in the house, so when Arjun asks him to fetch one specific toy, he gets distracted recalling all of them instead of focusing on the toy in front of him.
Where the picture breaks. Dadi's notebook doesn't automatically link entries or re-rank them by relevance, Amma resolves contradictions with judgement that fused-scoring systems still lack, and Bruno's distraction is instinct, not a tunable retrieval-depth setting an engineer can dial down.
Why it exists now
For years, memory quality for agents was mostly self-reported on ad hoc, non-reproducible tasks. That changed with standardised benchmarks: LoCoMo, with 1,540 questions across single-hop, multi-hop, open-domain and temporal recall categories; LongMemEval, with 500 questions across six scenario types; and BEAM, which tests memory at 1 million and 10 million token scales across ten categories, built specifically so it cannot be solved just by expanding the context window.
That matters because a bigger context window and persistent memory solve different problems. A context window lets a model see more tokens in one call; it does not persist anything across sessions or decide which facts are worth keeping. Memory systems do that job, and in 2026 that job became measurable: Mem0 published a new algorithm scoring 92.5 on LoCoMo and 94.4 on LongMemEval at between roughly 6,787 and 6,956 tokens per query depending on the benchmark, with its biggest single gains being +29.6 points on temporal reasoning and +23.1 points on multi-hop reasoning over its prior version (State of AI Agent Memory 2026 — Mem0, 2026). A competing system, Agent Zero Memory, reported 95.60% on LongMemEval and 93.60% on LoCoMo, ahead of Mem0's 92.50% on that benchmark (Agent Zero Memory — arXiv, 2026). As of October 2026, agent memory infrastructure spans 21 frameworks, 20 vector stores and three hosting models.
How it works
Illustrative: Papa says "aisle seat only" → 1 extracted as one new memory → 2 conflicts with an earlier "any seat is fine" → 3 resolved locally and linked to his profile, not rebuilt → weeks later agent is asked to rebook → 4 three scorers retrieve 50 candidate memories → 5 budget allows only 6,787 tokens, so most are dropped → 6 strongest preferences assembled first → agent books the aisle seat, skipping the early flight
Extraction
Pull out the fact, not the whole conversation
What it is. An LLM call reads new conversation turns or agent actions and pulls out atomic facts, rather than keeping the raw transcript.
Why it exists. Without extraction, every call would need to re-process a full transcript, which is slow and expensive at scale.
Where it breaks. Extraction treats agent-generated confirmations and recommendations with the same weight as things the user actually said, so an agent's own tentative suggestion can be stored as if it were a confirmed user fact, and that error carries forward into every later retrieval.
How that's handled. Benchmarks like LoCoMo, with its 1,540 questions, are built partly to catch this: single-hop and multi-hop categories expose whether an extracted fact actually traces back to the user.
In technical terms
In technical terms: an LLM call scans conversation turns or agent actions and extracts discrete facts, treating agent confirmations and recommendations with the same weight as user-stated facts.
ADD-only storage
Accumulate first, reconcile later
What it is. Newer systems skip the step that updates or deletes old memories and just keep adding new ones.
Why it exists. Mem0's redesigned pipeline stores each extracted fact with a single LLM call and no UPDATE or DELETE operation, which cuts latency and token cost compared with running a reconciliation call every time (mem0ai/mem0 README — GitHub, 2026).
Where it breaks. Papa's old hotel preference and his new one both sit in storage; nothing forces resolution until a query happens to surface both at once.
How that's handled. Conflict-resolution gets pushed to read time, inside fused retrieval, rather than resolved on write.
In technical terms
In technical terms: Mem0's newer approach extracts facts with a single-pass ADD-only pipeline, one LLM call per extraction, no UPDATE or DELETE operation, so memories accumulate rather than being overwritten.
Entity linking
Same name, same thread, across sessions
What it is. Mentions of the same person, place or thing across separate stored memories get connected to each other.
Why it exists. Entities extracted from stored memories are embedded and linked to other mentions of the same entity, which is what makes a multi-hop question like "what hotel did we pick for the trip we discussed last month" answerable at all.
Where it breaks. Linking depends on entities being recognised consistently; a hotel mentioned by nickname in one session and by full name in another can silently fail to link.
How that's handled. Multi-hop and temporal categories inside LoCoMo and LongMemEval specifically stress-test this, and this is where Mem0 reported its largest single jump, +23.1 points on multi-hop reasoning.
In technical terms
In technical terms: entities extracted from stored memories are embedded and linked to other mentions of the same entity across the memory store, improving downstream multi-hop retrieval (State of AI Agent Memory 2026 — Mem0, 2026).
Fused retrieval
Three signals vote, one ranking wins
What it is. At query time, semantic similarity, keyword matching and entity matching run in parallel and get combined into one ranked list.
Why it exists. Semantic embeddings catch paraphrased meaning, BM25 keyword matching catches exact terms, and entity matching catches named things; fusing all three outperforms relying on any single signal, because different queries favour different signals.
Where it breaks. Fusion is a ranking trick, not judgement: it cannot decide which of two contradictory stored facts is current, only which is more retrievable right now.
How that's handled. Nothing fully handles it yet; it's the reason ADD-only storage pushes conflicts downstream rather than solving them.
In technical terms
In technical terms: three scoring passes, dense vector similarity, BM25 keyword matching, and entity matching, run in parallel and their scores are fused into a single ranking before results reach the model's context (mem0ai/mem0 README — GitHub, 2026).
Context assembly
This is where the tokens actually get spent
What it is. Retrieved memory snippets get inserted into the model's live context window alongside the current task before it responds.
Why it exists. This is the step where the recurring cost of having memory at all is actually spent: Mem0's published benchmarks report roughly 6,787 to 6,956 tokens per query doing this assembly.
Where it breaks. Every retrieved snippet is a token cost and a potential distraction, which is the step implicated in the finding that more retrieved entries can hurt an agent's decisions.
How that's handled. DolphinBench, a 600-task benchmark built on real simulated actions rather than question-answering, requires every test to be verified with and without relevant history, specifically to expose this cost (DolphinBench — Mem0, 2026).
In technical terms
In technical terms: Mem0's published benchmark results report average tokens per query of 6,956 on LoCoMo, 6,787 on LongMemEval, 6,719 on BEAM at the 1M scale and 6,914 on BEAM at the 10M scale.
Retrieve more?
Helps answers, hurts actions
What it is. Pulling in more memory entries at retrieval time improves question-answering accuracy but can actively worsen an agent's ability to decide what to do next.
Why it exists. A 2026 holistic evaluation of memory substrates found that as retrieval depth increases, model attention shifts toward the retrieved memory block and away from the live task observation the agent needs to act on (Harness the Memory — arXiv, 2026).
Where it breaks. This is a genuine trade-off, not a bug waiting to be patched: there is no retrieval depth that is simultaneously best for recall-style questions and for in-the-moment action.
How that's handled. Teams tune retrieval depth separately for question-answering agents and acting agents, rather than assuming one setting serves both.
In technical terms
In technical terms: increasing the number of retrieved memory entries improves dialogue question-answering scores while measurably reducing performance on agentic decision tasks, because attention shifts away from the current task observation toward the retrieved block.
Lifecycle maintenance
Tidy locally, not globally
What it is. Memories need pruning and reorganising over time as the store grows.
Why it exists. A systematic study of agent memory data management found that highly structured memory systems incur far higher index-construction time and query latency without proportional accuracy gains, while localised maintenance stays cost-efficient as memory grows (Are We Ready For An Agent-Native Memory System? — arXiv, 2026).
Where it breaks. At very large scale, even well-maintained systems degrade sharply: the same class of system that scores 64.1 on BEAM at the 1M-token tier falls to 48.6 at the 10M-token tier, showing maintenance alone doesn't solve long-horizon temporal reasoning.
How that's handled. The honest answer, so far, is that it isn't fully handled; this remains an open problem at scale.
In technical terms
In technical terms: localised maintenance of memory indices is more cost-efficient than global reorganisation as store size grows, though no current approach closes the gap at BEAM's 10-million-token scale.
Compared with what came before
| Re-reading full history | Bigger context window | Agent memory | |
|---|---|---|---|
| What persists across sessions | Nothing automatically | Nothing beyond one call | Extracted facts, linked and stored |
| Cost per query | Grows with transcript length | Grows with window size | Roughly 6,787-6,956 tokens on Mem0's benchmarks |
| Handles contradictions | Not addressed | Not addressed | Pushed to retrieval time, not solved on write |
| Degrades at scale | Badly | Badly | Still degrades: BEAM score falls from 64.1 to 48.6 at 10M tokens |
None of these three options eliminates the trade-off between what's remembered and what it costs to retrieve it; memory simply moves the cost from re-reading everything to deciding, imperfectly, what's worth keeping.
Where it breaks
Not a bigger brain, a smaller chosen slice
Agent memory is not solved by retrieving more. A 2026 holistic evaluation found that increasing retrieval depth helps question-answering but hurts agentic decision-making, because attention shifts from the live task to the retrieved block.
Memory also isn't the same thing as retrieval-augmented generation over documents, and conflating the two causes real design mistakes. RAG retrieves static external documents; agent memory stores and updates dynamic, agent-specific facts, like preferences and prior confirmations, that change over time and must be reconciled with new information. Fine-tuning is a third, different thing: it bakes information into model weights permanently and offline, while memory systems update instantly at inference time without retraining.
The honest failure modes are: contested extraction (an agent's own tentative suggestion stored as fact), unresolved contradictions sitting side by side in storage, entity-linking misses on renamed or nicknamed things, token cost that scales with every retrieved snippet, and degradation at long-horizon scale that current maintenance strategies don't fully fix.
Numbers to carry
| Figure | Value |
|---|---|
| LoCoMo question count | 1,540 questions, four categories |
| LongMemEval question count | 500 questions, six categories |
| BEAM scale tiers | 1 million and 10 million tokens, ten categories |
| Mem0 LoCoMo / LongMemEval scores (2026) | 92.5 / 94.4 |
| Mem0 gains over prior algorithm | +29.6 points temporal reasoning, +23.1 points multi-hop |
| Mem0 tokens per query | 6,787-6,956, as of October 2026 |
| Mem0 old → new algorithm (GitHub) | LoCoMo 71.4 → 91.6; LongMemEval 67.8 → 93.4 |
| BEAM score, 1M vs 10M tokens | 64.1 vs 48.6 |
| Agent Zero Memory scores | 95.60% LongMemEval, 93.60% LoCoMo |
| Mastra / Zep LongMemEval scores | 94.87% / 71.20% |
| DolphinBench task count | 600 tasks |
| Infrastructure, as of October 2026 | 21 frameworks, 20 vector stores, 3 hosting models |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "A bigger context window is memory." | A context window holds more tokens in one call; it doesn't persist anything across sessions, which is why BEAM was built so it can't be solved by just expanding the window. |
| "More retrieval is always better." | A 2026 evaluation found more retrieved entries help question-answering but hurt agentic decision-making by shifting attention away from the live task. |
| "Memory systems resolve contradictions as they're stored." | Mem0's newer design is ADD-only, no UPDATE or DELETE, so contradictions sit in storage until retrieval surfaces them. |
| "Highly structured memory is always worth the overhead." | A systematic study found structured systems carry much higher index-construction and query-latency costs without proportional accuracy gains over lightweight stores. |
| "Benchmarks prove memory is a solved problem." | BEAM scores drop from 64.1 at 1 million tokens to 48.6 at 10 million tokens, showing long-horizon temporal reasoning is still open. |
Worked example
- Papa tells the trip-planning agent, across several sessions, that he wants an aisle seat and no early flights; the agent's extraction step pulls these out as atomic facts rather than keeping the whole transcript.
- The agent confirms a hotel booking in a later session; this confirmation is extracted and stored with the same weight as Papa's own stated preferences, which is exactly the point where a tentative suggestion could get mistaken for a confirmed fact.
- The storage layer, running ADD-only, adds this new hotel fact without checking whether it conflicts with anything stored earlier; nothing gets deleted.
- Weeks later, Papa changes his mind about the hotel. The entity-linking step connects this new mention to the same "hotel" entity from before, so both the old and new preference now sit linked in storage.
- When Papa next asks "what did we decide about the hotel?", fused retrieval scores semantic, keyword and entity matches and returns both the old and new hotel facts ranked near the top, because fusion ranks retrievability, not truth.
- Context assembly inserts both snippets into the model's context before it answers, spending real tokens presenting conflicting history before the live question is even addressed, in roughly the same range as the per-query token cost Mem0 reports on its benchmarks.
- The agent answers the question reasonably well, because question-answering benefits from more retrieved context, but if this same retrieval depth were feeding a live booking action rather than a question, the same abundance of history could compete with the current task for the model's attention.
Should you use it?
Agent memory earns its cost when an agent must recall preferences, prior decisions or facts across sessions that genuinely span time, and when temporal reasoning (what changed, and when) matters to the task. It is not a fit for single-turn, stateless tasks where nothing needs to persist, nor for retrieving systematically from large structured datasets that a plain database handles better, nor for agentic decision tasks where retrieving many entries has been shown to compete with the live observation for the model's attention. The cheapest first experiment is running a small slice of your own agent's sessions through LoCoMo-style single-hop and temporal questions before adding any production memory store at all, to see whether extraction alone, without fused retrieval, already captures what you need.
Say it in the room
"Use agent memory when facts need to persist and change across sessions; skip it, or tune retrieval depth down hard, when the task is acting now rather than answering a question about the past."
Lines to carry
- "Memory is not a bigger window, it's a smaller, chosen slice."
- "Storing facts is cheap; knowing which one is still true is not."
- "More retrieved does not mean more remembered correctly."
- "Fix memory locally and often, not globally and rarely."
- "Retrieval depth that helps an answer can hurt an action."
FAQ
Is agent memory just RAG with a different name?
No. Retrieval-augmented generation retrieves static external documents that don't change based on what the agent did; agent memory stores dynamic, agent-specific facts, like a stated preference or a prior confirmation, that update over time and sometimes contradict each other.
Why not just use a model with a huge context window instead?
Because a context window only holds more tokens inside one call; it doesn't persist anything once that call ends, and it doesn't decide which facts matter. BEAM was built specifically because this problem can't be solved by making the window bigger.
Does more memory always make an agent better?
Not for every task. A 2026 evaluation found that retrieving more memory entries improves question-answering accuracy but can hurt an agent's live decision-making, because attention shifts toward the retrieved block instead of the current task.
How do contradictory memories get resolved?
In systems like Mem0's newer design, they often don't get resolved at write time at all; facts are only added, never updated or deleted, so contradictions sit in storage until a retrieval happens to surface both.
Does bigger, more structured memory always perform better?
No. A systematic study found highly structured memory systems carry far higher index-construction and query-latency costs than lightweight stores, without a proportional gain in accuracy.
Glossary
- Agent — a software system built on a large language model that takes multi-step actions, such as calling tools, rather than only answering a single question.
- Memory (agent memory) — the subsystem that stores facts, preferences and past actions so an agent can use them in later, separate interactions.
- LoCoMo — a benchmark of 1,540 questions testing recall from long multi-session conversations.
- LongMemEval — a 500-question benchmark testing six memory scenarios including knowledge updates and multi-session recall.
- BEAM — a benchmark testing memory systems at 1 million and 10 million token scales across ten task categories.
- DolphinBench — a 2026 benchmark evaluating agent memory via real simulated actions and tool calls across 600 tasks.
- Context window — the maximum amount of text a language model can consider at once when generating a response.
- Retrieval-augmented generation (RAG) — supplementing a model's answer with text pulled from an external store of documents at query time.
- Vector store — a database that stores data as numerical embeddings so similar information can be found by closeness in that numerical space.
- Entity linking — recognising that mentions of a person, place or thing in different texts refer to the same entity, and connecting them.
- BM25 — a classic keyword-matching ranking algorithm based on term frequency rather than meaning.
- Temporal reasoning — tracking how facts change over time and answering questions about when something was true.
- Multi-hop reasoning — answering a question that requires combining information from more than one separate fact.
- p50 latency — the median response time: half of all requests complete faster than this value and half slower.
- Token — a chunk of text, roughly a word or part of a word, that language models process and that usage is measured by.
Sources
- State of AI Agent Memory 2026: Benchmarks & Trends Report — Mem0, 2026
- README.md — GitHub (mem0ai/mem0), 2026
- Introducing DolphinBench: Mapping the Pareto Frontier of Agent Memory — Mem0, 2026
- Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents — arXiv, 2026
- Are We Ready For An Agent-Native Memory System? — arXiv, 2026
- Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents — arXiv, 2026