Agent Memory: The Problem Isn't Recall, It's Forgetting

When people worry about agent memory, they almost always worry about recall: can the agent remember what I told it three sessions ago? It’s the intuitive fear, and it’s mostly the wrong one. The 2026 research on long-term agent memory has converged on a more uncomfortable picture, where the hard failures aren’t about remembering at all. They’re about how memory is architected, how it’s retrieved, and — the one nobody designs for — how it’s forgotten. Recall is the easy part. Everything around it is where agents break.

This matters because memory is what separates a clever one-shot tool from an agent you can actually work with over time. Get it wrong and the agent is either amnesiac or, worse, confidently wrong about stale facts. So it’s worth understanding what the benchmarks actually found, because each of the three failures has a different fix.

The context window is not memory

Start with the most common architectural mistake, named bluntly by a 2026 paper, The Missing Memory Hierarchy (arXiv 2603.09023): “The context window of a large language model is not memory. It is L1 cache: a small, fast, expensive resource that the field treats as the entire memory system.” That single reframing explains a lot of bad agent design. Teams stuff everything into the window because it’s the only storage they’ve engineered — and then act surprised when it’s expensive, wasteful, and doesn’t scale.

The paper measured the waste in production: 21.8% structural waste across 857 production sessions and 4.45 million input tokens. A fifth of what’s in the window, on average, doesn’t need to be there. The fix is the one operating systems figured out decades ago — a memory hierarchy with demand paging: keep the working set in the fast, scarce window, keep everything else in cheaper storage, and page the relevant pieces in on demand rather than holding it all resident. The payoff they report is large: up to 93% context reduction in live production (one session went from 5,038KB to 339KB), at a negligible 0.0254% fault rate across 1.4 million simulated evictions. Treat the window as L1 cache instead of as memory, and most of the bloat — and cost — disappears.

This is the systems-level companion to the point we made in context engineering: the window is a scarce resource to be managed, not a bucket to be filled. The memory hierarchy is the architecture that manages it.

Retrieval is lossy, and it misses what matters

Suppose you do build storage outside the window. Now the question is what you pull back in, and here’s the second failure: most memory systems retrieve badly. AMA-Bench (arXiv 2602.22769), a benchmark built on realistic agent trajectories — states, actions, observations, tool outputs, not just chat logs — found that “existing memory systems underperform because they fail to capture causal and objective information and rely heavily on lossy similarity-based retrieval.”

That phrase, lossy similarity-based retrieval, is the heart of it. The standard approach embeds everything and retrieves whatever’s most semantically similar to the current query. But similar is not the same as relevant — the fact you causally need to answer a question is often not the one that looks most like the question. Ask an agent “why is the staging deploy failing?” and similarity retrieval happily surfaces every past message containing “deploy” and “staging” — while the actual cause, a config change three sessions ago that never mentioned either word, ranks nowhere. The causally load-bearing memory and the textually similar memory are different sets, and similarity search optimizes for the wrong one. So the agent retrieves plausible-looking context and misses the load-bearing detail. Even the paper’s purpose-built system, AMA-Agent — which builds a causality graph and uses tool-augmented retrieval rather than raw similarity — reaches only 57.22% accuracy, outperforming the strongest baseline by 11.16%. Read that as a state-of-the-art ceiling: the best memory system tested gets barely more than half the answers right, and it does so precisely by moving beyond similarity to causal structure. Retrieval is far from solved.

The hardest problem is forgetting

The third failure is the one almost no one designs for, and it’s the most dangerous. Memora (arXiv 2604.20006), a benchmark spanning weeks-to-months-long conversations, set out to test not just whether agents remember, but whether they can update — and it introduced a metric for exactly this: FAMA, Forgetting-Aware Memory Accuracy, which penalizes reliance on obsolete or invalidated memory. The result is sobering: “Evaluations of four LLMs and six memory agents reveal frequent reuse of invalid memories and failures to reconcile evolving memories. Memory agents offer marginal improvements.”

Sit with frequent reuse of invalid memories. The user changed jobs; the API got deprecated; the architectural decision was reversed last month — and the agent keeps acting on the old fact, confidently, because it remembered it perfectly. Remembering is easy. Knowing that something you remember is no longer true is hard, and current systems are bad at it. Worse, dedicated memory agents — the systems specifically built to manage this — offer only “marginal improvements” over plain models. Forgetting isn’t a solved sub-problem of memory; it’s an open one, and it’s the failure most likely to make a long-lived agent quietly untrustworthy.

And notice why it’s structurally hard. A naive memory system is append-only: it writes new facts but never retracts old ones, so contradictory information accumulates and the retrieval step has no principled way to know which version is current. Fixing it isn’t just “store a timestamp” — the agent has to recognize that a new observation invalidates an old one (not merely supplements it), which often requires understanding that two differently-worded facts are about the same thing. That’s a reasoning problem wearing a storage problem’s clothes, which is exactly why bolt-on memory layers barely move the FAMA numbers: they make storage and retrieval better while leaving the invalidation judgment to a model that was never asked to make it.

Memory is an engineering discipline, not a model feature

Put the three together and the conclusion is clear: agent memory is not something you get by picking a bigger model or a longer context window. It’s a system you design, with three distinct problems to solve. Architecture: treat the window as L1 cache and build a hierarchy that pages information in on demand. Retrieval: pull what’s causally relevant, not just what’s superficially similar. Invalidation: decide what to forget, and reconcile facts as they change — the part with no good default. A bigger window makes the cache larger; it does nothing for retrieval quality or invalidation. These are engineering problems, and they live in the harness around the model, not in the weights.

That’s also why memory can’t be a generic, bought-off-the-shelf component. What counts as “the working set,” what’s “causally relevant,” and what makes a fact “stale” are all domain-specific judgments. A memory system for a customer-support agent and one for a code agent disagree about all three, and only someone who understands the domain can specify them.

Why this favors Southeast Asia

Here’s the part that matters for this region, and it’s the now-familiar shape. Designing an agent’s memory — its hierarchy, its retrieval logic, its invalidation rules — is pure systems engineering judgment. It costs no GPUs and no frontier model; it costs careful thinking about what the agent needs to keep, fetch, and discard. That’s a laptop discipline, and it’s exactly the kind of durable, capital-light capability we keep arguing the region should build — the same GPU-free leverage as the loop and context work it sits beside.

And the domain-specificity is an advantage, not a burden. There is no off-the-shelf memory system for a Khmer-language legal-document agent, a Cambodian bank’s compliance workflow, or a provincial logistics coordinator — because what’s relevant and what’s stale in those domains can only be specified by people inside them. A team that gets memory right for its own problem builds something a frontier lab has no incentive and no ability to replicate. The model vendors will keep extending the context window and calling it memory. The teams that understand it’s only L1 cache — and engineer the real memory system around it — are the ones whose agents will still be trustworthy a thousand turns from now.