Memory That Compounds: What "Persistent" Actually Means for an Agent
"Give the agent memory" is one of those phrases that sounds like a feature and is actually an architecture. Everyone agrees agents should remember things across sessions. Almost no one agrees on what that means — and the default implementation, a vector database that everything gets dumped into, produces agents that are confidently wrong about their own past.
This is a field note on how we think about memory for agents that run in production: what the pieces are, what each one is for, and the single principle that governs all of them — an agent's memory is only as useful as what it refuses to remember.
The default that quietly fails
The reflexive design is: embed every message, every tool result, every document, and store the vectors. At recall time, embed the current query and pull the top k nearest neighbors into the context window. This is easy, it demos beautifully, and it degrades in a specific, predictable way.
The failure isn't that retrieval "misses." It's that retrieval succeeds at the wrong thing. Nearest-neighbor search returns what is semantically similar to the query, not what is relevant to the task. Ask an agent that has processed a thousand invoices about "the Acme invoice" and it will happily surface eleven Acme invoices from three different quarters, none flagged as superseded, and splice them into context with equal authority. The model, being agreeable, will reconcile them into a confident answer that is subtly wrong.
The tell: your agent gives a great answer, then gives a contradictory great answer to the same question a day later, and both are traceable to "something it remembered." That's not a retrieval bug you can tune away. It's a sign that storage and recall were never separated from judgment about what mattered.
Four kinds of memory, four different jobs
It helps to stop saying "memory" and name the pieces. Human cognition is a loose analogy, but a useful one, because each type answers a different question.
Working memory is the current context window: the task, the recent turns, the tool outputs from this run. It is bounded, it is expensive, and it is the only memory the model can actually reason over. Everything else exists to decide what gets promoted into it.
Episodic memory is "what happened." Specific events, tied to time and outcome: on June 3rd we ran the reconciliation for Acme and it failed because the currency field was null; the human corrected it this way. Episodic memory is how an agent learns from its own history instead of repeating it.
Semantic memory is "what is true." Distilled facts that outlived the event that produced them: Acme bills in EUR; this customer's fiscal year starts in April; the approval threshold is 5,000. Semantic memory is what episodic memory becomes once a pattern has been seen enough times to trust.
Procedural memory is "how we do this here." The learned workflow — the order of steps, the checks that matter, the tool that actually works for a given task. It's the difference between an agent that re-derives the process every run and one that has a way of doing things.
The mistake is storing all four as undifferentiated embeddings. They have different write rules, different retrieval rules, and very different lifespans. A null-currency incident is an episode that should decay. "Acme bills in EUR" is a fact that should persist and be retrieved by entity, not by vector similarity to the current sentence.
Writing is a decision, not a side effect
The most important design choice in an agent's memory is not how it retrieves. It's what it writes down, and that should be a deliberate step, not an automatic log of everything that streamed through the context.
A memory-write worth making usually clears three bars:
- It's durable. It will still be true, or still be instructive, next week. A transient tool error is not; the lesson from a recurring tool error is.
- It's decision-relevant. It changes what a future run would do. If remembering it wouldn't alter any future action, it's noise wearing a fact's clothes.
- It's attributable. It carries where it came from and when. A memory with no provenance can't be trusted, updated, or retired — it can only accumulate.
In practice this means an agent should reflect before it writes: at the end of a run, summarize what changed, promote the durable facts to semantic memory, keep the specific episode with its outcome, and drop the rest. This is the same generate-then-verify discipline that governs a good agent loop, pointed inward.
Retrieval by structure, not just by similarity
Once memory is typed, retrieval stops being one vector search and becomes a small set of targeted ones. Facts about an entity are fetched by that entity. Recent episodes are fetched by recency and relevance, with the outcome attached so the model can weight a success differently from a failure. Procedures are fetched by task type. Vector similarity is one signal among several — useful for the fuzzy "have we seen anything like this before," dangerous as the only lens.
The other half of retrieval is budgeting. Working memory is finite and attention is not free; a context window stuffed with fourteen marginally relevant memories reasons worse than one holding the three that matter. Recall should be ranked and capped, and the cap should be small. More retrieved memory past a point makes agents worse, not better — the signal gets buried and the model starts pattern-matching on the loudest neighbor instead of the most relevant one.
Forgetting is a feature
Here is the part that feels wrong and is essential: a good memory system spends as much design effort on removal as on storage.
Facts get superseded — the approval threshold changed, the customer switched currencies, the API you learned to call was deprecated. If memory only appends, every one of those updates becomes a contradiction the model has to resolve at read time, usually by picking one at random. So memories need to be updatable (the new fact replaces the old, with the old one retired, not deleted silently) and decayable (episodes lose weight over time unless reinforced). "Remembers everything" is not the goal. An agent that cannot forget a fact that stopped being true is worse than one with no memory at all, because it is wrong with history behind it.
This is also where trust and safety live. Memory is an injection surface: content an agent reads today can be written into memory and re-executed as instruction tomorrow. Typed, attributed, expirable memory is what lets you audit that — to answer "why did the agent believe this?" with a specific provenance chain rather than a shrug at a vector store.
Why this compounds
The reason to do the harder version is that memory is the mechanism by which an agent gets better instead of just staying the same. A stateless agent performs the same on run 10,000 as on run 1 — it re-derives everything, repeats every avoidable mistake, and never accumulates the specific, hard-won knowledge of your domain that a good human teammate builds over months. An agent with typed, curated, decaying memory does the opposite: each run leaves the next one a little more competent, because the durable lessons persist and the noise doesn't.
At CatalEx this is what we mean when we say agents are self-evolving and carry context across sessions. It isn't a bigger vector store. It's the discipline of deciding, every run, what was worth remembering — and what the agent is better off forgetting.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.