How AI agent memory works.

A language model on its own is stateless. You feed it a prompt, you get back a continuation, and the moment the response is finished the model forgets you ever existed. There is no “previous conversation” living inside the weights.

An agent, by contrast, is the orchestration around the model: a loop that decides what context to pass in next. Memory is the part of that loop that carries information forward. Everything in this essay is a different answer to the same question — what should we put in the prompt this time?

Before we discuss anything fancy, let's acknowledge the simplest kind of memory there is: just keep stuffing the whole transcript back into the prompt every turn. As long as everything fits, the model has perfect recall.

The catch is the ceiling. Every model has a fixed context window. Once your conversation outgrows it, you have to drop something. The naive policy is FIFO: drop the oldest turn first. Watch what happens when you keep typing.

FIFO is the simplest possible policy. It's also the worst when the dropped turn happens to contain the user's name. Real systems do better — summarization, retrieval, hierarchical paging. We'll get to those.

At any given moment your agent is juggling two very different things. On one side: the live conversation — recent turns, pending tool calls, whatever the model is actively reasoning over. On the other: everything it has ever decided was worth keeping.

The handoff between them is where most of the interesting design happens. What gets encoded? When? At what granularity? Drag a message across the divide — or tap encode → on a desktop without drag support — to commit it to long-term memory.

In a real system the agent decides this for itself, usually by prompting another model with a question like “is this turn worth remembering?”. Self-editing memory architectures push further — letting the agent edit its own long-term store on the fly. Section 7 compares those approaches side by side.

An embedding is a vector — a list of numbers — produced by running text through a neural network trained for one purpose: put similar meanings near each other. Real embeddings live in 768 or 1,536 dimensions. Visualizing those is impossible, so we project to two.

The plot below contains a fictional user's memories, clustered by topic. Type a question. The query gets embedded the same way and placed on the plot. The closest points — measured by cosine similarity — are what the agent would retrieve.

The 2-D layout here is a deliberate cartoon. Real embedding models place these vectors in spaces with thousands of dimensions, where many independent “axes of meaning” can coexist without crowding each other. Cosine similarity, though, works the same way regardless of dimension — which is why the trick scales.

Cognitive science has been carving up “memory” for a century. Modern agent frameworks borrow the same vocabulary because the distinctions are actually useful: each kind has a different shape, a different write rule, and a different retrieval strategy.

Click each card to see a toy version of how that store works.

Everything from the previous sections clicks together here. RAG is the loop most production agents run on every turn: encode the user's message, find similar memories, splice them into the prompt, generate, and — critically — decide what to remember from what just happened.

Each stage in the loop is a place where you can spend engineering budget. Better embeddings, smarter retrieval, prompt compression, stricter governance — they're all the same pipeline, just with more care taken at one step.

Production retrieval is a small pipeline running on every turn: decide if you even need memory, rewrite the query, run several retrievers in parallel, merge their rankings, rerank, apply permission and temporal filters, then pack what survives into the prompt. Skipping any one of these stages is usually how things break.

If retrieval is warranted, you almost never want to pick a single retriever. Dense embeddings are great for paraphrase and intent; lexical (BM25) wins on exact IDs and rare tokens; a knowledge graph contributes structural answers. Run all three and fuse the rankings with RRF.

[memory · top-3]
  1. [current, score 0.048] User lives in Highland Park, LA.
  2. [current, score 0.033] User goes hiking in the canyons on Saturdays.
  3. [current, score 0.032] User prefers concise answers with code.

Same query, different stages — same outcome rarely. The interesting bugs are downstream: a retriever that's technically correct produces a memory that's outdated, or scoped wrong, or fine on its own but contradicts another retrieved row. Filters and rerank exist for those cases.

A single agent only has to answer “what do I remember?”. As soon as you have a team, the question becomes: which agent remembers which fact, on whose behalf, and who else can read it. Memory turns into a graph of stores with scopes, permissions, and propagation rules.

The default should be the opposite of the easy thing: private memory by default, shared memory explicit. A research note the Researcher agent jots down for itself does not need to land in the project channel. Cross-scope writes should be a deliberate act, with a policy attached.

The interesting failure modes aren't “the agent forgot” anymore. They're things like the wrong agent remembered the wrong thing on someone else's behalf. Six patterns show up over and over:

Frameworks like AutoGen, CrewAI, and LangGraph each give you the plumbing — checkpoints, persistence, message-passing primitives. None of them decide your governance for you. Scope, policy, ingestion, and evaluation are still yours.