AI Memory Systems Explained

Teams often say “memory” when they mean four different things: the chat buffer in the prompt, a store of past interaction episodes, a graph of general facts, and a durable profile of user preferences. Collapsing those into one word produces architectures that either forget what mattered yesterday or remember everything forever at ruinous cost. This piece separates the types and shows how each should flow into a budgeted context window.

Memory is not RAG#

RAG retrieves knowledge artifacts — policies, manuals, tickets — that exist whether or not a particular user is speaking. Memory stores state about interactions and entities: what happened in a past session, what a concept means in your domain, what this customer prefers. You can build excellent RAG with almost no personal memory. You can also build an assistant that retrieves the right policy but forgets the customer already chose option B.

If a fact should be true for every user, it usually belongs in a knowledge corpus. If a fact is about this user, thread, or past episode, it belongs in memory. Blurring that line is how PII ends up in shared indexes.

Four types of memory agents use#

Cognitive science gives a useful vocabulary. Production systems rarely ship four separate products, but if you cannot point to where each concern lives, you will under-build one of them.

Short-term memory#

The agent’s working buffer for the live session: recent prompts, responses, and in-flight task steps. Usually cleared or summarized when the session ends.

Failure mode: dumping the entire transcript into every call until cost spikes. Fix: keep recent turns verbatim, summarize older ones.

In systems terms, working memory is the structured cousin of this buffer — plan steps, tool results, loop counters. Persist it as a state document so retries do not restart from zero.

Long-term memory#

Retains facts and preferences across sessions via a profile store, vector index, or knowledge-graph projection.

Examples: “prefers email over phone,” “account is on Plan Enterprise.” Prefer structured records with sources, timestamps, and an explicit overwrite policy. Unreviewed free-form memories accumulate contradictions.

Episodic memory#

Episodic memory recalls past interactions — what happened, when, and what the outcome was — so the agent can adapt instead of treating every session as amnesia with a preference list.

This is how an agent remembers “last Tuesday we reset the VPN profile and it failed because MFA was stale,” not merely “user uses VPN.” Episodes are richer and heavier than profile keys. Retrieve by similarity or filters (time range, outcome, workflow), and summarize before they eat the window.

Semantic memory#

Semantic memory stores general facts and concepts — domain knowledge the agent should reason over, not a single past event.

In enterprise agents this looks like entity graphs (accounts, products, entitlements), glossaries, and curated fact tables. It overlaps with RAG when facts live in documents, but semantic memory prefers structured representations you can query precisely. Use it for “what is true about this domain?” rather than “what did we say in that meeting?”

How it fits together in production#

The cognitive types above map onto a small set of stores in a real agent. Short-term and episodic session state usually sit in Redis; working state in a job document; profile and semantic facts in a durable store; shared knowledge in the RAG index; and a semantic cache beside them for repeat answers. At request time everything competes for space in a budgeted context assembler before it reaches the LLM. Separately, an extractor job proposes new long-term facts — with an accept/reject gate — so the write path stays disciplined.

Write paths: the part most diagrams skip#

Reading memory is easy. Writing it safely is not. Three patterns recur:

1. Explicit writes — user or tool says “remember that…” into a structured record. Highest precision.
2. Extract-on-idle — post-session job proposes candidates; rules or humans accept. Needs dedupe.
3. Event-sourced facts — CRM/billing events project into memory. Best when the system of record already knows the truth.

Avoid letting the chat model silently append whatever it “thinks” it learned. That pollutes profiles and episodic history.

from dataclasses import dataclass
from datetime import datetime

@dataclass
class MemoryRecord:
    subject_id: str
    key: str
    value: str
    source: str
    memory_type: str  # profile | episodic | semantic
    confidence: float
    updated_at: datetime

def upsert_memory(store, rec: MemoryRecord, min_confidence: float = 0.7):
    if rec.confidence < min_confidence:
        return {"status": "rejected", "reason": "low_confidence"}
    existing = store.get(rec.subject_id, rec.key)
    if existing and existing.updated_at > rec.updated_at:
        return {"status": "stale_write_ignored"}
    store.put(rec)
    return {"status": "ok"}

Assembling context without drowning the model#

At request time, pull a budgeted mix: policy, working state, profile facts, optional episodic recalls, semantic facts, RAG chunks, recent turns. Retrieve memory by key, filter, or similarity — never dump years of history wholesale.

Privacy, cache, and debugging#

Profile and episodic memory are personal data — support deletion, export, and correction. Prefer event-sourced facts over model-extracted prose when the system of record already knows the truth. Scope by subject and purpose: a payroll agent should not load travel preferences “because we have them.”

A semantic cache stores reusable answers by near-duplicate intent. That is a performance layer, not semantic memory. Keep separate TTLs and never promote cache hits into durable profile or episodic stores.

When an agent “forgot,” classify the miss: never written, wrong key/type, written but not retrieved, overruled by other context, or dropped by summarization. Instrument accept/reject counts and assembler drops — otherwise every bug looks like “the model is dumb.”

Summary#

AI memory is not one store. Short-term buffers keep the live session coherent; episodic memory recalls what happened and how it ended; semantic memory holds domain facts; long-term profiles keep durable preferences — all distinct from shared RAG knowledge. Get lifetimes, write policies, and invalidation right before optimizing embeddings.