Context vs Memory in LLM Systems

A support agent "remembers" the customer preferred SMS — until the next session, when that preference is gone because nobody wrote it anywhere durable. Another agent "remembers" every transcript forever by stuffing history into the prompt until latency doubles and the model loses the task instruction. Both teams used the word memory. Only one of them had it. The other had an oversized context window.

Context is assembled state for one call#

Context is the payload you send to the model for this inference: system policy, task brief, retrieved docs, tool results, recent turns, output contract. It lives for the duration of the request (or the session buffer you intentionally keep hot). When the call ends, that payload is gone unless you explicitly persist pieces of it.

Context engineering answers: what enters the window, in what order, under what token budget, and what gets dropped when something must give. It is a packing problem with quality consequences. Bad context produces bad answers even when the underlying stores are fine.

What context is not#

Context is not a database. It is not durable by default. A field in the prompt that only exists because someone pasted it from a ticket does not become memory because the model saw it once. Teams that treat "we included it in the prompt" as equivalent to "we stored it" discover this on the first session boundary — or the first time a different channel handles the same user.

Context also is not retrieval. Retrieved chunks enter context for this call. Whether any of that material should survive the call is a separate write decision. RAG fills the evidence region; memory fills the profile region. Mixing them without labels makes debugging impossible.

Memory is durable state across calls#

Memory is state you deliberately keep so a future call can reconstruct what mattered without replaying the entire history. Profiles, episode summaries, preference records, case notes, consolidation outputs — these survive session boundaries. Memory has a write path, a retention policy, and usually an identity key (user, tenant, ticket, agent run).

Memory answers: what should remain true tomorrow about this entity, who is allowed to see it, and how conflicting updates get resolved. If you cannot point to a store, a schema, and an overwrite rule, you do not have memory. You have vibes.

Context vs memory at a glance#

The boundary is not "short vs long string." A 200-token preference row in a profile store is memory. A 40k-token transcript dumped into every turn is context abuse pretending to be memory.

Production failure: treating the window as the database#

Teams bolt "memory" onto chat by appending history. It works in demos. In production you get:

• Cost that scales with conversation length, not with value retained
• Contradictions nobody resolved (user changed address; both versions still in the window)
• Compliance exposure (full PII transcripts in every gateway log)
• No way to share state across channels (web chat vs ticket vs phone)

The fix is not a bigger context window. It is a write decision: extract durable facts into memory, keep the window lean, retrieve only what this task needs.

A real cross-channel break#

A fintech team ran chat and email through separate agent stacks. Chat stuffed the full thread into context; email started cold each time. The user updated their billing address in chat. Email support, three days later, shipped a statement to the old address because email had no memory read and chat had no durable write. The postmortem blamed "model inconsistency." The architecture had never separated context assembly from memory persistence.

How memory enters context (and when it should not)#

Memory does not bypass context. Selected memory hits still get assembled into the prompt for the current call — usually in a labeled region with a hard token cap. The flow is: read durable store → filter by identity and task → rank by relevance and freshness → inject into context region → model infers.

What you must resist: injecting the entire memory store because retrieval is cheap. A user with two years of episode summaries does not need all of them in every turn. Task-aware memory selection is context engineering applied to durable state.

Run scratchpad vs user memory#

Agent runs often need a third bucket: run scratchpad — state that persists across steps within one workflow but should not become user profile. Intermediate tool outputs, partial plans, retry counters. This is neither full context (it survives multiple model calls within the run) nor user memory (it should not appear in the next session). Naming three layers — context, scratchpad, memory — prevents run debris from polluting profiles.

A minimal split in code#

from dataclasses import dataclass
from typing import Any

@dataclass
class MemoryRecord:
    key: str
    value: str
    source: str
    updated_at: str

def assemble_context(
    *,
    policy: str,
    task: str,
    evidence: list[str],
    memory_hits: list[MemoryRecord],
    recent_turns: list[str],
    max_memory_chars: int = 1200,
) -> str:
    """Context = this call only. Memory hits are already curated."""
    mem_block = []
    used = 0
    for m in memory_hits:
        piece = f"- {m.key}: {m.value} (src={m.source})"
        if used + len(piece) > max_memory_chars:
            break
        mem_block.append(piece)
        used += len(piece)

    sections = [
        f"## Policy\n{policy}",
        f"## Task\n{task}",
        "## Memory\n" + ("\n".join(mem_block) or "(none)"),
        "## Evidence\n" + "\n".join(evidence[:6]),
        "## Recent\n" + "\n".join(recent_turns[-4:]),
    ]
    return "\n\n".join(sections)

def maybe_write_memory(extractor_result: dict[str, Any]) -> MemoryRecord | None:
    """Only durable, explicit fields become memory — not the whole transcript."""
    if not extractor_result.get("should_persist"):
        return None
    return MemoryRecord(
        key=extractor_result["key"],
        value=extractor_result["value"],
        source=extractor_result.get("source", "session"),
        updated_at=extractor_result["updated_at"],
    )

Notice the asymmetry: context assembly is greedy under a budget; memory writes are selective. That selectivity is the whole point.

Token budget trade-offs#

When context and memory compete for the same window, something loses. In practice:

• Policy and task instructions should never be truncated for memory hits. If they are, your budget math is wrong.
• Evidence from RAG usually beats episode summaries for task-specific questions. Memory wins for identity and preference.
• Recent turns matter for conversational coherence but decay fast. After four turns, extract durable facts and drop raw transcript from default assembly.

Teams that cap total context at 8k tokens but leave memory unbounded inside it often discover the model ignores task instructions buried below forty preference rows. Budget memory as its own region with a ceiling — 800–1500 tokens is a sane starting range for most support and assistant workloads.

Design practices that keep the layers honest#

1. Name the regions. Policy, task, evidence, memory, output contract — labeled in the assembled prompt so debugging is not archaeology.
2. Budget memory as a region, not as "whatever fits after RAG." Cap it hard.
3. Require a write schema. Free-form "remember this" blobs rot. Prefer typed keys with provenance.
4. Never use full transcript replay as the long-term store. Summarize or extract; then drop.
5. Separate identity scopes. User memory ≠ tenant knowledge ≠ run scratchpad.

Summary#

Context is the engineered payload for the current model call. Memory is the durable state you choose to keep so future calls do not start from amnesia or from a dump history. Architects who conflate them either starve the model of continuity or drown it in tokens. Draw the boundary in the stack — assembly path on one side, write/retention path on the other — and make both visible in reviews.