Caching vs Memory in AI Systems: Speed isn’t Continuity

Someone proposes "semantic memory" and slides a Redis TTL next to an embedding index. Latency drops on repeated prompts. Product calls it memory. Three weeks later a user changes their refund preference and the system keeps serving the cached answer for nineteen minutes — or worse, treats a cached completion as a durable preference for months because nobody distinguished performance state from meaning state.

Caching optimizes work you already decided#

A cache stores the result of expensive work so you can skip recomputing it: model completions, embedding vectors, retrieved chunk IDs, tool payloads, rendered prompts. Keys are usually content hashes or request fingerprints. Values are disposable. Wrong cache entries are fixed by invalidation or short TTLs. The product meaning of the answer does not live in the cache; the cache only accelerates getting back to a previous computation.

Good caches are boring: hit rate, p99, stampede control, explicit invalidation when upstream docs change.

Common cache layers in AI stacks#

• Completion cache — same prompt + model → same response (short TTL)
• Embedding cache — hash of text → vector (longer TTL, invalidate on model version change)
• Retrieval cache — query fingerprint → chunk IDs (invalidate on corpus version bump)
• Semantic cache — embedding similarity on prompt → prior answer (highest risk of false continuity)

Each layer is performance infrastructure. None of them answer "what do we believe about this user tomorrow?"

Memory preserves claims you still believe#

Memory stores assertions about the world of your product: this user prefers SMS, this ticket already confirmed identity, this agent run decided the next step is human review. Keys are identity-scoped. Values have provenance and overwrite rules. Wrong memory entries are product bugs and sometimes compliance incidents. TTL alone is not enough — you need correction and forget semantics.

Cache vs memory at a glance#

Semantic caches that return prior answers for similar prompts are still caches. They are not memory just because the similarity function uses embeddings.

When conflation hurts#

Stale personalization. You cache a completion that mentioned the old plan tier. Fine for nineteen minutes if labeled as cache. Catastrophic if that string is later written into a profile store as gospel.

False continuity. Hitting a semantic cache for "what did we decide?" skips the memory read. The user hears a confident replay of an outdated decision.

Security. Cached completions may include another tenant's phrasing if keys omit tenant ID. Memory bugs leak PII; cache bugs often look like cross-tenant bleed under bad key design — both bad, different mitigations.

Production incident: cache promoted to profile#

A B2B assistant used semantic caching aggressively. On cache hit, the orchestrator skipped both the LLM call and the memory read — the cached answer already contained personalized phrasing. When a user's role changed from viewer to admin, cached responses still described viewer-level permissions for up to thirty minutes. Worse: a background job scraped "preferred greeting" strings from cache hits into a profile table for "personalization analytics." Those strings were performance artifacts, not user preferences. The fix required separating cache keys from memory keys and banning any ETL from cache to profile without a typed write path.

Semantic cache vs memory: the similarity trap#

Semantic caches match on embedding distance. Memory retrieval can also use embeddings. The difference is intent and invalidation:

• Semantic cache: "Have we answered something like this recently?" Invalidate on TTL or corpus epoch.
• Memory read: "What do we know about this identity for this task?" Invalidate on memory write, forget API, or consolidation.

Using the same Redis cluster for both without namespace separation is how teams accidentally serve another user's cached answer when similarity thresholds are loose. Tenant ID in every cache key is non-negotiable. For memory, identity scoping is the whole point.

Keep the pipelines separate#

import hashlib
import json
from typing import Any, Optional

def cache_key(tenant: str, prompt_fingerprint: str, model: str) -> str:
    raw = f"{tenant}|{model}|{prompt_fingerprint}"
    return "cmpl:" + hashlib.sha256(raw.encode()).hexdigest()[:32]

def get_cached_completion(store, key: str) -> Optional[str]:
    return store.get(key)  # OK to miss; never treat as user truth

def put_cached_completion(store, key: str, text: str, ttl_s: int = 900) -> None:
    store.setex(key, ttl_s, text)

def read_memory(profile_db, user_id: str, field: str) -> Optional[dict[str, Any]]:
    row = profile_db.get(user_id, field)
    return row  # includes value, source, updated_at, confidence

def write_memory(profile_db, user_id: str, field: str, value: str, source: str) -> None:
    profile_db.upsert(
        user_id,
        field,
        {"value": value, "source": source, "updated_at": "server_now"},
    )
    # Invalidate related completion caches — memory change means answers may change
    profile_db.touch_cache_epoch(user_id)

Pattern: memory writes bump a per-user cache epoch so semantic/completion caches cannot outlive a preference change.

Invalidation direction matters#

Memory → cache invalidation is correct. Cache → memory promotion is almost always wrong.

When a user updates a preference in memory, bump their cache epoch and purge retrieval caches keyed to their identity. Do not write cache hits back into memory "for efficiency." If you need to log what the model said, store it as an append-only audit event with source=completion, not as a profile field.

Deploy invalidation is separate: flushing all completion caches on deploy is fine. Flushing memory on deploy is data loss.

Design rules#

1. Caches may approximate; memory must commit. Do not promote cache hits into profile fields without a typed write path.
2. Key caches by tenant + model + input fingerprint. Key memory by identity + field.
3. On memory write, invalidate related caches. Directionality matters: memory is source of truth.
4. Instrument separately. Cache hit rate is not memory quality. Track memory overwrite rate, conflict rate, forget requests.
5. Document TTLs as performance policy, not as "how long we remember the user."

Summary#

Caching makes repeated AI work cheaper. Memory makes continuity correct. They share infrastructure vocabulary — keys, stores, TTLs — and almost nothing else about product semantics. Draw them as different layers, wire invalidation from memory to cache (not the reverse), and stop calling Redis your memory system unless it actually holds governed, identity-scoped claims.