Why AI Memory Needs Confidence Scores, Not Just Facts

An extractor writes customer.loyalty_tier = platinum because the user said "we're basically VIP." No source quote. No confidence. Three weeks later the billing agent grants platinum-only concessions. Nobody can tell whether that field was confirmed in CRM, inferred from tone, or hallucinated by a previous model version. The store held a fact-shaped string. It did not hold uncertainty — and production systems need both.

Facts without weights become false certainty#

LLM-based memory writers are approximate. They compress messy dialogue into neat key-value pairs. That compression drops hedging ("I think", "for now", "my spouse handles that"). If your schema only has key and value, you silently upgrade guesses to doctrine.

Confidence scores — and the provenance that justifies them — let downstream components behave differently:

• High confidence + CRM source → inject into policy-sensitive prompts
• Medium confidence → use as soft hint, ask to confirm before irreversible actions
• Low confidence → exclude from tools that spend money or change access

Without that, every memory item competes equally in retrieval and in the model's attention.

What to store beside the value#

Confidence is not a substitute for access control. A high-confidence wrong address is still wrong — but at least you can require confirmation before shipping.

Where confidence comes from#

Not all confidence is model output. Production systems mix signals:

• Explicit user action — form submit, "remember this" → high band, fixed score (e.g. 0.95)
• System of record sync — CRM webhook → high, with source_ref to external ID
• Extractor logprob or self-report — useful but miscalibrated; validate against overrides
• Rule-based — regex match on structured input → deterministic score
• Behavioral inference — repeated pattern → medium, never high without opt-in

Document which signal produced each score. "confidence=0.87" without provenance is not actionable in an incident review.

Use confidence at read time, not only at write time#

from dataclasses import dataclass
from typing import Iterable

@dataclass
class Mem:
    key: str
    value: str
    confidence: float
    source_type: str

def select_for_prompt(
    items: Iterable[Mem],
    *,
    min_conf: float,
    max_chars: int,
) -> list[Mem]:
    ranked = sorted(
        (m for m in items if m.confidence >= min_conf),
        key=lambda m: m.confidence,
        reverse=True,
    )
    out: list[Mem] = []
    used = 0
    for m in ranked:
        line = f"{m.key}={m.value} ({m.source_type}, p={m.confidence:.2f})"
        if used + len(line) > max_chars:
            break
        out.append(m)
        used += len(line)
    return out

def allow_privileged_action(mem: Mem, action: str) -> bool:
    if action in {"issue_refund", "change_entitlement"}:
        return mem.source_type in {"crm", "user_explicit"} and mem.confidence >= 0.85
    return mem.confidence >= 0.55

Notice privileged actions demand both source class and score. A clever extractor alone should not unlock money paths.

Task-dependent thresholds#

The same memory item may need different confidence bars depending on what the agent is doing:

Hard-coding one global min_conf=0.6 for all reads leaves gaps. Encode thresholds per key prefix or per tool in policy config, not in prompt prose.

Calibration and conflict#

When two memories share a key:

1. Prefer higher confidence if sources are equal class.
2. Prefer user_explicit / system-of-record over inferred_behavior even at slightly lower score.
3. If both medium and disagree, do not silently overwrite — surface confirm or keep both with a conflict flag excluded from default prompt injection.

Track write confidence histograms and downstream override rates (how often humans correct a field). If overrides cluster on high-confidence writes, your extractor is miscalibrated.

Recalibration after model bumps#

When you upgrade the memory extractor model, replay a golden set of dialogues and compare score distributions. If the new model assigns 0.9 to cases the old model scored 0.6, your gates are stale. Version model_version on every row so you can quarantine or re-score legacy items without a full wipe.

Production lesson: the fluent wrong address#

A logistics agent stored shipping addresses from chat extracts at fixed confidence 0.8 — no source differentiation. CRM-synced addresses and casual "ship it to my mom's place in Portland" mentions scored identically. Retrieval returned both; the model preferred the fluent chat extract. The package went to the wrong state. Fix: CRM writes at 0.95 with source_type=crm; chat extracts below 0.7 excluded from shipping tools until user confirms. Override rate on address fields dropped sharply once gates matched actual trust.

Design practices#

1. Schema every memory item with confidence + provenance — no bare strings in the durable store.
2. Gate tools by source class, not by retrieval rank alone.
3. Show scores in internal debug UI so on-call engineers can see why the agent believed something.
4. Decay low-confidence items faster than CRM-backed fields.
5. Version extractors so you can re-score or quarantine after a bad model bump.

Summary#

Memory that stores only facts trains the rest of the stack to treat approximate extracts as ground truth. Attach confidence and provenance to every durable item, use those fields when assembling context and authorizing actions, and calibrate against human corrections. Uncertainty-aware memory is not academic caution — it is how you stop a polite guess from becoming an entitlement change.