How to Evaluate AI Systems in Production

Shipping an AI feature without an eval harness is like deploying a payments API with only manual clicks in staging. The system will look fine until traffic finds the weird cases — and fluent wrong answers do not trip your HTTP error rate. Production evaluation is how you notice quality cliffs when prompts, models, indexes, or tools change.

What you are measuring#

Separate three layers. Mixing them produces metrics nobody trusts.

1. Component checks — retrieval recall, schema validity, tool-arg parse rate
2. Trajectory checks — did the agent thrash, skip a required tool, or exceed budget?
3. Outcome checks — task success, faithfulness to sources, policy adherence, user repair rate

Uptime and p99 latency still matter. They do not substitute for outcome quality.

Start with a golden set you own#

Fifty to two hundred labeled examples per job class beat a vendor demo every time. Each item needs: input, expected constraints (citations, fields, refusal), and where possible expected evidence IDs — not only a preferred paragraph of prose.

Version the set in git. When product requirements change, update labels deliberately; do not "fix" failing cases out of existence the night before a launch.

from dataclasses import dataclass

@dataclass
class Case:
    id: str
    input: str
    must_cite: list[str] | None
    expect_status: str  # ok | cannot_answer | need_clarification

def score_case(case: Case, output: dict) -> dict:
    checks = {
        "status_ok": output.get("status") == case.expect_status,
        "valid_schema": validate(output),
    }
    if case.must_cite:
        cites = set(output.get("citations") or [])
        checks["citations"] = set(case.must_cite).issubset(cites)
    return checks

LLM-as-judge without fooling yourself#

Judges are useful for faithfulness and rubric scoring at scale. They are also biased toward longer, prettier answers. Mitigations that actually help:

• Give the judge the evidence, not only the answer
• Ask for structured findings (supported / unsupported spans), not a single vibe score
• Spot-check judge disagreements with humans every week
• Freeze judge model version like any other dependency

If the judge and the candidate share the same model family and prompt style, treat scores as suggestive, not gospel.

Online evaluation is not optional#

Offline sets miss prompt injection in live tickets, seasonal corpus drift, and tool outages. Sample production traces: regenerations, escalations, thumbs-down, and a random slice of "success" traffic. Score faithfulness and tool success on that sample. Alert on rate changes the same way you alert on error budgets.

Observability that supports evals#

Log enough to rebuild a trajectory: prompt regions or hashes, retrieved chunk IDs, tool args/results, model/version, token counts, stop reason. Redact secrets. Without traces, offline evals cannot explain online failures. Dashboards should show quality next to latency — faithfulness sample rate, schema fail rate, tool retry rate, cost per successful task — with deep links from failed cases to traces.

Regression gates#

Thresholds are product-specific. The structure is not.

Who owns the harness#

Platform teams should own the runner, reporting, and judge versioning. Domain teams own labels and pass/fail thresholds for their job class. If platform owns labels, evals drift from product reality. If every domain reinvents the runner, you cannot compare releases. Split ownership the same way you split CI: shared engine, local tests.

Eval cadence and release ritual#

The release artifact should be boring: a diff table of case ids that flipped, component metrics, cost per success, and a human sign-off when privilege or refusal behavior changed. If leadership asks for a single headline number, pick task success rate on the golden set plus cost per successful task.

Summary#

Production AI evaluation is a stack: owned golden sets, trajectory and outcome metrics, careful judges, online sampling, and release gates tied to traces. Treat quality like reliability — budgeted, monitored, and blocking — or your classic SLOs will keep saying green while users learn not to trust the feature.