Why AI Eval Frameworks Need Versioning Just Like Code

An engineer merges a prompt fix. Faithfulness on the golden set jumps eight points. Leadership celebrates. Two weeks later someone discovers the eval runner also changed — a judge prompt rewrite, unreviewed, merged the same week. The "improvement" was scoring leniency, not product quality. Without versioning on eval logic, datasets, and judge prompts, you cannot trust regression analysis any more than you would trust application metrics if CI randomly swapped test assertions. Eval frameworks need versioning discipline identical to production code.

Eval artifacts are behavior-defining code#

Application code determines what the system does. Eval artifacts determine what you measure — which is behavior-defining in practice because scores gate releases, allocate headcount, and justify model migrations.

Version these artifacts explicitly:

1. Golden datasets — case inputs, constraints, expected evidence IDs
2. Scoring logic — deterministic checks, parsers, rubric code
3. Judge prompts and models — LLM-as-judge instructions and pinned model aliases
4. Runner configuration — temperature, retry policy, parallelization, timeouts
5. Thresholds and baselines — pass rates that block promote

When any of these change without a version bump and changelog entry, historical scores become incomparable. "We regressed" and "we tightened the judge" look identical in a dashboard.

What unversioned evals break#

False regressions. Stricter faithfulness rubric drops pass rate. Team rolls back a good product change.

Hidden improvements. Lenient judge masks real quality decline. Model migration proceeds on bad data.

Audit failure. Compliance asks what you knew and when. You cannot reproduce March's scores with April's tooling.

Cross-team distrust. Domain team A tightens labels; domain team B does not. Platform compares pass rates across teams as if they measure the same thing.

Versioning model: treat eval repos like app repos#

Store eval artifacts in git — or an artifact store with content-addressed hashes referenced from git. Tag releases. Semver is appropriate:

• Major — label definition changes that invalidate historical comparison (new must-cite rules, removed cases, refusal policy redesigns)
• Minor — additive cases, new job classes, backward-compatible judge clarifications
• Patch — runner bug fixes, typos that do not affect scoring outcomes

Every eval run record should persist:

import hashlib
import json
from dataclasses import dataclass, asdict

@dataclass(frozen=True)
class EvalArtifactManifest:
    dataset_version: str
    runner_version: str
    judge_prompt_hash: str
    judge_model_alias: str

def hash_judge_prompt(prompt: str) -> str:
    return hashlib.sha256(prompt.encode()).hexdigest()[:16]

def manifest_fingerprint(m: EvalArtifactManifest) -> str:
    payload = json.dumps(asdict(m), sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]

@dataclass
class EvalRunRecord:
    manifest: EvalArtifactManifest
    system_commit: str
    pass_rate: float
    case_results: dict[str, bool]

def comparable(r1: EvalRunRecord, r2: EvalRunRecord) -> bool:
    return manifest_fingerprint(r1.manifest) == manifest_fingerprint(r2.manifest)

Call comparable() before diffing pass rates. If manifests differ, explain artifact changes first — do not treat the delta as a product regression verdict. When a case flips, triage in order: dataset version, judge version, then system under test. Only when all three match the baseline manifest should you block on a product regression.

Judge prompts deserve the same review as production prompts#

Judge prompts drift casually because "it's internal." They are not internal to decisions. A one-line change from "rate faithfulness 1-5" to "be generous when tone is professional" moves scores without moving users.

Require PR review for judge prompt changes. Attach eval-on-eval: run the new judge against a frozen calibration set where human labels are authoritative. Accept changes only when agreement with humans improves or holds within bounds. Pin the judge model alias separately from the application model alias — upgrading the judge is a semver minor at minimum; rerun baselines before trusting trend lines.

Dataset versioning without silent edits#

Golden sets grow. Version them as files or parquet with explicit changelogs: datasets/support-triage/v2.3.0/CHANGELOG.md and datasets/support-triage/v2.3.0/cases.jsonl.

Each entry: case ID added/removed/modified, reason, author. Avoid silent edits. "Fixing" a failing case label the night before launch destroys regression integrity. If product behavior changed, bump version and document. If the label was wrong, document that too — only transparency preserves trust in the harness.

Baselines are versioned artifacts too#

A blessed baseline is not "last green main." It is a specific run record: manifest fingerprint + system commit + pass rates per job class. Promote a new baseline deliberately after a release, not implicitly after any green nightly. When thresholds block promote, compare against that blessed baseline — not against yesterday's experimental judge tweak.

CI integration patterns#

• Eval runner Docker image tagged with runner_version
• Nightly publishes artifact bundle: results + manifest + system commit
• PR comments show case flips only when manifest matches base branch baseline, or clearly label "mixed comparison"
• Block merge on eval artifact changes without a changelog entry

Migration playbook for legacy unversioned evals#

If you inherit a spreadsheet and a bash script:

1. Export current cases to versioned JSONL v1.0.0
2. Hash current judge prompt as v1.0.0; pin model alias
3. Tag runner script commit as v1.0.0
4. Run once, store as provisional baseline
5. From this point, semver all changes

Do not pretend historical Slack screenshots constitute a baseline. Start the clock honestly.

Summary#

AI eval frameworks define what quality means in practice. Unversioned datasets, judge prompts, scoring logic, and baselines make regression analysis unreliable — false regressions, hidden declines, and audit gaps follow. Version eval artifacts with semver, persist manifest metadata on every run, review judge changes like production prompts, and promote baselines deliberately. Eval versioning is not bureaucracy; it is the precondition for scores anyone can trust enough to block or approve a release.