The Anatomy of a Production-Ready RAG Pipeline

Production RAG is not "embed PDFs, call OpenAI." It is a pipeline with nine distinguishable stages, each with inputs, outputs, failure modes, and owners. Teams that collapse those stages into one notebook cell cannot tell whether a bad answer came from stale ingestion, broken chunk boundaries, weak recall, or a model that ignored context. This article is a reference anatomy — a checklist you can lay next to your diagram and ask, for every box: do we have this, who runs it, and what do we measure?

Stage map overview#

A production-ready path looks like this:

1. Ingestion — pull sources, normalize format, attach metadata and ACLs
2. Chunking — split into retrieval units that preserve meaning
3. Embedding — vectorize chunks with a pinned model and schema
4. Indexing — store vectors, sparse terms, metadata in a searchable index
5. Retrieval — hybrid candidate generation under filters
6. Reranking — second-stage scoring for precision
7. Context assembly — pack evidence into a token budget
8. Generation — LLM with grounding contract and structured output
9. Citation and evaluation — link answers to evidence; score regressions

Skip a stage without documenting why, and you inherit its failure modes invisibly.

Ingestion: sources become documents#

Ingestion connects authoritative systems to the RAG corpus: CMS exports, Confluence APIs, ticket archives, git-backed docs, S3 buckets. Each run should produce document records with stable IDs, source URI, content hash, version or updated_at, language, and entitlement attributes.

Production requirements:

• Idempotent runs — same source version yields the same document ID
• Tombstones — deleted sources remove or mark chunks inactive
• Lineage — log which connector produced which record for audit

Without tombstones, RAG answers from retracted policies. Without ACL metadata carried forward, retrieval leaks restricted content into prompts.

Chunking: the unit of retrieval#

Chunks are what retrieval returns — not whole files. Bad chunking is the silent killer of RAG quality; see dedicated failure-mode writing elsewhere. At architecture level, specify:

• Strategy per content type — prose by heading, code by function, FAQs as Q+A pairs
• Max/min size with overlap only where continuity requires it
• Chunk schema — chunk_id, document_id, ordinal, heading_path, text

Version chunk schema changes. Rechunking is a reindex event, not a silent overwrite.

Embedding and indexing#

Embedding turns chunk text into dense vectors. Pin model name, dimension, normalization, and prefix rules (some models expect "search_document: " / "search_query: " prefixes). Mixed embedding models in one index destroy recall.

Indexing combines:

Index builds should be immutable snapshots. Serving queries against a half-built index is how you get missing documents with no error. Blue/green index swap is standard: build index_vN+1, validate, flip read pointer, retire old.

Retrieval: candidates, not answers#

Retrieval answers: "What passages might contain evidence?" — not the user's question directly. Production retrieval almost always:

1. Applies metadata filters (tenant, region, product, clearance)
2. Runs dense + sparse search in parallel
3. Fuses ranks (RRF is common)
4. Returns top 40–100 candidates for reranking — not top 5 for the prompt

Latency budget here is real. Parallelize search legs; cache frequent filter sets if needed.

Reranking: similarity to relevance#

First-stage retrieval optimizes recall. Cross-encoder or lightweight rerankers optimize precision over the candidate set. Typical pattern: rerank top 50, keep top 8–15 for assembly. Skipping rerank is a valid cost trade-off only if golden-set metrics prove precision is already sufficient — not if demos felt fine.

Context assembly: memory pressure#

The context window is finite. Assembly:

• Sorts reranked chunks by score and optionally diversity (MMR reduces near-duplicate paragraphs)
• Enforces a token budget leaving room for system prompt, query, and answer
• Drops trailing low-score chunks rather than truncating mid-chunk when possible
• Detects contradictions — flag conflicting passages in the prompt or refuse

Stuffing maximum context is an anti-pattern. More tokens raise cost and often hurt answer quality when passages disagree.

Generation: grounding contract#

The LLM step receives assembled evidence and a strict instruction set:

• Answer only from provided passages
• Cite chunk IDs inline or in a structured field
• Refuse when evidence is insufficient
• Optional: output schema (JSON) for downstream UI

Temperature for grounded QA is usually low. Creativity belongs in rewrite steps after facts are fixed, if at all.

from pydantic import BaseModel, Field

class GroundedAnswer(BaseModel):
    answer: str
    citation_ids: list[str] = Field(default_factory=list)
    confidence: float
    refused: bool = False

def generate_grounded(
    llm,
    query: str,
    chunks: list,
    token_budget: int,
) -> GroundedAnswer:
    packed = pack_chunks(chunks, token_budget)
    if not packed or packed[0].rerank_score < 0.2:
        return GroundedAnswer(
            answer="I cannot answer from available sources.",
            refused=True,
            confidence=0.0,
        )
    prompt = build_grounding_prompt(query, packed)
    return llm.parse(prompt, schema=GroundedAnswer)

Citation and user-facing provenance#

Citations are not decoration. They tie UI snippets to source documents users can verify. Architecture should pass chunk ID → document URL/title mapping to the client. If you only show model prose, trust erodes the first time prose paraphrases a number wrong.

Evaluation: close the loop#

Production pipelines need continuous measurement:

• Retrieval recall@k on labeled question → gold chunk sets
• Rerank MRR — did the best chunk move to the top?
• Answer faithfulness — is the answer supported by retrieved text?
• Refusal accuracy on unanswerable questions
• Index freshness lag — time from source update to searchable chunk

Run evals on index build promotion, embedding model changes, and prompt updates. Block promotion when metrics drop past thresholds.

Operational cross-cutting concerns#

Across all stages:

• Observability — trace ID linking query → retrieval scores → chunk IDs → prompt hash → response
• Security — filters before retrieval, not after generation
• Cost — embedding batch jobs vs query-time embed; rerank batch size
• Disaster recovery — rebuild index from source of truth, not from vector DB alone

The vector store is a cache of derived data. Source systems remain authoritative.

Stage ownership in practice#

In mid-size teams, a sensible default split:

• Data platform owns ingestion connectors, chunk jobs, index builds, tombstones
• Search or ML platform owns hybrid retrieval, rerankers, embedding model pins
• Application team owns context assembly, prompts, citation UX, product refusal copy
• Quality engineering owns golden sets, faithfulness scoring, promotion gates

Handoffs fail when "the RAG team" is one engineer who also owns the demo notebook. Explicit RACI on each stage prevents the index from rotting while prompts get all the attention.

Anti-pattern: collapsing stages#

Monolithic "RAG libraries" that hide ingestion and eval behind one query() call are fine for prototypes. In production, stage boundaries match team boundaries and deploy cadences. Ingestion may deploy daily; embedder upgrades monthly; generator prompts hourly. Coupling them in one opaque box prevents safe iteration.

Summary#

A production-ready RAG pipeline spans ingestion through evaluation — nine stages with distinct contracts and metrics. Use this anatomy as a reference architecture: name each box in your system, assign ownership, and measure the handoffs. Missing stages do not disappear; they show up as wrong answers you cannot diagnose. Build the pipeline so you can point to the stage that failed — then fix it with evidence instead of endless prompt tweaks.