The Difference Between Retrieval-Augmented and Retrieval-Only Systems

Your search bar returns ten blue links sorted by BM25. Your copilot returns a paragraph that synthesizes an answer and cites page three. Both "retrieved" something. They are not the same system class. Conflating retrieval infrastructure with retrieval-augmented generation leads to wrong SLAs, wrong evals, and roadmaps that stop at a vector index when the product promise was natural-language answers. The distinction is simple to state and easy to ignore until legal asks why the chatbot paraphrased a number that never appeared in the source.

Retrieval-only: find and return#

A retrieval-only system answers: "Which documents or passages match this query?" Output is structured search results — titles, snippets, scores, URLs. The user reads, interprets, and synthesizes. No generative model sits on the critical path.

Examples: enterprise search portals, e-commerce catalog search, legal research databases that return case excerpts, developer doc search that lists matching pages.

Properties:

• Correctness is about ranking quality — precision and recall of relevant documents
• Latency is dominated by index lookup and ranking
• Failure is empty results or bad rank — visible and familiar to search teams
• Audit is "here are the hits" — no synthesis layer to hallucinate

Retrieval-augmented generation: find, then synthesize under constraint#

RAG adds a generation step that consumes retrieved passages and produces natural-language output bounded by a grounding contract. Retrieval is necessary but not sufficient. The product delivers an answer, not only candidates.

Examples: support copilots that draft replies, internal Q&A that summarizes policy across sections, code assistants that explain APIs using doc excerpts.

Properties:

• Correctness splits across retrieval and faithfulness of generation to retrieved text
• Latency includes retrieval plus LLM completion (often multi-second)
• Failure includes fluent wrong answers when retrieval was mediocre or generation ignored context
• Audit must cover both which chunks were retrieved and what the model added

Side-by-side comparison#

The conflation failure mode#

Teams ship strong search, wrap results in a prompt, call it RAG, and wonder why stakeholders expect chatbot behavior when ops still runs a search SLA.

Symptoms:

• Product shows a generated paragraph but eval only measures click-through on links
• No grounding prompt — model summarizes from parametric knowledge with snippets as decoration
• Citations point to wrong chunks because generation did not bind sentences to evidence
• Incidents blame "bad search" when retrieval was fine but the model extrapolated

Retrieval-only excellence does not automatically produce trustworthy RAG. The generation layer needs its own contracts and tests.

When retrieval-only is the right product#

Prefer retrieval-only when:

• Users are experts who want raw sources — researchers, engineers, analysts
• Mis-synthesis has high downside — medical, legal, financial numbers without human review
• Regulatory rules require displaying exact source text, not paraphrase
• Latency budgets are sub-second and LLM calls do not fit

You can still add optional summarization as a separate, clearly labeled action ("Summarize these results") rather than making synthesis the default path.

When RAG is the right product#

Prefer RAG when:

• Users ask questions in natural language and want a direct answer
• Answers require combining multiple passages — no single snippet suffices
• The product promise is reduced cognitive load — not another search results page
• You can invest in faithfulness evals and refusal behavior

RAG is not "search but prettier." It is a different liability profile.

Architecture diagram in code terms#

Retrieval-only API:

@dataclass
class SearchHit:
    document_id: str
    title: str
    snippet: str
    url: str
    score: float

def search(query: str, filters: dict, k: int = 10) -> list[SearchHit]:
    candidates = hybrid_search(query, filters, k=50)
    ranked = rerank(query, candidates)[:k]
    return [
        SearchHit(
            document_id=h.doc_id,
            title=h.title,
            snippet=truncate(h.text, 280),
            url=h.url,
            score=h.score,
        )
        for h in ranked
    ]

RAG API — note the separate generation contract:

@dataclass
class RagResponse:
    answer: str
    citations: list[SearchHit]
    refused: bool
    retrieval_snapshot_id: str

def rag_answer(query: str, filters: dict, llm) -> RagResponse:
    hits = search(query, filters, k=15)  # shared retrieval leg
    if not hits or hits[0].score < MIN_EVIDENCE:
        return RagResponse(
            answer="No reliable answer in the knowledge base.",
            citations=[],
            refused=True,
            retrieval_snapshot_id=current_snapshot(),
        )
    answer = llm.generate_grounded(
        query=query,
        passages=[h.full_text for h in hits],
        must_cite=True,
    )
    return RagResponse(
        answer=answer.text,
        citations=answer.used_hits(hits),
        refused=False,
        retrieval_snapshot_id=current_snapshot(),
    )

Same search() internals possible; different public contract and observability.

Hybrid products: search UI + copilot#

Many enterprise products offer both: a search page (retrieval-only) and a chat panel (RAG). Architecture wins when:

• Both paths call the same retrieval service — one index, one ACL model
• Metrics stay separate — search CTR vs answer faithfulness
• UX labels differ — users know chat synthesizes; search returns sources
• Copilot refuses when retrieval is weak instead of falling back to parametric trivia

Sharing retrieval avoids dual indexes; blurring UX avoids neither team's accountability.

Eval implications#

Retrieval-only eval: labeled query-document relevance, preferably chunk-level for long docs.

RAG eval adds:

• Faithfulness — claims in answer supported by cited chunks?
• Citation precision — cited chunks actually used?
• Refusal quality — unanswerable queries do not produce confident nonsense?
• Completeness — multi-hop questions need multiple chunks; did assembly include them?

Running only search metrics on a RAG product leaves the highest-risk layer unmeasured. Schedule faithfulness evals on the same cadence as retrieval recall — weekly for active products, on every index promotion at minimum. A regression in citation precision is as ship-blocking as a drop in recall@5 when your UX presents synthesized answers as authoritative.

Latency and SLO expectations differ#

Search teams often target sub-200ms p95 for retrieval-only paths — users expect snappy result lists. RAG paths that add assembly plus an 800-token completion may legitimately run 2–5 seconds. Promising search SLAs on a copilot creates perpetual "performance bugs" that are actually product category mismatch.

Set separate SLOs: retrieval leg latency (shared) vs end-to-end answer latency (RAG only). When the copilot is slow, trace whether retrieval or generation dominated. Blaming the index when the model ran a 4k-token self-critique loop misdiagnoses the problem.

Naming discipline for engineering clarity#

Use precise language in design docs and tickets:

• "Retrieval service" — hybrid search + rerank returning passages
• "Search experience" — retrieval-only UI
• "Grounded generation" — LLM step with evidence constraint
• "RAG product path" — retrieval service → assembly → grounded generation → citation

Calling the vector database "the RAG" confuses infrastructure with the full pattern. Calling search "RAG" because embeddings are involved confuses ranking with synthesis.

Summary#

Retrieval-only systems find and return evidence; retrieval-augmented generation finds evidence and synthesizes answers under grounding rules. They may share indexes and ranking stacks, but outputs, failure modes, evals, and product liabilities differ. Design explicitly for one or both — and do not treat a finished search bar as a finished copilot. The generator is not a thin wrapper; it is a component that can invent fluent falsehoods unless retrieval, assembly, and faithfulness are engineered with the same rigor you apply to the index.