Why Vector Similarity isn't the Same as Relevance in RAG

The dashboard shows cosine 0.87. The demo felt great. Production users ask about "refund exceptions for EU enterprise accounts" and retrieval returns three chunks about "refund marketing copy," "enterprise SSO setup," and a footnote mentioning Europe in an unrelated privacy doc. All scored above 0.8. All useless. Vector similarity answered: "these embeddings are close in angle." It did not answer: "these passages contain the facts needed to respond." Treating similarity as relevance is the most common silent degradation in RAG — quality slides while metrics look healthy.

What similarity actually measures#

Embedding models map text into dense vectors such that semantic relatedness in training distribution correlates with distance. Cosine similarity (or dot product on normalized vectors) measures proximity in that space.

That is useful for candidate generation — surfacing plausibly related text cheaply at scale. It is not a guaranteed score of task relevance: whether a passage supports answering this question for this user with this intent.

Similarity conflates several distinct notions:

• Lexical overlap — weak in pure dense retrieval; homonyms and shared jargon inflate scores
• Topical relatedness — "refunds" and "billing disputes" cluster; exception clauses may not
• Entailment — passage must support the answer, not merely sit in the same topic cloud
• Specificity — generic FAQ outranks narrow policy addendum because it matches more queries in embedding space

Users care about entailment and specificity. Embeddings approximate topical relatedness. The gap is where wrong answers live.

Why the gap widens in production#

Training vs your corpus#

General-purpose embedders optimize open-web semantic similarity. Your corpus has SKUs, internal acronyms, ticket fragments, and tables. Out-of-distribution text embeds oddly — similar-looking boilerplate clusters while rare but correct passages sit farther away.

Chunking artifacts#

Half a policy section embeds like the whole policy. A chunk that mentions "refund" in passing matches refund queries without containing rules. Similarity rewards surface word overlap in vector space, not completeness of the retrieval unit.

Query-document asymmetry#

Many pipelines embed queries and documents with the same model but wrong prefixes — or embed concatenated title+body for docs and raw question for queries. Small asymmetries reorder ranks in ways demos never stress.

Score calibration#

Cosine 0.75 on one index build may mean something different after re-embedding with a new model. Teams set static thresholds ("accept if > 0.7") that silently stop working after a deploy.

Similarity vs relevance vs usefulness#

Production RAG needs a path from similarity to usefulness. Similarity alone stops at the first column.

What to add after vector search#

Hybrid retrieval#

Combine dense similarity with sparse lexical signals (BM25). Exact tokens — error codes, product IDs, statute numbers — often determine relevance more than semantic neighborhood. Hybrid fixes misses that look embarrassing in hindsight.

Metadata filters#

Similarity across the whole corpus when the user asked about product=Atlas and region=DE is self-inflicted noise. Filter first; similarity ranks within the correct slice.

Cross-encoder reranking#

Bi-encoders (separate query/doc embed) are fast but shallow. Cross-encoders score query-passage pairs jointly — much better proxy for relevance. Standard pattern: vector search for 50 candidates, rerank to 10.

def retrieve_relevant(query: str, index, reranker, k_final: int = 8):
    # Stage 1: high recall, similarity is OK here
    dense = index.dense_search(query, k=40)
    sparse = index.sparse_search(query, k=40)
    candidates = reciprocal_rank_fusion(dense, sparse)

    # Stage 2: relevance — do not skip without metrics
    scored = reranker.score_pairs(
        query,
        [c.text for c in candidates],
    )
    ranked = sorted(
        zip(candidates, scored),
        key=lambda x: x[1],
        reverse=True,
    )

    # Stage 3: threshold on reranker, not raw cosine
    results = []
    for chunk, rel_score in ranked:
        if rel_score < 0.15:  # calibrated on golden set
            break
        results.append((chunk, rel_score))
        if len(results) >= k_final:
            break
    return results

Notice the threshold applies to reranker scores calibrated on labeled data — not cosine copied from a tutorial.

Human-labeled golden sets#

Sample fifty to two hundred real queries. Label which chunks are fully relevant, partially relevant, or irrelevant. Measure recall@k and MRR at each stage. Without labels, you are tuning blind and calling cosine "good enough."

Silent degradation patterns#

Teams watch average similarity scores stay flat while answer quality drops. Common causes:

• Corpus drift — new docs cluster differently; old thresholds mis-rank
• Embedding model swap — scores not comparable; nobody recalibrated
• Chunk schema change — titles dropped from embed input; ranks shuffle
• Query shift — users ask narrower questions than the demo set; generic chunks win

None trigger alerts if you only monitor index size and mean cosine. Add downstream signals: faithfulness rate, citation precision, operator overrides, "wrong answer" tickets tagged with retrieved chunk IDs.

When to fine-tune embeddings#

Fine-tuning or training domain embedders helps when labeled pairs exist and hybrid + rerank still show a clear embedding recall gap. It is not step one. Fine-tuning on weak labels encodes the wrong notion of relevance and makes dashboards look improved while failures persist on edge cases.

Order of operations most teams should follow:

1. Fix chunking and metadata
2. Add hybrid retrieval
3. Add reranker + calibrated threshold
4. Build labeled eval set
5. Consider embedder fine-tune only with evidence from step 4

Skipping straight to step 5 because "we need better embeddings" is how teams burn a quarter without moving faithfulness numbers on real queries.

Do not ask the LLM to fix bad retrieval in the prompt#

"Pick the most relevant passages" in the prompt when you sent eight irrelevant chunks is not retrieval. It burns tokens and fails when all chunks look vaguely on-topic. Relevance judgment belongs before context assembly, not inside generation.

Instrument what you rank, not only what you generate#

Log per query: top-k chunk IDs, cosine scores, rerank scores, and which chunks entered the prompt. When a user flags a wrong answer, replay that tuple. Teams that only log final prompts discover retrieval drift weeks later.

Compare score distributions after index rebuilds. A shift in the 90th percentile cosine without a change in recall@10 on your golden set means your scores are miscalibrated — not necessarily that quality improved or collapsed.

Similarity has a role — know which one#

Use vector similarity for:

• First-stage recall at scale
• Deduplication and near-duplicate detection
• Clustering content for analytics

Do not use it as:

• The sole ranker for production answers
• A universal quality score comparable across model versions
• Proof that grounding will succeed

Summary#

Vector similarity measures proximity in embedding space — relatedness under a model's training — not user-meaningful relevance or answer-supporting entailment. RAG systems that equate the two degrade quietly: high cosines, wrong passages, fluent ungrounded answers. Treat similarity as the first stage of a retrieval stack, then add lexical signals, filters, reranking, calibrated thresholds, and labeled evals that target relevance and usefulness. The fix is architectural measurement, not a better cosine threshold copied from a blog post.