Why More Context Doesn't Improve LLM Output Quality

The retrieval team shipped a win: recall up eighteen points after raising top_k from five to fifteen. Product quality dropped. The model started citing tangentially related docs and ignoring the refund paragraph buried on page four of the pack. More context made the system worse — and more expensive — because nobody measured precision at the window boundary.

The intuition that "more information helps the model" comes from human reading. LLMs do not read like humans. They attend over everything you send, with uneven weight across position, relevance, and duplication. Past a point, additional tokens add noise, consume budget that task instructions needed, and slow prefill. Curation — deciding what not to send — is the engineering skill production systems lack.

Diminishing returns are real#

Quality often rises sharply when you move from missing critical evidence to including it. Gains flatten once the relevant facts are present. Adding marginally related paragraphs rarely helps and sometimes hurts.

Typical curve in RAG copilots:

The middle row is where engineering effort pays off. The last row is where demos go to die.

Negative returns: when more context hurts#

Several mechanisms produce negative returns:

Attention dilution. Long prompts spread attention across low-value tokens. The instruction to "answer using only provided sources" competes with thousands of tokens the model treats as equally available surface.

Contradiction exposure. Retrieval at high recall surfaces incompatible snippets — old policy next to new policy, marketing copy next to legal disclaimers. The model picks one fluently.

Position bias. Many models overweight early and late context. Material in the middle — often your retrieved answer — gets under-used. Adding more chunks pushes critical lines into the dead zone.

Duplicate signal. Near-duplicate chunks do not add information; they add repetition that models sometimes treat as emphasis, skewing answers toward repeated but wrong phrasing.

A procurement bot demonstrated all four. Engineers increased chunk count "so the model would not miss anything." The pack included three versions of the vendor policy and two unrelated FAQ entries that shared vocabulary with payment terms. Accuracy on payment questions fell while token cost tripled.

Curation as engineering, not prompt tuning#

You cannot prompt your way out of a bloated pack. "Be concise" does not remove tokens. "Ignore irrelevant context" is unreliable when irrelevant context is present — models often use it anyway.

Curation belongs in code before the provider call:

from __future__ import annotations

from dataclasses import dataclass
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

@dataclass
class ScoredChunk:
    id: str
    text: str
    embedding: np.ndarray
    score: float

def curate_evidence(
    query_embedding: np.ndarray,
    candidates: list[ScoredChunk],
    max_chunks: int = 4,
    min_score: float = 0.72,
    dedupe_threshold: float = 0.92,
) -> list[ScoredChunk]:
    """Prefer fewer, stronger, non-redundant chunks over raw recall."""
    filtered = [c for c in candidates if c.score >= min_score]
    filtered.sort(key=lambda c: c.score, reverse=True)

    kept: list[ScoredChunk] = []
    for chunk in filtered:
        if len(kept) >= max_chunks:
            break
        if _is_duplicate(chunk, kept, dedupe_threshold):
            continue
        kept.append(chunk)
    return kept

def _is_duplicate(
    candidate: ScoredChunk,
    kept: list[ScoredChunk],
    threshold: float,
) -> bool:
    if not kept:
        return False
    sims = cosine_similarity(
        candidate.embedding.reshape(1, -1),
        np.vstack([k.embedding for k in kept]),
    )[0]
    return float(sims.max()) >= threshold

max_chunks=4 is a product decision backed by evals, not a universal constant. The point is an explicit cap with deduplication — not unbounded append.

What to drop before you stretch the window#

When budgets force eviction, cut in this order unless your compliance regime says otherwise:

1. Low-score retrieval — below threshold, not "might help"
2. Redundant chunks — same doc, overlapping sections
3. Stale versions — superseded policy after version filter
4. Verbose tool dumps — summarise or store-by-reference
5. Old conversation turns — summarise memory slot instead
6. Decorative few-shots — keep one strong example, drop the rest

What you should almost never cut first: current task text, active policy version, required metadata, output schema contract.

Measuring curation quality#

Offline metrics should include pack precision, not just retrieval recall@k:

• Citation hit rate — cited IDs exist in the pack
• Answer groundedness — claims supported by included chunks (human or LLM judge on frozen packs)
• Contradiction rate — incompatible statements in selected chunks
• Task visibility — token distance from start of user message to task header
• Cost per successful answer — input tokens when downstream validator passes

Run ablations: same prompt, varying chunk counts. If quality peaks at three chunks and falls at eight, your production default should be three — not eight because the index allows it.

The "just in case" integration pattern#

Every new data source wants default inclusion: "Append our widget context; it's only four hundred tokens." Four integrations later you are at 1,600 tokens of "just in case" material nobody reads in traces. Feature teams optimise locally; assembly owners must refuse or budget.

Gate new context sources with:

• Documented user-visible benefit in evals
• Assigned region and token ceiling
• Removal policy when unused in traces for thirty days

Context is not free storage. It is recurring inference tax.

When more context does help#

Adding tokens is correct when:

• Evals show a specific missing fact causes failures
• The task requires multi-hop evidence you can label and order
• Legal or audit requires full text inclusion — then expand budget and model tier deliberately, not by accident

Even then, prefer structured addition (one labeled block with ID) over dumping raw files.

Worked example: support macro suggestions#

A SaaS support team built macro suggestions on top of ticket history plus ten retrieved macros. Offline evals with top_k=10 looked strong — the right macro appeared somewhere in the list ninety-four percent of the time. In production, agents complained the model "picked weird macros."

Traces showed packs averaging 6,200 tokens. The actual ticket question occupied line forty. Macros one through six were loosely related past tickets with overlapping vocabulary. The model latched onto repeated phrases in early macros and ignored the ticket body.

The fix was not prompt wording. Engineers cut to four macros with deduplication, moved the ticket summary into a [task] header immediately after metadata, and added a hard 2,800-token evidence ceiling. Recall@10 in the index still mattered for indexing health; pack precision became the launch metric. Suggestion acceptance rose twenty-one points. Input tokens fell fifty-five percent.

That is the curation story in miniature: the information existed in the index; the window contained too much of it in the wrong shape.

Pair curation with retrieval ownership#

Retrieval teams own recall. Context engineering owns what crosses the provider boundary. Without a handshake, you get optimisations that help offline dashboards and hurt live users.

Define SLAs together:

• Retrieval returns ranked candidates with scores — not final packs
• Assembly applies caps, dedupe, and version filters
• Weekly review samples traces where citation IDs were wrong or absent
• Regression tests fail if average pack size grows more than ten percent without eval approval

When someone proposes raising top_k, ask for the ablation chart. If answer quality peaks lower, the proposal is a latency and cost regression dressed as an improvement.

Summary#

More context does not monotonically improve LLM output. After critical evidence is present, extra tokens often add noise, contradictions, latency, and cost while burying the task. Treat curation as engineering: caps, deduplication, score thresholds, and pack-level evals — not larger paste buffers and hope. The best context window is the smallest one that still contains the facts the decision requires.