Context Windows as Engineered State

Most “prompt bugs” I see in production are not wording problems. They are packing problems. The window fills with history, policy, retrieved chunks, and tool dumps until the model has no room left to do the job you asked for.

Context engineering is the discipline of deciding what enters the window, in what order, under what budget, and what gets dropped when something has to give.

The packing failure#

A support copilot starts clean: system policy, the ticket body, two retrieved articles. Three months later every turn ships the full transcript, a CRM blob, and twelve “maybe relevant” chunks. Latency climbs. Answers drift. Nobody changed the model. They changed the cargo.

If you cannot say, for a given request, how many tokens went to policy vs evidence vs task, you are not engineering context. You are concatenating strings.

This shows up in reviews as “the model got worse after we improved retrieval.” Often retrieval did improve — you just flooded the window until the task instruction was a footnote. The fix is not a warmer temperature. It is a packing policy you can explain to another engineer.

Regions beat one blob#

Treat the window like a small memory map, not a diary:

Keep region boundaries visible in the assembled prompt (labels or structured message roles). When quality tanks, you can tell whether policy collided with evidence or whether evidence simply ate the budget.

Budget first, then fill#

Pick a hard ceiling for the model you actually call. Reserve tokens for the completion. Split the rest on purpose.

type Region = "policy" | "task" | "evidence" | "memory" | "output";

const BUDGET: Record<Region, number> = {
  policy: 800,
  task: 400,
  evidence: 3500,
  memory: 600,
  output: 300, // instructions for the schema, not the answer tokens
};

function pack(region: Region, chunks: string[], maxTokens: number): string {
  // Prefer recent / higher-scored chunks; drop the rest — never silently overflow
  const kept: string[] = [];
  let used = 0;
  for (const chunk of chunks) {
    const n = estimateTokens(chunk);
    if (used + n > maxTokens) break;
    kept.push(chunk);
    used += n;
  }
  return kept.join("\n\n");
}

estimateTokens can be rough. A wrong estimate that still enforces a ceiling beats perfect counting with no ceiling.

Order inside a region matters too. Put the highest-signal evidence first when the model is known to under-use the tail of long contexts. Do not bury the ticket id under six “related” paragraphs the retriever loved and the agent never needed.

What belongs outside the window#

Long transcripts belong in summary form or in a retriever, not pasted wholesale. Large tool payloads belong in a store with an id the model can request again. Raw HTML, base64, and full JSON trees are how you burn money without improving answers.

A useful rule: if a human reviewer would not read it for this decision, do not ship it to the model for this decision.

That rule also covers “debug leftovers.” Engineers paste stack traces, raw HTTP dumps, and yesterday’s failed plans into the prompt “just for this incident,” then forget to remove them. Those leftovers become the new baseline. Treat the assembler as code review territory: unexpected fields in the packed prompt are bugs.

Operational habits for context budgets#

1. Log region token counts on every production call.
2. Version the assembler the way you version code — not by editing a prompt in a dashboard.
3. Score failures by region: wrong evidence, conflicting policy, broken output contract.
4. Cap retrieval before you invent a cleverer prompt.

Before and after a budgeted window#

Ticket reply workflow, same model:

• Before: full CRM dump + 15 chunks + entire chat (~9k input tokens). Answers wandered; cost per ticket looked “fine” until volume hit.
• After: policy 700, task 250, top-4 chunks by score, 8-turn summary instead of raw chat (~3.2k input). Clarification rate went up slightly. Wrong-article citations dropped. Finance noticed before the model vendors did.

That is the usual shape of a context win: less cargo, clearer regions, measurable eviction.

If you need a second iteration, change one region budget at a time and keep a golden set of ten tickets. Otherwise you will credit the wrong knob when quality moves — and you will ship the next packing change blind.

Summary#

Context engineering is state management for probabilistic systems. Budget the window, separate regions, evict deliberately, and keep bulky data behind retrieval. Clever phrasing still helps at the margins. Packing discipline is what keeps the model useful when the product grows.