Token Budgets as an Architectural Constraint in AI-Native Systems

Architecture reviews for AI-native systems cover auth, idempotency, rate limits, and data residency. Token budgets rarely appear — until the first invoice, the first p95 latency breach, or the first incident where the model ignored instructions because evidence consumed ninety percent of the window. Token limits are not prompt trivia. They are structural constraints like memory, CPU, or request payload size. Design for them upfront or pay in production firefighting.

A token budget is a per-region ceiling on what assembly may send to the model, plus a reserved slice for completion. Treating budgets as architecture means they appear in diagrams, ADRs, code modules, and SLO discussions — not only in a footnote on the system prompt.

Why tokens belong in architecture docs#

Every LLM call consumes a scarce shared resource: context window capacity. That capacity binds:

Traditional APIs fail loudly on 413 Payload Too Large. Most LLM integrations silently truncate or send oversized prompts until quality rots. Budgets make overflow explicit — a design choice with a defined failure mode.

Budget layers: model, request, and region#

Region budgets sum to less than request budget. The gap is headroom — buffer for tokenizer mismatch and future additions without emergency refactors.

export const TICKET_REPLY_BUDGET = {
  modelLimit: 32_768,
  completionReserve: 2_048,
  regions: {
    policy: 900,
    examples: 600,
    evidence: 4_500,
    memory: 800,
    task: 500,
    outputContract: 250,
  },
} as const;

export function assertBudget(regions: Record<string, string>): void {
  const inputCap =
    TICKET_REPLY_BUDGET.modelLimit - TICKET_REPLY_BUDGET.completionReserve;
  const used = Object.entries(TICKET_REPLY_BUDGET.regions).reduce(
    (sum, [key, cap]) => {
      const text = regions[key] ?? "";
      const tokens = countTokens(text);
      if (tokens > cap) {
        throw new BudgetViolationError(`region ${key}: ${tokens} > ${cap}`);
      }
      return sum + tokens;
    },
    0,
  );
  if (used > inputCap) {
    throw new BudgetViolationError(`total input ${used} > cap ${inputCap}`);
  }
}

BudgetViolationError should surface to metrics and block the call — not fall back to "best effort" truncation without logs.

Architectural patterns driven by budgets#

Tiered models. Small model for summarisation within tight budget; large model only when assembled pack exceeds threshold or task class demands it.

Store-by-reference. Tool outputs larger than N tokens go to object storage; context carries handle + summary. Same pattern as passing IDs instead of JOIN results in SQL services.

Prompt caching. Stable policy + examples prefix cached at provider; architecture assigns those tokens to cacheable system region to cut cost and latency.

Progressive assembly. First pass retrieves titles only; second pass fetches bodies for top two — bounded work instead of unbounded fetch-then-paste.

Workflow decomposition. Task needs more reasoning room than budget allows → split into chained steps with compact handoff state instead of one megaprompt.

Each pattern is an architecture trade-off written because tokens are finite.

Budgets in non-functional requirements#

Include token budgets in NFRs the way you include response time:

• Max input tokens per endpoint and percentile
• Max output tokens configured on the client, not unlimited default
• Cost ceiling per user session derived from token price × expected turns
• Degradation path when budget would be exceeded — queue, summarise, refuse

SLO dashboards should chart p95_input_tokens alongside p95_latency. They correlate. Teams that only watch latency often miss the root cause in assembly.

Anti-patterns in architecture reviews#

1. Unbounded history append — chat stored as raw string growth; no summarisation job, no turn cap.
2. Integration passthrough — partner webhook body concatenated to prompt without size check.
3. Prompt as storage — static knowledge pasted in system message because vector index was "phase two."
4. Silent truncation helper — utility that cuts strings at character boundary before send; destroys instructions mid-word.
5. Budget only in comments — // keep under 4k with no runtime enforcement.

Reject these in design review the way you reject unbounded SELECT *.

Cross-team contract#

Platform publishes budget constants per workflow. Product teams request new regions with justification in tokens and eval impact. Security assigns minimum policy region size — not "add another paragraph" ad hoc.

Changes to region budgets require:

• Updated ADR with latency/cost estimate
• Eval run on golden set with new packing
• Trace dashboard segment for before/after comparison

Budgets in system diagrams#

If your architecture diagram shows "LLM" connected to "Vector DB" with no box between them, budgets are missing from the drawing. Add an assembly service node with explicit inputs and ceilings:

• Policy registry → policy region (800 tokens max)
• Retriever → evidence region (4k tokens max, eviction policy v2)
• Session store → memory region (summarised, 600 tokens max)
• API gateway → task region (user payload, validated size)

Reviewers should ask what happens when any arrow exceeds its label. If the answer is "the SDK truncates," the diagram is aspirational.

Capacity planning uses budgets too. Forecasting spend as monthly_requests × avg_input_tokens × input_price beats guessing from GPU hours. When product projects ten million new requests, finance should see token impact before launch — not after the first consolidated invoice.

Testing budgets like any constraint#

Unit tests on packing functions: given oversized evidence, expect eviction order, not throw uncaught. Contract tests: assembly output never exceeds request budget with worst-case fixtures. Load tests: p95 tokens under peak retrieval, not just p95 milliseconds.

import pytest
from assembly import pack_region, BudgetExceeded

def test_eviction_drops_lowest_score_first():
    chunks = [
        {"id": "a", "text": "x" * 400, "score": 0.91},
        {"id": "b", "text": "y" * 400, "score": 0.55},
        {"id": "c", "text": "z" * 400, "score": 0.88},
    ]
    packed = pack_region(chunks, max_tokens=500, strategy="drop_lowest_score")
    ids = {c["id"] for c in packed}
    assert "b" not in ids
    assert "a" in ids

def test_refuses_when_task_region_overflows():
    with pytest.raises(BudgetExceeded):
        pack_region("task text" * 10_000, max_tokens=200, strategy="fail")

Incident pattern: death by accretion#

The most common budget failure is slow accretion, not one bad deploy. A compliance paragraph in January. A partner feed in March. Full chat history in May. By July, task instructions truncate and nobody knows which edit pushed the pack over the cliff because assembly never logged region sizes.

Post-incident, the team added budgets in a hurry — but only as a global if len(prompt) > N: chop. That chop removed JSON schema instructions while leaving six redundant FAQ chunks. Quality stayed broken; latency improved slightly.

The durable fix matched architecture to ops: region-level enforcement, eviction order documented, traces with per-region token counts, and design review gates for any new context source. Budgets are not a one-time patch after an outage. They are the guardrail that prevents the outage.

Finance and platform should share one spreadsheet row: budget tokens × expected volume × price per million. When product asks for "full document context" on every call, translate the request into dollars and milliseconds before approving. Architecture is where that translation happens — not in the first production invoice.

Summary#

Token budgets are architectural constraints: they define what your AI-native system can know, how fast it responds, and what it costs per request. Assign region ceilings, reserve completion headroom, enforce overflow in code, and document eviction as policy. Budgeting belongs in diagrams and ADRs — not as emergency surgery on a prompt that grew until the window broke.