Inference Infrastructure is Where AI Features Survive Production

The demo worked. The eval set looked clean. Then production traffic arrived and the feature broke in ways nobody predicted — not because the model was wrong, but because inference infrastructure could not sustain the load, cost envelope, or failure modes the product assumed. Prompts and model weights get the attention. Inference infrastructure determines whether any of it survives contact with real users.

Prompts do not set your p95#

A team I worked with cut average output tokens by 30% through prompt refinement. p95 latency went up. The reason: shorter outputs increased request rate on a shared GPU pool with fixed batching capacity. More requests per second meant more queue wait time. The prompt change was correct. The infrastructure had no isolation between interactive and batch traffic, and no queue-depth alerting.

Latency is a property of the full path — gateway queue, provider time-to-first-token, tool round-trips, streaming delivery, and client-side assembly. Model choice affects one segment. Infrastructure owns the rest. Teams that optimise prompts without measuring queue depth are tuning the engine while the transmission fails.

Cost is an infrastructure allocation problem#

Finance asks "why did AI spend triple?" The answer is rarely "we picked GPT-4 instead of GPT-3.5." More often:

• Retry multipliers doubled after a tool schema change
• A batch embedding job shared a rate limit with online traffic, causing retries
• Context windows grew because nobody enforced max input tokens at the gateway
• A fallback model routed 40% of traffic during a partial outage — at 3× the unit cost

Spend caps, per-workflow envelopes, and automatic degraded modes belong in inference infrastructure. Application code can request fewer tokens; only the gateway can refuse a request that would blow the budget.

Reliability is not provider uptime alone#

Vendor SLAs cover their API availability. They do not cover your system's ability to survive their degradation gracefully. Inference infrastructure must implement:

• Fallback routing to a secondary model when primary pool error rate exceeds threshold
• Circuit breakers on tool calls that stall the inference path
• Load shedding with pre-tested degraded modes before full feature outage
• Timeout budgets per workflow, enforced centrally

A provider at 99.9% uptime still gives you 43 minutes of downtime per month. If you have no fallback alias and no degraded playbook, that is 43 minutes of user-visible failure — regardless of how good your prompts are.

from dataclasses import dataclass
from enum import Enum

class DegradedMode(str, Enum):
    FULL = "full"
    NO_CRITIQUE = "no_critique"
    SMALL_MODEL = "small_model"
    REFUSE = "refuse"

@dataclass
class InferencePolicy:
    alias: str
    fallback_alias: str | None
    max_input_tokens: int
    max_retries: int
    timeout_ms: int
    degraded_mode: DegradedMode

POLICIES: dict[str, InferencePolicy] = {
    "support-reply": InferencePolicy(
        alias="support-reply-primary",
        fallback_alias="support-reply-fallback",
        max_input_tokens=12000,
        max_retries=2,
        timeout_ms=8000,
        degraded_mode=DegradedMode.SMALL_MODEL,
    ),
}

def resolve_alias(workflow: str, error_rate: float, spend_rate: float) -> str:
    policy = POLICIES[workflow]
    if error_rate > 0.05 and policy.fallback_alias:
        return policy.fallback_alias
    if spend_rate > 1.5:  # 150% of hourly envelope
        return policy.fallback_alias or policy.alias
    return policy.alias

This logic lives in the gateway, not in every microservice. Centralised policy is the difference between a controlled degradation and a cascading failure.

The operational foundation product teams ignore#

Product roadmaps list features. Platform roadmaps list gateways. The gap between them is where AI systems die slowly.

Early-stage teams skip infrastructure because managed APIs abstract it away. That works until:

• You add a second model family and routing becomes tribal knowledge
• Finance wants chargeback by product line and nobody tagged requests
• An agent loop multiplies provider calls and rate limits appear "randomly"
• A security review asks where prompts are logged and for how long

Each of these is an inference infrastructure milestone. Delaying them does not avoid the work — it moves the work to incident response at 2 a.m.

Latency SLOs require infrastructure SLOs#

User-facing latency targets like "answer within 5 seconds" decompose into infrastructure budgets:

If you have not assigned budgets to each segment, you cannot tell whether a latency regression is a model problem, a queue problem, or a tool problem. Infrastructure teams that own queue and gateway segments can act before product teams notice.

Security and compliance live at the boundary#

Prompt logging, PII redaction, data residency enforcement, and audit trails are inference infrastructure concerns. Application code should not decide whether a prompt containing customer data goes to a model in an unapproved region — the gateway should block it.

Tool credentials, MCP server access, and external API keys invoked during agent loops need the same custody model as database credentials. Scattered in service configs, they become rotation nightmares. Centralised in the gateway or a secrets-aware proxy, they become auditable.

Teams that bolt security on after launch discover that retrofitting redaction into twelve SDK call sites costs more than building the gateway first.

What "good" infrastructure feels like to product engineers#

Product teams should experience inference infrastructure as invisible when it works:

• Stable aliases they reference without knowing the underlying model version
• Clear error messages when budgets or limits are hit — not opaque 429s from a vendor
• Dashboards that show their workflow's latency, cost, and error rate without filing a ticket
• Degraded modes that activate automatically during outages, with comms templates ready

When infrastructure is bad, product teams feel it as "the model is flaky" or "OpenAI is down again." That misdiagnosis wastes weeks on prompt iteration that cannot fix a routing problem.

Building the foundation in the right order#

1. Gateway with aliases and tagging — before a second model
2. Token-level logging and tracing — before finance asks
3. Per-workflow spend envelopes — before the first cost surprise
4. Fallback aliases and degraded modes — before the first outage
5. Queue isolation for batch workloads — before batch jobs starve interactive traffic

Model bake-offs and prompt libraries can proceed in parallel. Skipping steps 1–3 and jumping to self-hosted GPUs is a common way to spend capital without improving reliability.

Incident patterns that trace back to infrastructure#

Review your last three AI-related incidents. Common root causes that look like model failures:

• Rate limit 429 cascade — no central backoff; every service retries independently until the provider blocks the account.
• Context overflow — application assembled 14,000 tokens; gateway had no limit; provider rejected or truncated silently.
• Stale alias — control plane updated model revision; one region still routed to retired pool; quality dropped for 6% of traffic before anyone noticed.
• Batch job starvation — nightly embedding run consumed all gateway concurrency; interactive p95 tripled during business hours in another timezone.

Each pattern has an infrastructure fix and a prompt fix. Teams that reach for prompt changes first extend the incident. Teams that fix routing, limits, and queue isolation prevent recurrence.

Summary#

Inference infrastructure is where AI systems live or die because reliability, latency, and cost are enforced at that layer — not in prompts and not in weight files. Retries, routing, queue depth, spend caps, fallback policies, and token-level telemetry determine whether a capable model becomes a dependable product. Own the operational foundation before you optimise the model. The teams that survive production traffic are the ones that treated inference as infrastructure from the start, not as an API call wrapped in a try block.