Building Retry Logic for LLM Structured Output Failures

Structured outputs fail in boring ways: truncated JSON, missing braces, wrong types, hallucinated keys. The instinct is to retry the model. Sometimes that is right. Often it is how you turn a one-token mistake into a three-call bill and a slower page. Good retry logic knows what failed, whether another sample helps, and when to stop and fix the design.

Bad retry logic is one of the fastest ways to multiply cost without improving reliability. A team sees a 12% structured-output failure rate, adds a retry loop, and celebrates when failures drop to 4%. Nobody checks that p95 latency doubled, token spend tripled, and the remaining 4% are systematic schema mismatches that no amount of resampling will fix. Good retry logic makes failures cheaper. Bad retry logic makes them expensive and hides the design bugs causing them.

Two failure classes#

Retriable (sampling / transient)#

• Network or gateway timeouts
• Empty response
• Truncation from max-tokens too low (after you raise the limit once)
• Occasional JSON syntax errors on an otherwise good schema
• Rate-limit responses (429) with a retry-after header

Another draw can succeed. Cap attempts. Back off. These failures are uncorrelated with the prompt — a different sample from the same prompt may parse cleanly.

Non-retriable (design)#

• Schema and prompt disagree (required field never mentioned)
• Model systematically invents enums not in the schema
• Output too large for the task contract
• Validation fails on business rules the model was never told
• Consistent wrong types (model always returns "amount": "500" instead of integer)

Retrying here is superstition. Fix prompt, schema, or model choice. If three consecutive attempts fail with the same schema_mismatch on the same field, you do not need a fourth attempt — you need a design review.

A minimal retry policy#

import json
import time
from pydantic import BaseModel, ValidationError

class Extract(BaseModel):
    title: str
    tags: list[str]

TRANSIENT = {"timeout", "empty", "json_syntax"}
MAX_ATTEMPTS = 3

def classify(err: Exception, raw: str | None) -> str:
    if raw is None or raw.strip() == "":
        return "empty"
    try:
        json.loads(raw)
    except json.JSONDecodeError:
        return "json_syntax"
    if isinstance(err, ValidationError):
        return "schema_mismatch"
    return "unknown"

def generate_structured(call_model, prompt: str) -> Extract:
    last = None
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            raw = call_model(prompt)
            return Extract.model_validate_json(raw)
        except Exception as e:
            kind = classify(e, locals().get("raw"))
            last = (kind, e)
            if kind not in TRANSIENT or attempt == MAX_ATTEMPTS:
                break
            time.sleep(0.2 * attempt)  # simple backoff
    kind, err = last
    raise RuntimeError(f"structured_output_failed kind={kind}: {err}")

Notes that matter in production:

• Do not retry schema_mismatch by default — log and page the owner
• Raise max_tokens once on truncation, then fail
• Record attempt and kind on the trace
• Include the raw model output (truncated) in error logs for debugging
• Emit a metric per failure kind so you can dashboard retriable vs design failures

The classify function is the most important piece. Without classification, every failure looks retriable and every retry looks reasonable. With classification, you can prove that 80% of your retries are wasted on schema mismatches — and fix the prompt instead of looping.

Failure classification in practice#

The last row is a circuit breaker. If three attempts produce the same failure, the problem is not sampling variance — it is your contract.

When to repair instead of resample#

A constrained repair pass can be cheaper than a full resample: "Return only valid JSON for schema S; keep field meanings." One repair max. If repair still fails, stop. Repair is not an infinite critic loop.

Repair works well for:

• Minor syntax issues (trailing comma, unclosed brace)
• Missing a single optional field
• Correct content in wrong nesting

Repair fails for:

• Fundamentally wrong content (model misunderstood the task)
• Business rule violations (amount exceeds limit)
• Hallucinated data (invented vendor name)

The repair prompt should include the invalid output and the validation errors — not re-run the entire task. "Here is the JSON you returned. Fix these three validation errors. Do not change any other fields." That is a repair. "Try again" is a resample. Know which one you are doing.

Cost and latency budgets#

Each retry multiplies p95. Policy ideas:

• Interactive UX: max 2 attempts total (initial + one retry)
• Async jobs: up to 3 with backoff
• Hard spend: abort when retry tokens would exceed remaining budget
• Wall-clock cap: abort if total elapsed exceeds user-facing SLA

If success needs retries more than ~5–10% of the time, you have a design problem, not a luck problem. Dashboard first_try_success_rate alongside overall_success_rate. A gap between them tells you how much retry is papering over.

Production scenario: a document tagging service runs at 50,000 documents per day. First-try success is 88%. With two retries, overall success reaches 97%. Sounds good — until you calculate that 12% retry rate means 6,000 extra model calls per day. At $0.003 per call, that is $18/day in retry tax — $6,500/year — on one feature. Fixing the prompt to raise first-try success to 95% saves more than half that cost and cuts p95 latency by thirty percent.

Prompt and schema hygiene that cuts retries#

• Put the schema in the prompt and enforce it in code
• Prefer provider structured-output modes when available — still validate
• Keep enums short and named in the prompt
• Avoid asking for prose and JSON in one blob
• Match max_tokens to expected output size with headroom, not unlimited
• Include one few-shot example of valid output in the prompt

Retries cannot compensate for an underspecified contract. If the model does not know reason_code must be one of five values, retrying three times produces three payloads with invalid reason codes. Fix the prompt. Add the enum list. Add a few-shot with a valid example. Then measure first-try success again.

Interaction with tools#

Never retry a side-effecting tool blindly because the structured plan looked weird. Retry the planning call. Tool retries need idempotency. Mixing the two is how you double-charge a card.

The correct sequence for an agent that plans then executes:

1. Generate structured plan → validate plan schema
2. If plan invalid and transient → retry plan generation (bounded)
3. If plan valid → execute tool with idempotency key
4. If tool fails with transient error → retry tool (bounded, same idempotency key)
5. If tool fails with design error → do not retry; escalate

Steps 2 and 4 are independent retry policies. Collapsing them into one loop is how agents double-execute actions.

Summary#

Retry structured LLM outputs only for transient, retriable failures — and bound those retries. Schema and prompt bugs need design fixes, not loops. Measure first-try success, classify errors, and keep side-effecting tools behind idempotent policies. Retries are a scalpel. Used as a hammer, they mostly produce cost.

Build the classifier first. Add the retry loop second. Dashboard first-try success from day one. When the dashboard tells you retries are masking a design bug, fix the design — and celebrate the retry rate dropping, not the overall success rate staying flat.