From Stateless API Calls to Stateful AI Workflows

A stateless request arrives, does its work, and vanishes. No server remembers it unless something else writes that memory down. A stateful AI workflow is the opposite: it is the memory — carrying forward context, partial results, decisions, and open questions across steps, failures, deploys, and calendar time. Confusing the two is how teams end up stuffing Redis keys into a chat API and calling it orchestration.

Stateless by default#

Most HTTP APIs are stateless on purpose. Each request carries everything needed to produce a response: auth token, payload, idempotency key maybe. The server processes and returns. Scale horizontally, rotate pods, crash workers — clients retry with the same payload.

For LLM endpoints this pattern dominates: POST a prompt, get completion, disconnect. Stateless inference is simple to reason about and bill. It fits tasks where the entire problem fits in one context window and completes in one sitting.

Stateful when the process has a story#

A stateful AI workflow treats each business case as a narrative that unfolds over time. Step two needs step one's output — but also the decision step one made when confidence was borderline. Step five waits for a human who may respond Thursday. Step seven retries with a different retrieval query because step six's validation failed.

That accumulated history is workflow state: not just data, but where you are in the graph, what events have fired, what is still pending, and what must not happen twice.

Context is not the same as state#

Teams conflate "we pass chat history" with stateful workflows. Chat history in a session is ephemeral state owned by the client or a session store — it dies when the session ends or the window overflows. Workflow state is durable process memory with schema, access control, retention policy, and correlation to business entities (order id, claim id, employee id).

An AI-native workflow often assembles context for each model step from workflow state: prior step outputs, retrieved documents, human comments, policy version. Context is what the model sees this turn. State is what the system remembers forever (as per compliance policy).

from dataclasses import dataclass, field
from datetime import datetime
from typing import Any

@dataclass
class WorkflowState:
    instance_id: str
    definition_version: str
    current_node: str
    status: str  # running | waiting | completed | failed
    step_outputs: dict[str, Any] = field(default_factory=dict)
    decisions: list[dict[str, Any]] = field(default_factory=list)
    pending_events: list[str] = field(default_factory=list)
    updated_at: datetime = field(default_factory=datetime.utcnow)

def assemble_context(state: WorkflowState, node_id: str, max_tokens: int) -> str:
    """Build model context from durable state — not from whatever fits in RAM."""
    parts = [
        f"Workflow {state.instance_id} at node {node_id}",
        f"Prior outputs: {summarize(state.step_outputs, max_tokens // 2)}",
        f"Human decisions: {format_decisions(state.decisions)}",
    ]
    return truncate("\n".join(parts), max_tokens)

If your assemble_context function reads from a chat buffer instead of WorkflowState, a pod restart will wipe the conversation mid-workflow.

When to stay stateless#

It fits when:

• The user is actively waiting for a synchronous response
• No downstream system depends on partial progress
• Failure means "show error, user retries manually"
• Audit needs only request/response logs, not step trajectory
• Lifetime is measured in seconds, not days

Inline copilots, quick classifiers, and one-shot summarizers belong here. Fight the urge to add workflow state because "we might need it later."

When state becomes mandatory#

Reach for stateful workflows when:

• Time gaps exist — human approval, external system callbacks, SLA timers
• Multi-step dependencies — step B needs validated output from step A, possibly after retry
• Compensation — undo partial work if a later step fails
• Exactly-once side effects — idempotency keys tied to workflow instance and step
• Compliance — reconstruct who saw what model output before a decision posted

A loan underwriting assist that spans credit pull, document OCR, policy check, analyst review, and core banking post is inherently stateful. Calling the LLM five times from a stateless API route without instance persistence is a incident report waiting for a date.

Patterns for durable state#

Explicit workflow store. Table or document per instance: current node, payload, history, timestamps. Temporal, Step Functions, and homegrown job tables all implement variants. Pick one; do not scatter state across logs and Redis TTL keys.

Event sourcing lite. Append events (StepCompleted, HumanApproved, ValidationFailed) and derive current state. Helps audit and replay without storing opaque blobs only.

Optimistic concurrency. Workflow instances get version numbers; updates fail if two workers race. Essential when webhooks and timers can fire close together.

State vs context separation. Store full artifacts in object storage or JSON columns; pass summaries into model context. Never dump a 200-page extraction into every subsequent prompt.

Failure modes from pretending to be stateless#

Each pattern feels faster than adopting a workflow engine. Each becomes a rewrite project at scale.

Observability and operations#

Stateless systems monitor RED metrics: rate, errors, duration. Stateful workflows need instance dashboards: count by status, age of oldest waiting instance, stuck-node alerts, step failure heatmaps.

On-call for stateless: spike in 500s, roll back deploy. On-call for stateful: "847 instances waiting on human_review because webhook handler deployed broken — instances are not failed, they are paused." Different runbooks, different tools, different pager pain.

Security and retention#

Workflow state often holds PII, model outputs, and human decisions. Stateless request logs may redact aggressively and expire quickly. Workflow state may need seven-year retention with field-level redaction, legal hold, and access auditing.

Design retention at the workflow layer, not as an afterthought on log shipping. "Delete user" GDPR requests must reach instance state, not just chat transcripts.

Bridging the two worlds#

Many systems combine both: stateless LLM calls inside stateful workflow steps. The inference request remains ephemeral; the workflow persists inputs, outputs, and metadata around it.

interface ModelStepRecord {
  nodeId: string;
  model: string;
  promptVersion: string;
  inputHash: string;
  output: unknown;
  validation: "pass" | "fail" | "escalate";
  tokensUsed: number;
  completedAt: string;
}

interface WorkflowInstance {
  id: string;
  cursor: string;
  modelSteps: ModelStepRecord[];
  humanTasks: { assignee: string; status: string }[];
}

Each model call is stateless; the collection of ModelStepRecord entries is durable state. That separation keeps inference scaling simple while the process remains operable.

Testing stateful workflows#

Stateless endpoints test with request fixtures: given input X, expect output Y. Stateful workflows need scenario tests across time: start instance, simulate step completion, inject timer fire, deliver human signal, assert cursor and side effects. Property-based tests on state transitions catch bugs that single-shot API tests miss — especially double-fire on webhooks and lost wakeups after deploy.

Invest in a test harness that can drive signals into your workflow runtime. If testing requires manually editing database rows, your team will skip it until production teaches the lesson.

Capacity planning differs#

Stateless inference scales on request rate and tokens per second. Stateful workflows scale on open instance count — especially instances in waiting that consume metadata, timers, and support attention without using GPU. Plan storage and index capacity for instance queries. Plan human throughput for approval nodes. Stateless autoscaling patterns do not transfer cleanly.

Design checklist#

1. Assign every long-running AI process a workflow instance id on creation.
2. Persist cursor + status after every step transition, not only on completion.
3. Build context assembly from stored artifacts, not from live session memory.
4. Expose operator queries by instance id, business key, and current node.
5. Document retention and redaction before storing model outputs long-term.

Summary#

Stateless requests treat each LLM invocation as an isolated transaction. Stateful AI workflows treat each business case as a durable process that accumulates decisions, artifacts, and obligations over time. Context feeds the model; state feeds the organization. Mix them deliberately — stateless inference inside stateful orchestration — but never confuse a session id with a workflow instance when the process owns side effects, humans, or calendar time.