The Difference Between Orchestrated AI Workflows and Ad Hoc Scripts

The first version of every AI feature is a Python script. Someone wires OpenAI, adds a for loop, drops it on a cron schedule, and ships. The trouble starts when the script becomes production infrastructure without ever becoming a workflow — retries tacked on with sleep calls, state stuffed in a JSON file, "workflow" defined as main.py growing to 800 lines. There is a real inflection point where ad hoc scripts stop being pragmatic and start being operational debt.

What ad hoc scripts do well#

Scripts excel at exploration and narrow automation. One input file, one output, one owner who understands every line. Change the prompt, rerun locally, diff the CSV. No cluster, no workflow DSL, no deployment pipeline for graph versions.

For early AI features with low blast radius — internal report generation, batch labeling for eval sets, one-off data cleanup — scripts are the correct tool. They fail gracefully when requirements change weekly and nobody knows the final shape yet.

What orchestrated workflows add#

Orchestrated AI workflows express process structure explicitly: nodes, edges, timers, human tasks, compensation paths, and versioned definitions executed by a runtime that owns scheduling, persistence, and retry semantics.

The runtime might be Temporal, AWS Step Functions, Inngest, Windmill, or a disciplined in-house engine. The brand matters less than the properties: durable state, at-least-once delivery with idempotent handlers, visibility into in-flight instances, and separation between definition (the graph) and execution (workers).

The inflection point#

You have outgrown scripts when operators — not the author — must run the process reliably at scale. Concrete signals:

1. More than one person deploys or modifies the pipeline — tribal knowledge in main.py becomes a bus factor.
2. Failures require partial replay — "rerun from step 4" not "rerun everything and duplicate emails."
3. Humans are in the loop asynchronously — approvals, clarifications, external vendor callbacks.
4. SLAs exist — "complete within 48 hours" implies tracking age per instance, not cron success logs.
5. Side effects multiply — CRM updates, payments, ticket creation need idempotency and audit.
6. Compliance asks for proof — who approved, on which model output, before the write.

If three or more apply, start migrating. If all six apply and you still run a script, your on-call rotation is a coping mechanism.

Anatomy of script rot#

Script rot follows a pattern. Version one: 80 lines, clear. Version two: add retry decorator. Version three: write checkpoint to /tmp/progress.json. Version four: second script consumes the JSON. Version five: race condition when two crons overlap. Version six: "temporary" Redis lock. Version seven: engineer reimplements half of Temporal poorly.

# Script rot smell: workflow semantics in procedural clothing
import json
import time
from pathlib import Path

STATE_FILE = Path("/tmp/onboard_state.json")

def load_state():
    if STATE_FILE.exists():
        return json.loads(STATE_FILE.read_text())
    return {"step": 0, "records": []}

def save_state(state):
    STATE_FILE.write_text(json.dumps(state))

def run():
    state = load_state()
    if state["step"] == 0:
        state["records"] = fetch_intake()
        state["step"] = 1
        save_state(state)
    if state["step"] == 1:
        for rec in state["records"]:
            rec["summary"] = call_llm(rec)  # no idempotency key
        state["step"] = 2
        save_state(state)
    if state["step"] == 2:
        post_to_crm(state["records"])  # duplicate risk on retry
        STATE_FILE.unlink()

if __name__ == "__main__":
    for attempt in range(3):
        try:
            run()
            break
        except Exception:
            time.sleep(2 ** attempt)

Every line is understandable. None of it belongs in production orchestration for a customer-facing process.

What orchestration looks like for the same process#

The same onboarding flow as a workflow definition separates concerns: fetch intake (activity), summarize each record (model activity with idempotency key), validate (gate), wait for human approval (timer + signal), post to CRM (compensatable activity). The runtime tracks instance onboard-8842 at node await_approval since Monday.

Workers stay small. The graph is reviewable in PR. Operators query stuck instances without reading Python.

// Conceptual — not tied to one vendor SDK
const onboarding = defineWorkflow({
  id: "customer-onboarding",
  version: "3",
  steps: [
    { id: "fetch", type: "activity", handler: "fetchIntake" },
    { id: "summarize", type: "model", handler: "summarizeRecord", retry: { max: 3 } },
    { id: "validate", type: "gate", handler: "validateSummary" },
    { id: "approve", type: "human", timeout: "72h", escalate: "manager_queue" },
    { id: "crm_post", type: "activity", handler: "postCrm", idempotent: true },
  ],
});

The migration cost is real. The operating cost of not migrating is worse — it just arrives as outages instead of sprint points.

When scripts remain correct#

Keep scripts when:

• The process completes in one run, under minutes
• Failure blast radius is internal and low
• No human waits mid-process
• Throughput is batch-oriented with explicit "rerun whole batch" recovery
• The team is still discovering requirements weekly

Migrate when the script becomes infrastructure — referenced in runbooks, tied to revenue, or touched by teams who did not write it.

AI-specific pressure on scripts#

LLM steps add non-determinism that scripts handle poorly. A script author writes if not valid(json): retry() — but validity is not correctness. Orchestrated graphs let you insert validation nodes, alternate model paths, and human queues without nesting another four levels of indentation in main.py.

Token cost also accumulates across script retries that rerun the entire pipeline. Workflow engines retry at step granularity — re-run extraction without re-running expensive upstream OCR. Scripts hide that cost until finance asks why GPU spend doubled.

Tool choice is secondary to properties#

Teams debate Temporal vs Step Functions while their script loses state on deploy. Required properties first: durable instance ids, step-level retry and timeouts, human wait/signal support, idempotent activities, and a graph visible to non-authors. Pick the engine that fits your cloud and language. Do not pick "none" once the inflection point passed.

Summary#

Ad hoc scripts are the right starting point for AI automation with fuzzy requirements and low stakes. Orchestrated workflows become necessary when processes gain duration, humans, side effects, SLAs, and operators who were not in the room when main.py was written. Recognize the inflection point early — partial replay, async approval, compliance audit — and migrate deliberately.