How to build custom AI workflows that improve output quality

A custom AI workflow is a fixed sequence of steps, written in code, in which language models handle the parts that need language understanding and ordinary software handles the rest: validation, routing, retrieval, retries and hand-offs to people. Workflows improve output quality because each step gets a narrow job, its own checks and its own evaluation.

That last part is what separates a workflow that improves quality from a chain of prompts that just adds latency. This guide covers when to decompose a task, the patterns that work, how to validate and verify each step, where specialized models fit, and how to run the workflow so it keeps getting better.

What is the difference between a workflow and an agent?

Anthropic’s “Building effective agents” draws the line clearly. Workflows are “systems where LLMs and tools are orchestrated through predefined code paths.” Agents are systems where LLMs “dynamically direct their own processes and tool usage.”

The practical difference is who decides the next step. In a workflow, your code does, so every run follows a path you designed, tested and can trace. In an agent, the model does, which buys flexibility on open-ended tasks at the price of predictability, cost and testability.

Most business processes (triaging tickets, extracting fields from contracts, drafting replies under a policy, generating reports from data) have known steps. They are workflow problems. The same post’s central advice applies: find the simplest solution that works, often a single well-prompted call, and add complexity “only when it demonstrably improves outcomes.”

Which workflow patterns should you know?

The post describes five patterns. They compose: a real system often routes first, then chains, then runs an evaluator loop on one step.

Pattern	What it does	Use it when
Prompt chaining	Each call works on the previous output, with checks between steps	The task splits cleanly into fixed subtasks
Routing	Classify the input, send it to a specialized handler	Categories differ and can be classified reliably
Parallelization	Run subtasks at once (sectioning) or the same task several times (voting)	Independent parts, or more attempts raise confidence
Orchestrator-workers	A model plans subtasks on the fly and delegates them	Subtasks cannot be predicted in advance
Evaluator-optimizer	One model generates, another critiques, in a loop	Criteria are clear and refinement measurably helps

Orchestrator-workers sits closest to an agent, since a model decides the subtasks. The other four keep control in your code.

How We Build Effective Agents: Barry Zhang, AnthropicBarry Zhang, Anthropic · AI Engineer · YouTube · 2025

When should you decompose a task?

Splitting a task is not free. Each step adds a model call (latency and cost), an interface that can break, and another component to evaluate. Decompose when at least one of these is true:

• The subtasks need different context or instructions. Classifying intent needs the taxonomy; drafting needs the policy documents. One prompt that carries both is longer, costlier and easier to confuse.
• An intermediate result can be checked. If you can validate extracted fields before using them, an error stops at the step that made it instead of spreading into the final output.
• One part keeps failing. Error analysis shows the model handles most of the task but gets one part wrong, such as the calculation or the citation. Isolate that part and give it a dedicated step, a tool or deterministic code.
• Different parts justify different models. A cheap model can classify; a stronger one can reason about edge cases.

Do not decompose when you cannot evaluate the steps separately, or when one prompt already meets the bar. Start with the single-call baseline, measure it, and add steps only where error analysis points. The comparison of prompting, RAG and fine-tuning covers how to decide what to change first.

How do you make each step’s output trustworthy?

Structured outputs

Make every step that feeds another step return structured data, not free text. Two layers help:

• Constrained decoding makes the model’s output follow a JSON Schema. OpenAI’s Structured Outputs constrains completed, non-refusal output to your schema; for self-hosted open-weight models, vLLM’s structured outputs support JSON Schema, regex, fixed choices and grammars.
• Validation in code. Parse every output with a typed model, for example Pydantic’s model_validate_json, and check business rules the schema cannot express: totals that must add up, dates in range, IDs that must exist. OpenAI’s own guide tells you to handle refusals and incomplete responses and to validate business rules separately.

Schema-valid is not the same as correct. A perfectly shaped JSON object can hold the wrong category. Validation catches malformed output; verification and evaluation catch wrong output.

Retrieval as its own step

When a step needs facts (policies, product data, account history), retrieve them in an explicit step and pass them in with stable IDs. That makes the step’s inputs visible in the trace, lets the verifier check claims against the same context, and lets you tell retrieval failures (the right document never arrived) from generation failures (the model ignored it). Evaluate retrieval separately, for example with recall of the documents a human marked as needed.

Tools and deterministic code

If a step can be done in code, do it in code. Arithmetic, date math, lookups, format conversion and permission checks are cheaper and fully reliable as functions. Where the model must decide when to call something, give it a tool with a narrow, well-documented interface, validate the arguments before executing, and never let a model-generated argument reach a write operation without a check.

How do verification loops and self-consistency improve quality?

Evaluator-optimizer loops

A verifier step reviews the output against explicit criteria (is every claim supported by the context, does the reply follow the policy) and either passes it or returns specific problems for one retry. Keep these loops tight:

• Cap the retries, usually at one or two. If a draft fails twice, the input is probably unusual, and a person should see it.
• Feed back specific problems, not “try again.” The verifier’s list of unsupported claims is the useful signal.
• Use a different model, or at least a different prompt, as verifier. LLM judges have documented biases, including self-enhancement bias, a tendency to favor answers from their own model family; a model checking its own output also tends to share its blind spots.
• Run deterministic checks first. Cited IDs that do not exist, figures absent from the source, missing required fields: catch these in code before paying for a model-based check.
• Calibrate the verifier against human labels, like any judge. The evaluation framework guide covers how.

Self-consistency

Wang et al. (2022) proposed sampling several reasoning paths and taking the most common final answer instead of trusting one greedy output. Over standard chain-of-thought prompting, the paper reports accuracy gains of 17.9% on GSM8K, 11.0% on SVAMP and 12.2% on AQuA. It is the “voting” form of parallelization.

It fits steps whose answers can be compared exactly, such as a label, a number or a yes/no, and it doubles as an uncertainty signal: if three samples disagree, route the case to a person. The cost is linear in the number of samples, so use it on cheap, decisive steps such as classification rather than on long generations.

When should a step use a specialized tuned model?

Routing is not only about input categories. It is also about matching each step to the cheapest model that meets its quality bar. Research on cascades and routers shows the potential: FrugalGPT reported up to 98% lower cost while matching the best single model’s accuracy in its experiments, and RouteLLM cut costs by over 2 times in certain cases without lowering response quality. Those are best cases on specific benchmarks; your gain depends on your traffic.

Workflows make specialization practical, because each step is narrow and has its own eval. A step is a good candidate for a smaller fine-tuned model when:

• it runs at high volume, so per-call savings add up;
• the job is narrow and stable, such as classification, extraction or drafting in one format;
• a prompted model misses the bar or costs too much on it; and
• you have a step-level eval and enough reviewed examples to train and test on.

Low-volume or open-ended steps usually stay on a general model, and many steps need no model at all. Prove every swap on the step’s held-out eval and the end-to-end eval before shipping; measuring whether fine-tuning improved your model explains how. Attempts does this kind of step-level work: tuning an open-weight model for one high-volume step and measuring it against the step’s current prompt and model, or recommending against training when a better prompt is enough.

What does a custom AI workflow look like end to end?

Here is an example workflow for drafting replies to billing support tickets. It combines routing, retrieval, a tuned drafting model, an evaluator-optimizer loop and human review.

1. Classify ×3. A small model labels the intent three times; the majority wins. No majority, or a “needs human” flag, sends the ticket to review.
2. Retrieve the policy excerpts for that intent, each with a document ID.
3. Draft with the model assigned to the intent, here a small tuned model for the high-volume billing intent, returning the reply and the IDs it relied on.
4. Verify in two layers: code checks that cited IDs were actually retrieved, then a separate model checks that every claim is supported by the context. On failure, retry once with the verifier’s specific problems.
5. Send or escalate. Supported drafts go out (or to an agent for approval); repeated failures and budget overruns go to a person.

The sketch below shows the control flow. call_model, retrieve and log_span stand in for your model client, search index and tracer; model names are placeholders.

"""Sketch: classify -> route -> retrieve -> draft -> verify -> (retry | human review).

`call_model(model, prompt, schema=None)` returns (text, usage) and stands in for your client.
`retrieve(query, k)` and `log_span(**fields)` stand in for your search index and tracer.
"""
import json
import time
from collections import Counter
from typing import Literal

from pydantic import BaseModel, ValidationError

MAX_ATTEMPTS = 2
BUDGET_USD = 0.03            # hard per-request budget across all steps, retries included


class Triage(BaseModel):
    intent: Literal["billing", "technical", "account", "other"]
    needs_human: bool


class Draft(BaseModel):
    reply: str
    cited_doc_ids: list[str]


class Verdict(BaseModel):
    supported: bool
    problems: list[str]


ROUTES = {                   # the high-volume billing intent goes to a small tuned model
    "billing": "billing-tuned-small",
    "technical": "general-large",
    "account": "general-large",
}


class BudgetExceeded(Exception):
    pass


class Budget:
    def __init__(self, limit: float):
        self.limit, self.spent = limit, 0.0

    def charge(self, usd: float) -> None:
        self.spent += usd
        if self.spent > self.limit:
            raise BudgetExceeded(f"budget exceeded: ${self.spent:.4f}")


def step(name, model, prompt, schema, budget, request_id):
    """One traced, validated, budgeted model call."""
    start = time.monotonic()
    text, usage = call_model(model, prompt, schema=schema.model_json_schema())
    budget.charge(usage["cost_usd"])
    try:
        parsed, error = schema.model_validate_json(text), None
    except ValidationError as exc:
        parsed, error = None, str(exc)
    log_span(request_id=request_id, step=name, model=model, ok=parsed is not None,
             error=error, latency_s=time.monotonic() - start, **usage)
    return parsed


def classify(ticket, budget, request_id, votes=3):
    """Self-consistency for the routing decision: sample several labels, take the majority.

    The votes only differ if the router samples at temperature > 0.
    """
    labels = []
    for _ in range(votes):
        t = step("classify", "router-small", f"Classify this ticket:\n{ticket}", Triage,
                 budget, request_id)
        if t is not None:
            labels.append((t.intent, t.needs_human))
    if not labels:
        return None
    (intent, needs_human), count = Counter(labels).most_common(1)[0]
    return Triage(intent=intent, needs_human=needs_human or count * 2 <= votes)  # no majority -> human


def handle(ticket: str, request_id: str) -> dict:
    budget = Budget(BUDGET_USD)
    try:
        triage = classify(ticket, budget, request_id)
        if triage is None or triage.needs_human or triage.intent not in ROUTES:
            return {"status": "human_review", "reason": "triage"}
        docs = retrieve(ticket, k=5)
        context = "\n\n".join(f"[{d['id']}] {d['text']}" for d in docs)
        feedback = ""
        for _ in range(MAX_ATTEMPTS):
            draft = step("draft", ROUTES[triage.intent],
                         f"Policy excerpts:\n{context}\n\nTicket:\n{ticket}\n{feedback}",
                         Draft, budget, request_id)
            if draft is None:
                feedback = "Return valid JSON that matches the schema."
                continue
            if not set(draft.cited_doc_ids) <= {d["id"] for d in docs}:
                feedback = "Cite only the document ids provided."
                continue
            verdict = step("verify", "verifier-large",
                           f"Context:\n{context}\n\nReply:\n{draft.reply}\n"
                           "Is every claim supported by the context?", Verdict,
                           budget, request_id)
            if verdict is not None and verdict.supported:
                return {"status": "ok", "reply": draft.reply, "cost_usd": budget.spent}
            problems = verdict.problems if verdict else ["verifier output invalid"]
            feedback = "Fix these problems:\n" + json.dumps(problems)
        return {"status": "human_review", "reason": "verification_failed"}
    except BudgetExceeded as exc:
        return {"status": "human_review", "reason": str(exc)}

What the sketch gets right: every model call goes through one step function that validates, traces and charges the budget, so nothing bypasses those controls; every validation, verification and budget failure ends in an explicit status rather than a silently bad reply (provider timeouts and outages are left to your client’s retry and fallback logic, covered below); and the routing table makes “which model handles which step” a reviewable configuration change.

How do you observe and evaluate each step?

You cannot improve a workflow you cannot see. Log one trace per request with a span per step, recording the model and prompt version, inputs and outputs (with sensitive data handled under your data policies), validation result, token usage, cost and latency. The OpenTelemetry project maintains semantic conventions for generative AI covering spans, metrics and events, which keeps traces portable across tools.

Then evaluate at two levels:

• Per-step evals answer “which step is failing?” Classification accuracy for the router, recall for retrieval, faithfulness for the draft, agreement with human labels for the verifier.
• End-to-end evals answer “is the output good?” and catch interactions that step evals miss, such as a router change that sends harder tickets to the small model.

Two habits from Anthropic’s guidance on agent evals apply to workflows too. Read the transcripts, because that is how you learn whether a failure was the system’s or the grader’s. And when consistency matters to users, measure it: pass^k, the probability that all k trials succeed, is stricter than pass@k, the probability that at least one does.

How do you budget cost and latency per step?

Treat cost and latency as per-step budgets, not end-of-month surprises. For each step, estimate:

cost per request = calls per request × (input tokens × input price + output tokens × output price)

and add the expected retries, weighted by how often they happen.

Example: assume a small tuned model priced at $0.20 per million input tokens and $0.60 per million output tokens, and a large model at $3 and $15 (hypothetical prices for illustration).

Step	Model	Tokens per call (in / out)	Cost per request
Classify ×3	small	800 / 20	$0.00052
Draft	small tuned	3,000 / 300	$0.00078
Verify	large	3,300 / 100	$0.01140
Total, no retry			$0.0127

With a retry on 15% of requests, the expected cost rises to about $0.0145. The lesson is in the shape, not the numbers: the verifier is about 90% of the cost. The fixes are to run deterministic checks first and call the model verifier only when they pass, verify a sample instead of every draft on low-risk intents, or move the verifier to a smaller model calibrated against human labels. Each of those is a quality trade-off to measure, not assume.

Latency adds up the same way along the critical path. Run independent calls in parallel (the three classification votes, or retrieval alongside classification when the index does not depend on intent), stream the final step if a user is waiting, and set a timeout per step. The guide to reducing LLM costs covers the wider set of levers.

How should a workflow handle failures and escalations?

Design for each failure mode

Every step can fail in several ways: the provider times out or rate-limits, the output is malformed, the output is well-formed but wrong, or retrieval returns nothing relevant. Design for each:

• Timeouts and retries with backoff for transient provider errors, with an idempotency key on anything that writes.
• A fallback model for provider outages, evaluated in advance so the fallback is known to meet the bar.
• Validation failures retry once with the error message, then escalate. Unlimited retries burn budget on inputs the model cannot handle.
• Empty or low-relevance retrieval should produce an “I can’t answer from the policy” path, not a confident guess.
• A hard per-request budget, as in the sketch, so a pathological input cannot loop.
• Explicit terminal states (“ok”, “human_review” with a reason) that you can count on a dashboard. A rising escalation rate is often the first sign of drift.

Where do humans belong in the loop?

Human checkpoints are a design element, not a fallback for broken automation. Put them where an error is expensive or irreversible:

• Before irreversible actions: refunds above a threshold, emails to customers in regulated contexts, database writes.
• On low-confidence cases: no majority in a vote, verifier failure after retry, out-of-scope intent, budget exceeded.
• On a random sample of confident cases, to measure the error rate of the automated path rather than assuming it.

Make the reviewer’s job easy: show the input, the retrieved context, the draft and the verifier’s notes, and record the decision and any edit in a structured form. Those records become labelled data.

How do you turn production feedback into better models?

A workflow in production generates exactly the data you need to improve it. Capture it deliberately:

1. Log every trace with its terminal status.
2. Collect signals: human-review decisions and edits, verifier failures, user corrections, escalations, and a random sample of “ok” outputs reviewed for errors.
3. Triage failures by step, using the traces, so a retrieval miss is not mistaken for a drafting error.
4. Add reviewed cases to the eval sets first, especially the regression slice, so each fixed failure stays fixed.
5. Add reviewed, corrected outputs to training data for the steps that use tuned models, keeping eval items out of training.
6. Retrain or re-prompt, then re-evaluate each step and the whole pipeline against the current production version before shipping.

This loop, often called a data flywheel, is where workflows pay off over time: each step’s narrow scope makes its data easy to label, and each round of reviewed failures sharpens both the evals and the models. Techniques such as distillation fit naturally here, using reviewed outputs from a large model’s step to train a smaller one where the large model’s terms of use allow it, as long as the result is measured against the step it replaces.

The right starting point is the single best prompt and an eval. Find where it fails. Add the one step, check or tool that addresses the biggest failure, and measure again. Add tracing and per-step budgets before the second step, not after the tenth. Specialize a step with a tuned model only when it is high-volume, narrow and measured. A workflow built this way stays simple where it can and gets more capable only where the evidence says it should.