Reliable Pipelines and Validation

A contract-extraction pipeline must handle model errors alongside ordinary software failures: malformed records, missing evidence, stale inventory, and incorrect routing. This lesson makes those failure paths explicit so that plausible output does not become an accepted business record without checks.

The first reliability rule is that schema adherence is not correctness. A field value can be well typed and still cite the wrong clause, represent the wrong room category, or violate an accessibility requirement. The rest of the pipeline exists to keep those different failure classes visible.

Building on Post 01's component boundaries, consider a pipeline that extracts fields from a hotel contract, normalizes currencies and dates, checks availability constraints, and produces a comparison sheet. Reliability depends on defining each stage, the operations it may perform, and the checks that determine what happens next.

This is where control logic enters. In a reliable LLM application, the model is one component inside a larger workflow. The workflow decides when to call the model, what inputs to pass, what output shape to require, which validators must pass, when to retry or stop, and when to route a record to human review or an exception queue. Stage decomposition makes failure paths inspectable, but reliability still depends on sound checks and end-to-end evaluation.

The fictional OptiVerse Travel example ingests a hotel contract, extracts rates, normalizes values, and combines the result with separate date-specific inventory evidence. Contract IDs, rates, dates, and budgets are teaching fixtures, and a contracted room allocation is not a live availability confirmation.

{
  "hotel": "KYO-H12",
  "accessible": false,
  "rateUSD": 240
}

A Reliable Pipeline Has Stages and Decision Boundaries

Defining explicit stages and decision boundaries makes control logic concrete. For the OptiVerse Travel copilot, consider this task: ingest hotel partner contract KYO-H12 and produce a rate-and-availability comparison for a 10-day accessible Japan trip (booking reference JPN-2026-0417).

That task can be decomposed into six stages.

1. Ingestion

The system receives the contract PDF along with basic metadata such as source, upload timestamp, and document identifier. Ingestion makes the document available to later stages in a controlled way rather than interpreting its contents.

Typical outputs:

• document ID (e.g., KYO-H12)

• source metadata (partner name, contract effective dates)

• file type and checksum

• processing status

Decision boundary:

If the file is unreadable or missing required metadata, stop and report the missing input. When a duplicate arrives, reuse a prior result only if the content, access scope, and relevant processing versions match; otherwise reprocess deliberately, since a duplicate submission is not automatically corrupt.

2. Extraction

The system extracts the seasonal rate table, room-type descriptions, accessibility annotations, meal-plan options, and cancellation terms from the contract. Some of this work may use deterministic parsers; some may need model assistance when the layout is irregular or the extraction target is semantically defined.

Typical outputs:

• raw extracted text blocks

• table candidates (rate grids, room inventories)

• clause references (e.g., Clause 4.3: seasonal rate table with accessible room allocation)

• candidate fields such as room types, nightly rates, seasonal date ranges, meal-plan inclusions, and accessible room counts

Decision boundary:

The system checks whether extraction produced the minimum evidence the task needs. If the contract has no recoverable rate table or its accessibility sections are missing, the workflow should not silently continue as if the record were complete.

3. Normalization

Raw extractions are not yet system-ready: rates may appear in different currencies, date ranges may use inconsistent formats, and room-type labels may use aliases that differ across partner contracts. Normalization converts extracted values into the canonical forms the rest of the system expects.

Typical outputs:

• normalized currency (preserving the original amount and currency alongside the USD estimate, dated exchange-rate source, conversion direction, and rounding rule)

• canonical date format (ISO 8601)

• standardized room-type names

• explicit nulls for unresolved fields

Decision boundary:

Normalization must be able to say "unresolved" rather than force a guess. A footnote and a rate grid can legitimately list different prices for different room categories, dates, tax treatments, or discount conditions. If two figures conflict under the same applicable terms, surface the conflict instead of treating it as a formatting difference.

4. Validation

Validation checks whether a normalized record is acceptable for downstream use by combining schema constraints, evidence comparisons, and business rules. If confidence scores influence routing, calibrate the chosen threshold on labeled examples; a model-reported confidence value is not automatically a probability of correctness.

Typical checks:

• required fields exist (room type, nightly rate, seasonal dates, accessibility flag)

• numeric ranges follow an explicit anomaly policy (in this fixture, $5 or $50,000 triggers review rather than proving an error outright)

• currencies match the canonical schema

• provenance links point back to a specific clause, table cell, or contract section

• contradictory values are flagged (e.g., availability_confirmed=true paired with available_accessible_room_count=0 on the same dates and inventory snapshot; an accessibility feature flag alone does not assert availability)

Decision boundary:

Only validated records advance to comparison. Records that fail route to retry, repair, or human review according to failure class.

5. Comparison and Synthesis

With a cleaned, validated evidence packet in hand, the synthesis model performs the work it handles well: composing a concise rate-and-availability comparison. The synthesis model should not be forced to recover from upstream ambiguity that the pipeline already detected.

Typical outputs:

• rate comparison table across room types and seasons matching the requested travel dates

• accessible-room availability status backed by an independent inventory source for the requested nights, or unknown when unconfirmed

• hotel-cost subtotal, assessing the full $13,000 to $15,000 family budget only when transport, meals, taxes, fees, and other required expenses are also present

• references back to the extracted evidence packet

Decision boundary:

If producing the comparison would require inventing availability or pricing assumptions, abstain or return a partial summary with explicit gaps.

6. Final Formatting and Delivery

Delivery prepares the output for its target channel: consultant dashboard, client proposal, or booking service. Formatting is not cosmetic: it determines which metadata, warnings, and provenance links travel with the output.

Typical outputs:

• rendered comparison sheet

• linked evidence (contract clauses, rate table sources)

• processing log

• final status

Decision boundary:

When a destination channel requires provenance links or an approval status, enforce that contract at delivery rather than assuming an upstream step already satisfied it.

Why Prompt-Only Orchestration Breaks Down

Prompt-only orchestration pushes more workflow responsibility into one model call instead of designing explicit stages around the model. It often sounds efficient: ask the model to read the contract, extract the rates, normalize currencies, decide whether room availability is sufficient, call tools if needed, and write the comparison in one pass.

That can work for a demo, but it is fragile in a production system.

Hallucination is only part of the problem. Downstream requirements are usually sharper than natural language: a parser expects a field to exist, a routing step expects one of a small set of statuses, a validator expects currencies normalized to one standard, a review queue expects confidence and provenance metadata, and a compliance or audit process expects the system to show where a rate or availability figure came from. Prompt-only orchestration blurs those boundaries.

Once boundaries blur, failures become harder to see and harder to contain. A model may return valid-looking JSON with the wrong currency conversion, produce a plausible room-type label that does not match the agency's canonical naming, omit a required field and compensate with fluent prose, or combine extraction and interpretation so tightly that the system cannot tell whether the error came from contract parsing, normalization, or synthesis.

Stronger prompting is not the same thing as better architecture. Prompt quality matters, but it does not replace explicit workflow design. When a task has multiple transformations with different correctness criteria, the safer pattern is to separate those transformations and check each one.

Control Logic Is the Reliability Layer

Control logic is the set of rules that decides what happens next. In a traditional software pipeline, those rules are written directly in code. In an LLM system, some decisions may still be deterministic and some may be delegated to a model, but the application still owns the workflow.

That ownership matters. Tool use does not remove control logic from the application. A model can propose a tool call or produce structured arguments for one, but the application decides which tools are exposed, what schema the arguments must follow, whether the arguments are acceptable, whether the tool result should trigger another step, and what happens when the result is missing or inconsistent.

In practice, control logic answers questions like these:

• Did contract parsing recover enough structure to proceed?

• Which extraction path matches this document layout?

• Are all required fields populated?

• Did currency normalization complete without ambiguity?

• Is this record verified enough to enter comparison?

• Should the pipeline retry, abstain, or escalate to human review?

These are normal systems questions. The model can help answer some of them, but it does not erase them.

Structured Outputs Help, but They Do Not Eliminate Validation

Many teams improve reliability by requiring the model to return structured output that conforms to a schema. It is a good pattern: it narrows the output shape, makes downstream parsing more robust, and forces the application to define the fields it actually needs.

Three Output Modes

Keep three distinct mechanisms separate instead of lumping them together as "the model returned JSON":

1. Prompted JSON asks for JSON in natural language without structural enforcement.

2. JSON mode constrains output to valid JSON syntax but does not enforce your schema.

3. Strict schema-constrained output enforces a supported schema's shape during decoding. Under OpenAI's current strict Structured Outputs, shape adherence is reliable for supported schemas, but semantic errors remain possible, and applications must still handle refusals, content filtering, and incomplete responses (OpenAI, 2026).

Design against the exact capability and failure contract of your deployed provider rather than attributing strict constrained-decoding guarantees to best-effort JSON prompts.

Even strict schema adherence stops at structure.

A schema-valid record can still be semantically wrong: a required field can hold the wrong value, a plausible number can come from an adjacent rate entry, a permitted label can be assigned to the wrong room category, or a field can be marked complete when the evidence was weak or ambiguous.

Schemas and validators answer different questions:

• A schema asks: "Does the output match the required shape?"

• A validator asks: "Is the output acceptable for this workflow?"

Those questions are not interchangeable.

Suppose a record defines room_type, nightly_rate_usd, accessibility_flag, and source_clause. A schema can require a numeric rate, a non-empty clause reference, and a room_type drawn from a fixed enum. It cannot by itself establish that the cited clause supports the rate, that the currency conversion is correct, or that the source room was mapped to the correct canonical category. A permitted label can still be the wrong label for this record.

In production, validation usually combines three layers:

• Structural validation verifies types, required keys, enums, and shape.

• Semantic validation verifies plausibility, cross-field consistency, and source alignment.

• Business-rule validation verifies compliance with organizational policy and trip constraints.

That last layer matters more than many prototype builders expect. A travel-agency workflow may require every extracted rate to link to a specific contract clause and date range, not just to free text. That requirement is a workflow contract, not a language problem.

Stage Contracts and Tool Handoff Boundaries Need To Be Explicit

Reliability also depends on explicit handoff contracts between model steps and application code. Whenever a model can propose tool calls, tool schemas become part of your reliability boundary.

A well-designed tool schema constrains what the model can request and makes the application-side contract testable. Tools such as normalize_currency, lookup_room_type_alias, or fetch_prior_bookings should each expose a narrow, typed interface with clear failure behavior.

For example, a currency-normalization tool should not accept vague free-form instructions when the application needs exact conversions. It should accept a defined amount, source and target currency codes, a dated exchange-rate source or explicit rate, and a rounding policy. It should preserve the original value and return either a traceable normalized result or an explicit error state. If the model passes malformed arguments, the application should reject the call rather than guess.

This boundary is where many teams discover that "the model can call tools" is not a design in itself. The design lives in the schemas, the argument checks, the retry policy, the fallback path, and the decision about whether a tool failure is recoverable or terminal.

This post covers the contracts between pipeline stages; Post 06 covers the runtime tool loop, retries, idempotency, and reconciliation. Function calling remains a multi-step exchange in which the application receives a proposed call, executes code, and returns the result to the model (OpenAI, 2026).

The same logic applies to handoffs between stages. Each stage should produce an output contract that the next stage can inspect without reinterpreting the whole history. That reduces hidden state and keeps failures local. If the comparison step has to read a long conversational trace to infer whether normalization succeeded, the system has already lost important control.

Choose Deterministic Steps for Defined Rules

It is tempting to make the model responsible for every transformation because that seems more flexible. In practice, many high-value workflow steps are better handled deterministically once the problem is understood.

Use deterministic logic when:

• the transformation follows an explicit rule

• valid outputs come from a narrow set

• error conditions are unambiguous

• reproducibility outweighs open-ended flexibility

Common deterministic steps include file-type routing, currency conversion, ISO date formatting, enum alias mapping, duplicate detection, threshold checks, and channel-specific rendering.

Use model-directed steps when:

• inputs are irregular or weakly structured

• accuracy depends on reading context and nuance

• valid outputs cannot be fully enumerated up front

• synthesis or compression adds clear value

Common model-assisted steps include extracting rate tables from irregular contract layouts, linking footnotes to surrounding clauses, and drafting a readable comparison from a validated evidence packet.

This distinction is not ideological; it is operational. Overusing the model adds avoidable latency, cost, and failure surface, while underusing it can make messy tasks brittle. A reliable architecture allocates each step to the mechanism that fits it.

Worked Example: A Contract-Ingestion Workflow

To see these boundaries in action, suppose the OptiVerse Travel team uploads partner contract KYO-H12 to prepare a rate comparison for booking reference JPN-2026-0417, a 10-day accessible Japan itinerary during cherry blossom season for a family of four including a wheelchair user, budgeted at $13,000 to $15,000.

A fragile prototype sends the raw contract text and a sweeping prompt to one model call:

"Read this contract, extract the seasonal rates and accessible room availability, convert currencies to USD, identify the most relevant options for the requested dates, and write a comparison summary."

If the output looks reasonable, the prototype appears successful, but the workflow has hidden too many decisions in one place:

• Did extraction miss the rate table in Clause 4.3?

• Did yen-to-dollar conversion use the right rate and rounding rule?

• Did the model conflate standard room allocations with wheelchair-accessible allocations?

• Did it overlook blackout dates or minimum-stay rules?

• Can anyone inspect what was actually used?

A staged pipeline makes every handoff inspectable.

First, the system parses the contract PDF and extracts candidate tables, clauses, and sections. Next, it runs field extraction for a bounded set of targets: room type, nightly rate, currency, seasonal date range, accessible room count, meal-plan options, and source clause references. Then it normalizes currencies and date formats and validates the record's structure, semantics, and business rules. Only after that does it ask the model to produce the comparison sheet, using the validated record as the source package.

This version makes the failure boundaries visible. If a rate entry is missing provenance, the workflow can stop before comparison. If normalization finds conflicting currencies or date ranges, the record can be flagged for review. If the comparison step fails, the structured extraction remains available for inspection or reprocessing.

That is what reliability looks like in an LLM pipeline: not perfect outputs, but bounded failure and legible state.

Hands-On Exercise

Define a stage envelope with record_id, source_version, stage, status, payload, and errors. Then write tests for this candidate extraction:

{
  "record_id": "KYO-H12-rate-04",
  "source_version": "contract-v3",
  "room_type": "accessible_twin",
  "nightly_rate_usd": 240,
  "accessibility_flag": true,
  "source_clause": "KYO-H12#4.3"
}

Treat contract-v3 as a symbolic fixture version, not a SHA-256 digest. In this exercise's fictional source table, clause KYO-H12#4.3 states room_type=accessible_twin, nightly_rate_usd=280, accessibility_flag=true, and an inclusive valid-night interval from 2026-04-01 through 2026-04-10. These USD values are already normalized; no real exchange rate is implied. The candidate above is deliberately wrong: its corrected rate is 280, and even with that correction, a request for the night of 2026-04-11 still fails the date rule. This validation fixture is separate from Post 03's corpus (fixture-v1) and the connected lab, where KYO-H12 has a 240 USD rate. Keep their dataset and version identities separate, and do not merge examples solely because a property ID matches.

Create five fixtures:

• malformed JSON from best-effort prompting;

• a provider refusal;

• an incomplete response caused by an output limit;

• a schema-valid record whose source clause states 280, not 240;

• a semantically correct rate that violates the requested travel-date business rule.

Your tests must report five distinct failure classes. A generic model_error result does not pass.

Deliberately Broken Case

First define the fields: accessibility_flag describes the room type, available_accessible_room_count is date-specific inventory, and availability_confirmed means a matching room is currently confirmed available for the requested interval. An accessible room type with zero available rooms is valid data, not a contradiction; the business rule must still withhold any availability claim.

For the broken fixture, explicitly allow all three fields in the schema, set available_accessible_room_count=0, and set availability_confirmed=true for the same property, room type, interval, and inventory snapshot. A shape-only schema can accept these values; the semantic cross-field check must reject them. Availability is also not a completed reservation.

The repair is not a stricter prompt. Add a cross-field invariant, preserve the source cells that produced both values, and route the record to review or re-extraction.

Common Operational Failure Modes

Production failures usually come from mundane mismatches between stage assumptions.

One common failure in prompted JSON or non-strict modes is malformed structured output. Strict constrained output changes that failure surface, but the application still needs explicit handling for refusals, filtering, incomplete generation, unsupported schemas, and semantically wrong values.

Another is semantic drift inside valid structure. A field may be present and well typed but still represent the wrong thing. These errors are easy to miss because the pipeline appears healthy.

A third is silent corruption from partially valid records. If the system lets comparison proceed when one critical field failed normalization, the final answer may read as confident even though the evidence packet was incomplete.

Routing errors matter too. A workflow may send a low-confidence extraction to the normal comparison path instead of a repair queue. That kind of control-logic mistake can look like a model problem when the real issue is stage policy.

Repeated model calls can add cost and latency. For tasks with a suitable deterministic implementation, compare that baseline with the model-based approach and require evidence that any extra calls improve the relevant outcome. Repetition alone does not establish a reliability gain.

Design exception paths alongside the happy path. An extraction pipeline may encounter unreadable files, conflicting fields, unsupported layouts, or ambiguous contract language. Define how to detect, reject, retry, or escalate those cases before relying on the output downstream.

Practical Design Guidance

Start with the reliability target, not the feature list. Ask what must be correct enough for the workflow to be useful and what kinds of failure are acceptable.

Decompose tasks by transformation type. Extraction, normalization, validation, comparison, and delivery usually have different error modes and should not be collapsed without a reason.

Define output contracts for every stage. A stage should produce something the next stage can inspect directly, including failure states.

Use schemas for model outputs, but treat them as input to validation rather than the final guarantee.

Keep tool interfaces narrow. If a tool matters operationally, give it a typed schema, explicit error behavior, and application-side checks.

Design exception paths early. Decide when to retry, when to abstain, when to surface uncertainty, and when to hand off to a human reviewer.

Measure where cost and latency actually go. Reliable systems often improve by removing unnecessary model calls, not by adding larger ones.

Above all, preserve evidence across handoffs. A polished comparison sheet should always be traceable back to the record that produced it.

Exit Criteria

Your artifact is complete only if:

• every stage has typed success and failure outputs;

• structural, semantic, and business validation are separate tests;

• refusals and incomplete responses never enter comparison as records;

• the schema-valid contradiction is rejected before synthesis;

• every accepted field is traceable to a source version and location;

• a failed stage can resume or route to review without reconstructing state from prose.

Knowledge Check

Answer without looking back.

1. What does strict schema adherence guarantee, and what does it not guarantee?

2. Why must refusal and incomplete output be separate from invalid structure?

3. Is accessibility_flag=true with zero available rooms contradictory? What if availability_confirmed=true for the same inventory snapshot?

4. Why does Post 06, rather than this post, own retry and idempotency mechanics?

Answer Key

1. It guarantees adherence to the supported output shape, without guaranteeing that field values are true, source-supported, mutually consistent, or compliant with business rules.

2. They represent distinct provider or completion states that require different recovery paths and user-facing behavior.

3. The first combination can describe a sold-out accessible room type and should pass semantic validation. The second, availability_confirmed=true on the same zero-inventory snapshot, contradicts the defined field meanings and requires a semantic rejection; business policy separately blocks an unsupported availability claim.

4. This post defines stage contracts; Post 06 covers runtime execution and recovery inside the model-tool loop.

Expected fixture classes are invalid_structure, refused, incomplete, unsupported_value, and business_rule_failed. Check provider completion and refusal status before accepting a parsed record, then compare field values against the source clause and the requested dates. Do not concatenate partial JSON into a success record or retry a refusal as if it were a parsing error.

What This Post Is Not

This is not an argument that every AI system should be a rigid deterministic pipeline. Some tasks do require more open-ended model control, and later posts in this series cover retrieval, memory, and agent loops.

Nor is it an argument that structured outputs solve reliability by themselves. They improve one boundary; they do not replace validation, monitoring, or workflow design.

Before asking whether the system needs an agent, check whether its pipeline has clear stages, explicit schemas, validation rules, and controlled handoffs. Then evaluate the additional choices and failure modes that model-directed control introduces. Autonomy is a design choice whose benefit must be demonstrated for the task.

The Bridge to Retrieval and Grounding

A well-structured pipeline still has one major limitation: it can only be as informed as the evidence packet it receives. If the system must answer questions that depend on external partner contracts, internal booking histories, accessibility databases, or prior client itineraries, reliability also depends on how the right evidence is found and attached to the workflow.

That is where retrieval enters. The retrieval tutorial moves from internal pipeline control to grounding: how a system retrieves the right context, preserves provenance, and knows when it should abstain instead of guessing. Control logic and validation do not disappear there. They become even more important, because retrieval adds another stage where bad inputs can quietly degrade the final answer.

Continue the Shared Local Lab

Download the travel-copilot lab and unpack it, or use examples/travel-copilot in the repository. Run commands inside that directory. Python 3.9+ is sufficient; start with README.md and keep one out directory across lessons. Validate the source contracts; duplicate a source ID and confirm that ingestion stops.

python3 lab.py pipeline --out out

Run pipeline, retrieve, and evaluate in that order before the action stage. The lab uses fictional records, a small bilingual lexical dictionary, extractive answers, and a local SQLite supplier. Approval is simulated; no model API, real reservation, payment, OCR, or image analysis is invoked. It implements one inspectable slice of the broader architecture, with explicit extension boundaries in the README.

Sources and Further Reading

This post draws on the following primary and official sources:

• BAIR. "The Shift from Models to Compound AI Systems." Reference for control logic as a core design choice in compound AI systems. bair.berkeley.edu/blog/2024/02/18/compound-ai-systems

• OpenAI. "Function calling." Official reference for tool calling as a structured interaction between model output and application-side code. developers.openai.com/api/docs/guides/function-calling

• OpenAI. "Structured model outputs." Official reference for schema-constrained outputs; used here with the caveat that schema adherence does not guarantee semantic correctness. developers.openai.com/api/docs/guides/structured-outputs

• National Institute of Standards and Technology. "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile." Reference for risk controls, evaluation, monitoring, provenance, and trustworthy deployment framing. nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence