Evaluation and Observability for AI Systems: Build a Regression Gate

An AI system can return a polished answer while failing at retrieval, using stale evidence, calling the wrong tool, or changing the wrong record. If the team evaluates only the final prose, those failures collapse into one vague score. If the team logs everything without a measurement plan, it gets a large pile of traces but no release decision.

This tutorial turns evaluation and observability into one working control loop. You will build a versioned evaluation specification for the OptiVerse Travel copilot, run a small golden set through component and end-to-end metrics, define production service-level objectives, and decide whether a candidate release should ship. The design follows a simple rule: evaluate the component that can fail, observe the path that produced the result, and verify the real outcome rather than trusting what the model says happened.

That separation is supported by current evaluation practice. ARES evaluates context relevance, answer faithfulness, and answer relevance as distinct dimensions, while MTRAGEval separates retrieval, generation, and full multi-turn RAG because retrieval errors compound downstream (Saad-Falcon et al., 2024; Rosenthal et al., 2026). For tool-using systems, tau-bench scores whether the expected environment state was reached, not whether the agent merely claimed success (Yao et al., 2024).

Disclaimer: OptiVerse Travel, its bookings, contracts, records, and measurements in this tutorial are fictional. The metrics are tutorial fixtures, not production results.

Prerequisites

You should already be able to:

• distinguish a model call from a compound AI system

• describe a staged pipeline with explicit input and output contracts

• explain baseline RAG, citation provenance, abstention, and task state

• read JSON or YAML and calculate ratios and percentiles

You will need a spreadsheet or a short script for the calculations. No particular model provider, tracing product, or evaluation framework is required.

Learning Outcomes

By the end, you will be able to:

1. freeze an evaluation unit and version every dependency that can change behavior;

2. build a golden set that includes answerable, unanswerable, ambiguous, multi-turn, and failure-path cases;

3. separate retrieval, generation, workflow, outcome, latency, and cost metrics;

4. design a privacy-aware trace that connects a result to its evidence, tools, state, and versions;

5. set an SLO and error budget with explicit denominators, sample sizes, and uncertainty reporting;

6. apply a regression gate and explain exactly why a release passes or fails;

7. use production signals to trigger investigation, rollback, and golden-set updates.

Build Artifact

Create one file named travel-copilot-eval-v1.yaml. It will contain:

• the frozen task and dataset version

• model, prompt, retriever, index, tool, and policy versions

• component metrics, thresholds, denominators, and sampling plans

• production SLIs and SLOs

• trace fields and privacy rules

• the release-gate decision

Start with this skeleton and keep the version identifiers immutable after a run:

evaluation:
  id: travel-copilot-eval-v1
  task: accessible-japan-itinerary
  dataset_version: golden-2026-08-29.1

sample_plan:
  functional:
    cases: 6
    trials_per_case: 30
    proportion_interval: wilson_95
  latency_load:
    warmup_tasks: 100
    eligible_tasks: 1000
    offered_load_rps: 5
    good_event_threshold_ms: 8000

system_under_test:
  model: model-snapshot-A
  prompt: itinerary-prompt-v7
  retriever: hybrid-retriever-v3
  index_snapshot: partner-index-2026-08-28T18:00Z
  tools:
    availability: availability-api-v4
    budget: budget-checker-v2
  policy: travel-action-policy-v5

gates:
  offline_functional:
    retrieval_recall_micro:
      minimum: 0.95
      denominator: required_evidence_units
    citation_faithfulness:
      minimum: 0.98
      denominator: all_cited_claims
    abstention_accuracy:
      minimum: 0.95
      denominator: all_abstention_trials
    outcome_success_rate:
      minimum: 0.90
      denominator: all_functional_trials
    hard_safety_failures:
      maximum: 0
      denominator: all_targeted_safety_trials
  production_canary:
    outcome_good_event_rate:
      minimum: 0.99
      denominator: all_eligible_canary_tasks
    latency_good_event_rate:
      minimum: 0.99
      denominator: all_eligible_canary_tasks
      threshold_ms: 8000
    mean_cost_usd_per_task:
      maximum: 0.05
      denominator: all_eligible_canary_tasks

decision:
  status: pending
  failed_gates: []

Step 1: Freeze the Evaluation Contract

An evaluation is not “ask the model some questions.” It is a contract defining the unit under test, inputs, permitted state, expected outcome, graders, and denominator. Anthropic's current agent-evaluation guidance distinguishes a task, a stochastic trial, the transcript or trajectory, and the final environment outcome; it recommends multiple trials because one pass or failure does not establish a stable task success rate (Anthropic, 2026).

The sample plan separates two populations. Functional trials repeatedly exercise labeled cases so stochastic behavior and outcome correctness can be estimated. The latency-load sample exercises a stated offered load and includes every eligible request after warm-up. Do not use three sequential golden-set trials to estimate a production tail, and do not let 1,000 unlabeled load requests substitute for functional grading.

30 functional trials per case is a tutorial floor for exposing variation, not a universal production certification number. For every binary rate, retain the numerator and denominator and report a predeclared interval such as a 95% Wilson interval. For example, 27/30 = 90.0% has a wide 95% Wilson interval of approximately 74.4%–96.5%; the point estimate alone is not precise evidence of a 90% underlying rate. Choose a larger denominator from the decision's required precision and risk, and use exact binomial limits when failures or samples are few. The NIST/SEMATECH handbook documents Wilson and exact binomial intervals and explains why sample size must follow the allowable error rather than a copied trial count (NIST confidence intervals; NIST sample sizes).

Define the Unit Under Test

For this tutorial, one task begins with a travel-planner request and ends in one of four terminal states:

• described: evidence-backed options were returned;

• recommended: a recommendation was produced without executing a booking;

• abstained: the system correctly reported insufficient or conflicting evidence;

• escalated: the system created a human-review case.

Booking and payment are intentionally out of scope. If a test tool mutates the environment, the grader checks the simulated database after the run. The final sentence is never accepted as proof that the action occurred.

Version Every Behavioral Dependency

Record the exact model snapshot, prompt, retriever, index snapshot, tool schema, policy, and dataset. A release comparison is uninterpretable if two variables change silently. The NIST Generative AI Profile treats measurement, monitoring, provenance, and lifecycle risk management as connected responsibilities rather than a one-time model test (NIST, 2024).

Do not use labels such as latest in an evaluation record. If a hosted service exposes only an alias, record the alias, request time, returned model identifier, and provider request ID.

Step 2: Build a Golden Set That Includes Failure Paths

A golden set is a versioned collection of tasks with inspectable expected evidence and outcomes. It is not a list of only happy-path questions. MTRAGEval's 2026 benchmark is useful because it explicitly includes unanswerable, underspecified, non-standalone, and unclear multi-turn questions (Rosenthal et al., 2026).

Use this minimum set:

ID — Input condition — Required evidence — Expected behavior

T01 — Accessible Kyoto hotel query — Contract KYO-H12 clause 4.3 plus current availability record — Describe supported options with both sources

T02 — Hakone taxi confirmation is missing — No authoritative confirmation exists — Abstain on that leg and state the missing evidence

T03 — Old and new rate cards conflict — Newer effective-dated rate card plus conflict record — Use the current rate and disclose the conflict

T04 — Follow-up: “What about the second one?” — Prior turn plus candidate list — Rewrite the query to the referenced hotel and answer from the same case

T05 — “Book the best hotel” with no approved itinerary — Approval state is absent — Do not act; ask for the missing decision or escalate

T06 — Availability tool times out — Tool error and retry budget — Retry according to policy, then return a bounded failure state

For every answerable case, annotate evidence units at the smallest useful level: contract clause, table row, record ID, or image region. For every unanswerable case, annotate what is missing. Without those labels, retrieval recall and justified abstention cannot be measured.

Step 3: Score Components Before the Final Answer

Retrieval Metrics

For one task, define:

\[ \text{evidence recall} = \frac{\text{required evidence units retrieved}}{\text{required evidence units}} \]

\[ \text{context precision} = \frac{\text{retrieved units judged relevant}}{\text{all retrieved units}} \]

Across tasks, report both macro averages and micro totals. Macro averaging gives every task equal weight. Micro averaging combines evidence-unit counts, so evidence-heavy tasks contribute more. State which denominator you use.

Recall@k, nDCG, or MRR can be added when rank position matters. ARES and related component-level frameworks are useful diagnostic aids, but automated judges require human calibration and should not be treated as ground truth (Saad-Falcon et al., 2024).

Generation and Attribution Metrics

Break the response into checkable claims. Then measure:

\[ \text{citation faithfulness} = \frac{\text{cited claims supported by their cited evidence}}{\text{all cited claims}} \]

\[ \text{citation completeness} = \frac{\text{material claims with adequate citations}}{\text{all material claims requiring citations}} \]

Score answer correctness separately. A claim can be factually correct but unsupported by its cited source, and a faithful summary can still omit a decisive fact.

For T02, the success condition is not fluent prose. It is calibrated abstention: the system identifies the unsupported subtask, preserves supported parts, and does not invent confirmation.

Workflow and Outcome Metrics

For tool-using tasks, check:

• the correct tool and arguments were used;

• authorization and approval policies were respected;

• retries and terminal states matched policy;

• the expected environment record exists or does not exist;

• no unrelated record changed.

This is why tau-bench evaluates an agent against policy and resulting database state rather than final text alone (Yao et al., 2024). A statement such as “the review case was created” passes only if the simulated review table contains the correct new record.

Step 4: Capture a Trace You Can Actually Use

Observability is the ability to explain system behavior from emitted signals. Logging every prompt is neither necessary nor sufficient. OpenAI's current agent-evaluation guidance describes a trace as the end-to-end record of model calls, tool calls, guardrails, and handoffs, then uses trace graders to find workflow-level regressions (OpenAI, 2026).

Use a trace envelope like this:

{
  "trace_id": "tr_01JPN0417",
  "task_id": "T03",
  "run_id": "run_20260829_003",
  "versions": {
    "model": "model-snapshot-A",
    "prompt": "itinerary-prompt-v7",
    "retriever": "hybrid-retriever-v3",
    "index": "partner-index-2026-08-28T18:00Z",
    "policy": "travel-action-policy-v5"
  },
  "retrieval": {
    "query_hash": "sha256:example",
    "selected_evidence_ids": ["KYO-H12#4.3", "RATE-2026-04#row-8"],
    "candidate_count": 12
  },
  "tool_calls": [
    {
      "call_id": "call_01",
      "tool": "availability.lookup",
      "schema_version": "v4",
      "status": "ok",
      "latency_ms": 184
    }
  ],
  "terminal_state": "described",
  "outcome_verified": true,
  "latency_ms": 2410,
  "cost_usd": 0.041,
  "privacy": {
    "raw_prompt_stored": false,
    "tool_payloads_redacted": true,
    "retention_class": "evaluation-30d"
  }
}

Do not log passport numbers, payment data, full customer messages, access tokens, or unrestricted tool payloads merely because a tracing SDK makes it easy. OpenTelemetry's generative-AI semantic conventions provide common attribute names, but standardization does not make sensitive arguments or results safe to retain (OpenTelemetry, 2026). Prefer stable record IDs, hashes, redacted summaries, access-controlled payload stores, and explicit retention classes.

Step 5: Define SLIs, SLOs, and the Regression Gate

An SLI is a measured signal. An SLO is the target for that signal over a stated population and time window. Google SRE guidance emphasizes defining what is measured, which events count, and the target before treating an SLO as an operational contract (Google, 2020).

For the travel copilot, use one latency gate contract:

• outcome_success_rate: verified successful outcomes divided by all functional trials;

• outcome_good_event_rate: verified successful outcomes divided by all eligible canary tasks;

• latency_good_event_rate: tasks reaching a terminal artifact within 8,000 ms divided by all eligible canary tasks;

• p95_end_to_end_ms: a diagnostic distribution statistic calculated from individual observations, not the release gate in this artifact;

• unsupported_claim_rate: unsupported material claims divided by material claims;

• escalation_precision: justified escalations divided by all escalations;

• mean_cost_usd_per_task: total evaluated cost divided by all eligible canary tasks.

Do not gate on “99% under 8 seconds” in one place and p95 <= 8,000 ms in another. Those contracts differ: the event-ratio contract permits at most 1% bad latency events over the stated population, while a percentile contract depends on the quantile estimator and treatment of ties and censored timeouts. This artifact uses the event ratio for its frozen gate and reports p95 only for diagnosis.

Suppose a one-day canary receives 1,000 eligible tasks after 100 warm-up tasks at the frozen offered load:

• 970 reach the correct verified outcome;

• 982 finish within 8 seconds;

• 1,600 tool calls include 70 retryable failures, of which 60 recover;

• total cost is USD 39.00.

Then:

\[ \text{outcome good-event rate} = 970 / 1000 = 97.0\% \]

\[ \text{latency good-event rate} = 982 / 1000 = 98.2\% \]

\[ \text{retry recovery} = 60 / 70 = 85.7\% \]

\[ \text{mean cost} = 39.00 / 1000 = \$0.039 \]

The approximate 95% Wilson intervals are 95.7%–97.9% for outcome good events and 97.2%–98.9% for latency good events. If the frozen canary SLO requires 99% for each event ratio, this canary fails despite meeting the USD 0.05 cost target. Cost does not compensate for correctness or latency failure, and reporting intervals does not change the predeclared denominator or threshold.

Worked Example: Score One Candidate Release

First inspect four representative tasks. Evidence and claims are annotated by humans. This small slice demonstrates the arithmetic and can reject an obviously broken candidate early; it cannot establish a pass before the frozen functional sample plan is complete.

Task — Required evidence retrieved — Supported cited claims — Expected terminal outcome reached

--- — ---: — ---: — ---

T01 — 2 / 2 — 3 / 3 — yes

T02 — not applicable — 2 / 2 — yes, abstained

T03 — 1 / 2 — 2 / 3 — no

T04 — 2 / 2 — 2 / 2 — yes

For the three answerable retrieval tasks:

\[ \text{micro retrieval recall} = (2 + 1 + 2) / (2 + 2 + 2) = 5/6 = 83.3\% \]

Across all cited claims:

\[ \text{citation faithfulness} = (3 + 2 + 2 + 2) / (3 + 2 + 3 + 2) = 9/10 = 90.0\% \]

Across all tasks:

\[ \text{outcome success} = 3/4 = 75.0\% \]

The candidate is rejected early against the tutorial thresholds of 95% retrieval recall, 98% citation faithfulness, and 90% outcome success; a candidate may fail before reaching the planned denominator, but it may not pass early. The trace for T03 shows the root cause: retrieval selected an expired rate card and omitted the effective-dated replacement. The correct action is to fix and re-evaluate retrieval/index freshness, not to rewrite the final-answer prompt.

Update the artifact:

decision:
  status: fail
  failed_gates:
    - retrieval_recall_micro
    - citation_faithfulness
    - outcome_success_rate
  primary_failure_cluster: stale-index-retrieval
  recovery:
    component: partner-index
    mode: verified-known-good-or-rebuild
    snapshot_id: null
    requirements:
      integrity_manifest: required
      last_passing_eval_run: required
      rerun_all_gates: required

snapshot_id is deliberately null: an older timestamp is not evidence that an index is good. Restore only an immutable snapshot whose integrity manifest and recorded evaluation show that it passed the relevant gates. If no such snapshot exists, quarantine the candidate index, rebuild from versioned source records, and run the full component and end-to-end gates before receiving traffic.

Implementation Exercise

Run the two frozen samples separately.

For the functional sample, run 30 trials for each T01–T06 case. For every trial, record:

1. required and retrieved evidence IDs;

2. material claims and supporting source IDs;

3. selected tools, arguments, errors, and retries;

4. terminal state and verified environment outcome;

5. latency and cost;

6. all model, prompt, data, tool, and policy versions.

Calculate micro retrieval recall, macro task success, citation faithfulness, abstention accuracy, and mean cost. For every binary rate, report successes / eligible trials, exclusions, and the 95% Wilson interval. Three trials may be retained as a smoke test, but not reported as a stable success-rate estimate.

For the latency-load sample, exclude the 100 declared warm-up tasks, then send 1,000 eligible tasks at 5 offered requests per second using the same system versions. Count every timeout and failed terminal state in the denominator. Gate on latency_good_event_rate = tasks <= 8,000 ms / 1,000; calculate p50, p95, and p99 from the 1,000 individual observations only as diagnostics. Do not mix these load observations into the labeled functional denominator.

Then compare a candidate against a frozen baseline. Do not average away a safety regression: any unauthorized action, cross-tenant access, or false claim of action is an automatic release failure even if aggregate quality improves.

Deliberately Broken Case: The Dashboard That Lies

Assume a dashboard reports 99.2% “answer quality.” Investigation reveals four defects:

• the denominator excludes abstentions and tool failures;

• a single LLM judge scores style and factuality together;

• production uses itinerary-prompt-v8, but traces record v7;

• the dashboard samples only completed requests, excluding timeouts.

Repair it by separating dimensions, restoring all eligible tasks to the denominator, recording returned runtime versions, and treating timeout as an outcome. Recompute the metric. If 200 completed requests include 198 good results but 10 additional eligible requests timed out, the honest completion-conditioned score may be 198/200 = 99%, while the user-visible success rate is:

\[ 198 / 210 = 94.3\% \]

This is an observability failure, not just a charting mistake. The system hid the population that experienced the failure.

Production Monitoring, Drift, and Rollback

Offline evaluation and production monitoring answer different questions. A golden set asks whether a version can handle known, labeled cases. Production signals ask whether the live workload, dependencies, and users are changing.

Monitor distributions, not only totals:

• request type, language, trip region, and answerability class;

• retrieval result count, score distribution, and evidence age;

• tool error code, retry count, and dependency latency;

• terminal state, escalation reason, and approval outcome;

• latency percentiles, token use, and cost by task class;

• model, prompt, index, schema, and policy version.

When a threshold burns its error budget or a safety invariant fails:

1. stop or reduce the affected release;

2. segment traces by version, task class, tenant, and dependency;

3. inspect representative successful and failed trajectories;

4. restore the smallest responsible component only from a verified-known-good snapshot, or rebuild it from versioned inputs;

5. add the newly understood failure to the golden set;

6. rerun component and end-to-end gates before restoring traffic.

Observability is complete only when it can support that decision path. A dashboard without a verified recovery artifact or rebuild plan, version identifiers, or inspectable exemplars is reporting, not operational control.

Measurable Exit Criteria

Your tutorial implementation is complete only if:

• all six golden cases have explicit evidence and expected terminal states;

• every trial records model, prompt, retriever, index, tool, and policy versions;

• retrieval and generation are scored separately;

• action claims are checked against environment state;

• 30 functional trials per case can be reproduced from a clean test environment;

• the separate load sample contains 100 excluded warm-up tasks and 1,000 eligible tasks at 5 offered requests per second;

• no raw secrets, payment data, or unrestricted customer content appears in the trace;

• all hard safety invariants pass;

• every numerical gate includes numerator, denominator, population, window, and uncertainty-reporting method;

• one intentionally failing candidate is rejected for the correct component-level reason;

• the artifact names a verified-known-good recovery snapshot or a versioned rebuild plan and an owner for follow-up.

Retrieval-Practice Check

Answer without looking back.

1. Why can a final-answer score not diagnose a retrieval failure?

2. What is the difference between a task and a trial?

3. Why must an action claim be checked against environment state?

4. What is the micro retrieval recall in the worked example?

5. Why does a 99% score over completed requests differ from user-visible success?

6. Name three dependencies that must be versioned besides the model.

7. What exact latency contract does the frozen artifact gate on, and why are the functional and load samples separate?

Answer Key

1. The same bad answer can result from missing evidence, misuse of good evidence, a tool failure, or generation error; component metrics isolate the failing boundary.

2. A task is the fixed test case and success contract; a trial is one stochastic attempt at that task.

3. The model can say an action succeeded even when the tool failed or modified the wrong record.

4. 5/6 = 83.3%.

5. Completed-only scoring removes timeouts and other failed experiences from the denominator.

6. Any three of prompt, retriever, index snapshot, tool/schema, policy, dataset, or runtime configuration.

7. It gates on eligible canary tasks completed within 8,000 ms divided by all eligible canary tasks, with a 99% minimum. Functional trials estimate labeled behavior under stochastic repetition; the separate offered-load sample estimates latency without pretending unlabeled requests prove correctness.

What to Build Next

The next prerequisite is identity and authorization. Evaluation can reveal that a system took an unauthorized action, but it does not itself prevent the action. The following tutorial builds the principal, permission, delegation, trust-boundary, and prompt-injection controls that the regression suite should test.

Bibliography

• Saad-Falcon, J., Khattab, O., Potts, C., and Zaharia, M. “ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems.” NAACL, 2024. Component-level evaluation with a human-calibration requirement. aclanthology.org/2024.naacl-long.20

• Rosenthal, S., Shah, V., Katsis, Y., and Danilevsky, M. “SemEval-2026 Task 8: MTRAGEval.” ACL/SemEval, July 2026. Multi-turn retrieval, generation, and end-to-end RAG evaluation. aclanthology.org/2026.semeval-1.447

• Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. “tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.” 2024. Policy and environment-state evaluation for tool agents. arxiv.org/abs/2406.12045

• Anthropic. “Demystifying evals for AI agents.” January 9, 2026. Current practitioner guidance on tasks, trials, trajectories, outcomes, and graders. anthropic.com/engineering/demystifying-evals-for-ai-agents

• OpenAI. “Evaluate agent workflows.” Current documentation, accessed August 29, 2026. Trace grading and repeatable datasets for workflow evaluation. developers.openai.com/api/docs/guides/agent-evals

• National Institute of Standards and Technology. “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.” NIST AI 600-1, July 2024. Lifecycle measurement, monitoring, provenance, and risk management. doi.org/10.6028/NIST.AI.600-1

• OpenTelemetry. “Generative AI attributes.” Semantic conventions accessed August 29, 2026. Common trace attributes; payload privacy still remains an application responsibility. opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai

• Thurgood, S., and Ferguson, D., with Hidalgo, A., and Beyer, B. “Implementing SLOs.” *The Site Reliability Workbook*, online edition. SLI, SLO, population, and error-budget practice. sre.google/workbook/implementing-slos

• NIST/SEMATECH. “Confidence intervals” and “Sample sizes required.” *e-Handbook of Statistical Methods*. Wilson and exact binomial intervals, allowable error, and proportion sample planning. itl.nist.gov confidence intervals; itl.nist.gov sample sizes