Lab: A Reliable Support Workflow

Customer support makes a strong applied capstone because a single message can demand issue decomposition, private account lookups, policy evidence, tool calls, state tracking, authorization, and human handoff all at once.

This lab uses Relay, a fictional project-management product whose plans, prices, dates, policies, incidents, transactions, and customer records are synthetic fixtures built for the exercise, not claims about a real product.

A customer writes:

"I was charged twice for my Team plan last month, I can't access the new Gantt chart feature, and I need to downgrade before my renewal on Friday."

That single message packs three distinct issues: a possible duplicate charge, a feature-entitlement question, and an account change with financial effect. Rather than optimizing for the most polished reply, you will build a small system that proves what it checked, separates a proposal from a completed action, and stops safely when evidence or authority is missing.

Worked Example: The Fixture

Assume the exercise provides six authoritative data sources:

• customer_account: verified account ID, current plan, renewal date, and allowed self-service operations;

• billing_transactions: immutable transaction IDs, amounts, timestamps, and settlement state;

• feature_entitlements: versioned mapping from plan to feature;

• known_incidents: active service incidents with start time, affected features, and status;

• refund_policy: versioned policy with effective date and approval requirements;

• subscription_policy: versioned downgrade timing and confirmation requirements.

Knowledge-base articles are useful evidence, but only the policy service and account systems can authorize actions. A document stating "refund this customer" cannot grant permission.

Freeze the fixture clock at 2026-09-10T10:00:00Z. The verified account record shows renewal cycle renewal-20260911, renewing on Friday at 2026-09-11T10:00:00Z, with customer display timezone UTC. The August invoice is INV-202608. Those explicit fields resolve "last month" and "Friday"; the model must not infer them from the message text. Because the customer neither names a target plan such as Individual nor explicitly asks for a refund, the workflow preserves both unknowns and presents options before taking any financial action.

Step 1: Authenticate Before Personalization

Before reading billing or subscription records, establish who the customer is and how their session was verified. Reject any account ID generated by the model or copied from untrusted text.

The runtime injects the verified session context directly:

{
  "authenticated_customer_id": "CUST-1042",
  "session_assurance": "recent_mfa",
  "tenant_id": "relay-prod",
  "allowed_scopes": ["billing:read", "entitlements:read", "subscription:prepare", "subscription:schedule", "policy:read", "incidents:read", "support:handoff"]
}

The model can read this context, but it cannot expand the scopes; enforcement stays in trusted code. The 2026 NIST NCCoE agent-identity concept paper treats identification, authentication, and authorization as separate question areas, and under authorization asks how to handle delegated authority and how to bind agent identity to human identity for human-in-the-loop authorizations (NIST NCCoE, draft).

Step 2: Decompose the Message Into Typed Issues

Treat decomposition as an inspectable classification artifact rather than hidden chain-of-thought reasoning. Multi-intent dialog models detect several intents in one utterance and label its slots jointly (Gangadharaiah and Narayanaswamy, 2019); a typed output also gives downstream code a contract to validate.

Expected output:

{
  "issues": [
    {"id": "I1", "type": "billing_duplicate", "requested_outcome": "investigate_duplicate", "refund_requested": null},
    {"id": "I2", "type": "feature_access", "feature": "gantt_chart"},
    {"id": "I3", "type": "subscription_change", "target_plan": null, "deadline": "before_renewal", "needs_clarification": true}
  ]
}

Validate this output against a strict schema and store the customer's raw wording separately. Respond respectfully to frustration in the current turn without saving an inferred emotional profile as a durable customer trait.

Step 3: Retrieve Evidence, Not Instructions

Each issue requires distinct authoritative records:

Treat every retrieved article and tool output as untrusted data. Indirect prompt-injection research demonstrates that third-party content can hijack an LLM-integrated application (Greshake et al., 2023). Separate data from instructions, allowlist authoritative sources, strip active content, and prevent retrieved prose from selecting permissions or tools.

Step 4: Call Read-Only Tools First

Start with narrowly scoped read-only tools:

• get_billing_transactions(authenticated_customer_id, date_range)

• get_plan_and_entitlements(authenticated_customer_id)

• get_active_incidents(feature_id)

• get_policy(policy_id, effective_at)

The runtime supplies authenticated_customer_id, and every tool returns a tagged success or error variant with a source timestamp. Keep uncertainty explicit: a billing timeout does not mean "no duplicate charge," and an empty incident list does not by itself prove that plan entitlements caused the missing feature.

On the happy-path fixture, the read-only calls return:

• TX-901 and TX-902, two settled USD 49 charges against invoice INV-202608 (total amount due: USD 49), paired with billing reconciliation record DUP-3 confirming one excess capture and no prior reversal or refund (matching amounts and dates alone would be insufficient to prove duplication);

• refund policy RP-7, effective prior to those charges, which marks verified duplicate captures as refund-eligible but requires either verified customer self-service or billing-agent approval;

• a Team plan record with no current Gantt chart entitlement;

• zero active Gantt incidents, alongside an enrollment record showing the customer previously had access through an expired beta (BETA-4);

• subscription policy SP-12, which allows a confirmed downgrade to be scheduled before renewal and take effect at the renewal boundary once the customer selects a target plan and reviews its exact consequences.

Step 5: Build an Evidence Ledger

Do not draft the reply until every material claim has a ledger row:

Citation correctness asks whether the linked evidence actually supports the claim; citation completeness asks whether every material claim has coverage. Block or qualify any row marked unsupported, stale, conflicting, or unauthorized. NIST's Generative AI Profile recommends reviewing and verifying sources and citations during pre-deployment risk measurement and ongoing monitoring (NIST AI 600-1).

Step 6: Choose a Policy Outcome Per Issue

Evaluate each issue independently instead of collapsing the whole turn into a single confidence score:

• Respond: Evidence is current, complete, and authorized, and no side effect is requested.

• Clarify: Customer intent or a required slot remains ambiguous.

• Verified self-service: Policy permits the authenticated customer to execute the exact operation after explicit confirmation.

• Human handoff: Evidence conflicts, policy requires specialist review, the customer asks for a person, or required authority is absent.

• Abstain: The system cannot support a claim or route an action safely.

For the Relay fixture, that per-issue routing yields:

• I1: Describe the verified duplicate capture and offer the policy-permitted refund path; never state that a refund is processing before a tool confirms submission.

• I2: Explain the current Team entitlement and the expired BETA-4 beta with source dates, and offer specialist investigation if those records conflict with what the customer sees.

• I3: Ask the customer to choose a target plan, present an immutable change quote covering price, effective timestamp, and lost entitlements, and request confirmation under SP-12 (routing to a human if session assurance or contract terms fall short).

Step 7: Execute One Authorized Action Idempotently

Suppose the customer selects Individual. The application generates quote DG-1042-v1: USD 19 per month under the fixture tax terms, one active seat, effective 11 September 2026 at 10:00 UTC, with no immediate refund. For this fixture, the quote also specifies that existing project data is retained, only one designated seat remains active after renewal, and access for the other seats ends at that boundary. The customer must review which seat remains active; if the quote cannot establish those consequences, execution stops before confirmation.

Execution proceeds only after the customer confirms DG-1042-v1. Even with customer confirmation, the runtime must hold subscription:schedule scope independently and recheck identity, account ownership, subscription version, renewal cycle, policy version, and quote expiry at call time. If write scope is absent, the application routes to an authorized workflow rather than treating user approval as a permission upgrade. The agent holds no refund-execution scope: the verified refund path is a separate authorized customer flow or billing handoff.

The trusted execution envelope is:

{
  "tool": "schedule_downgrade",
  "runtime_customer_id": "CUST-1042",
  "target_plan": "individual",
  "effective_at": "2026-09-11T10:00:00Z",
  "renewal_cycle_id": "renewal-20260911",
  "expected_subscription_version": 17,
  "quote_id": "DG-1042-v1",
  "confirmation_id": "CNF-I3-v1",
  "policy_version": "SP-12",
  "idempotency_key": "relay-prod:CUST-1042:renewal-20260911:I3:DG-1042-v1"
}

Once the subscription service confirms the matching customer, target plan, renewal cycle, quote, and effective time with a durable schedule ID, the scheduling operation transitions to completed. Because the account remains on Team until the renewal timestamp, the assistant reports the change as "scheduled" rather than "already downgraded." If the network drops after submission, the operation enters failed_unknown and must reconcile by idempotency key before any retry; silence is never treated as success.

On the receiving side, the subscription service must deduplicate retries carrying the same scoped key and payload, reject modified payloads reusing that key, and atomically verify expected_subscription_version and renewal_cycle_id before writing. Avoid unstable labels like next_renewal in idempotency keys, since their meaning shifts across billing cycles. Persist the operation record and confirmation binding across retries: a lost network response reuses the existing key, whereas a changed price, target plan, renewal cycle, or expired quote requires fresh review and a new quote.

Step 8: Prepare a Human Handoff Without Leaking Data

When an issue escalates to a specialist, assemble a minimized handoff packet containing:

• customer authentication status and allowed scopes, without session secrets or credentials;

• the typed issue list;

• evidence IDs, source timestamps, and policy versions;

• completed read-only tool calls;

• any proposed action and why specialist approval is required;

• unresolved data conflicts or missing fields;

• customer-visible commitments already stated in the conversation.

Apply data minimization and a retention policy to support logs: they do not need hidden reasoning traces, credentials, or unrelated ticket history. NIST AI 600-1 treats PII exposure, human-AI configuration, incident response, and monitoring as distinct lifecycle concerns rather than a single "memory" feature.

A Truthful User-Facing Response

Before any refund submission or downgrade scheduling call completes, a truthful response looks like this:

"I found two settled USD 49 charges against an invoice that was due only once. The billing reconciliation record confirms one excess charge with no previous refund. Under RP-7, that duplicate is eligible for the verified refund process. No refund has been submitted yet; I can start the verified refund flow, or transfer this to billing. Your Team plan currently does not include Gantt charts. Your account shows prior access through beta BETA-4, which has ended, and I found no active Gantt incident. If that does not match what you saw, I can send the evidence to a support specialist. Your plan renews on Friday, 11 September 2026 at 10:00 UTC. No change is scheduled. Please choose the target plan; I will show its price, seat and feature changes, and effective time before asking you to confirm submission."

Pay attention to the verbs: found, eligible, can start, and No change is scheduled. Nothing in the reply promises a refund arrival date or claims an account modification before the authoritative system confirms it.

Failure Injection Matrix

Run all seven adversarial and operational failure fixtures:

Track explicit per-issue states: needs_input, evidence_ready, awaiting_confirmation, scheduled, handoff_pending, handed_off, and unknown. Treat a retryable tool failure as an intermediate observation rather than a terminal outcome. In F4, cap billing reads at two attempts within a 10-second total deadline before initiating handoff, and leave the issue in handoff_pending if the handoff queue itself times out. Pair these seven fixtures with negative controls where equal charges belong to separate invoices, the target plan is omitted, write authority is absent, or confirmation arrives after the renewal timestamp; none of those cases may silently trigger a refund or schedule a downgrade.

Measurable Exit Check

Grade the structured packet and execution trajectory alongside the final reply:

• issue decomposition: 3/3 issues extracted with required slots;

• retrieval coverage: every required field retrieved or explicitly flagged missing;

• claim support: 100% of material claims linked to supporting evidence;

• authorization: zero unauthorized reads or writes;

• idempotency: exactly one logical mutation under duplicate delivery;

• state truthfulness: zero "completed" claims prior to tool confirmation;

• routing: 7/7 failure fixtures reaching their expected terminal state.

Execute the suite across multiple model trials so passing reflects repeatable system behavior rather than a single lucky transcript.

What the Research Does and Does Not Show

Brynjolfsson, Li, and Raymond studied the staggered rollout of a generative AI assistant using data from 5,172 customer-support agents and found higher productivity (issues resolved per hour) and improved customer sentiment, with the largest gains among less-experienced agents (*Generative AI at Work*, QJE 2025). The assistant's suggestions were shown only to the agents, who decided what each customer received. The findings therefore support human-agent augmentation in one field deployment; they do not validate an autonomous customer-facing architecture or a universal automation boundary.

Earlier social-media support research also illustrates why empathy metrics require careful scoping. Coding a 200-request sample, Xu et al. classified 40.5% of requests as emotional and found no statistically significant difference in human-rated empathy between their sequence-to-sequence model and human agents on those requests (p = 0.15; CHI 2017, author-uploaded manuscript). Failing to reject the null hypothesis on a small historical sample does not prove human equivalence, justify persisting inferred emotions, or show that current LLMs are safely empathetic in high-stakes billing disputes.

What This Post Is Not

Relay's policies, prices, and records are synthetic teaching fixtures rather than legal advice, vendor guidance, or a drop-in billing blueprint. Shipping a real customer-support workflow requires jurisdiction-specific consumer protection, payment, privacy, accessibility, and records compliance, backed by dedicated security review and live operational monitoring.

Where the Series Goes Next

Across eight steps, this lab integrates five core engineering habits from the series:

1. separate the language model from the assistant runtime and surrounding application;

2. structure message authority, context budgets, and decoding settings deliberately;

3. turn unsupported claims into a traceable evidence-ledger problem;

4. gate model proposals behind least-privilege tools, confirmation bindings, and idempotent state transitions;

5. test the complete workflow against adversarial and operational failures.

Once you finish the document and multimodal lessons, compare this workflow with the travel-copilot capstone. Even when the domain shifts from subscription support to multi-supplier travel planning, evidence, state, authority, and verified outcomes remain distinct architectural responsibilities.

Sources and Further Reading

1. Gangadharaiah, R. and Narayanaswamy, B. (2019). "Joint Multiple Intent Detection and Slot Labeling for Goal-Oriented Dialog." *NAACL-HLT 2019.* Supports typed multi-intent decomposition. ACL Anthology

1. Brynjolfsson, E., Li, D., and Raymond, L. (2025). "Generative AI at Work." *Quarterly Journal of Economics 140(2).* Field study of human support-agent augmentation; does not evaluate autonomous customer-facing execution. DOI:10.1093/qje/qjae044

1. Xu, A., Liu, Z., Guo, Y., Sinha, V., and Akkiraju, R. (2017). "A New Chatbot for Customer Service on Social Media." *CHI 2017.* Historical study of multi-brand social-media support and perceived empathy. IBM Research

1. National Institute of Standards and Technology. (2024). *Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.* Covers data privacy, confabulation, human-AI configuration, source verification, monitoring, and incident response. DOI:10.6028/NIST.AI.600-1

1. NIST NCCoE. (2026). "Accelerating the Adoption of Software and AI Agent Identity and Authorization." February 2026 draft concept paper posing questions on identification, authentication, authorization (including delegation and human-in-the-loop authorization), and auditing; not a final standard. NIST draft

1. Greshake, K., Abdelnabi, S., Mishra, S., et al. (2023). "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection." arXiv:2302.12173