Benchmarking Inference Capacity

An inference benchmark asks whether a versioned system can sustain a declared workload while meeting its latency and reliability criteria for the required share of requests. A raw tokens-per-second figure cannot answer that capacity question without the workload, arrival process, cache state, measurement population, and acceptance criteria.

This tutorial turns the placement from I-06 into an operations artifact. You will freeze a mixed workload, run both saturation and rate-based tests, calculate request metrics and goodput, derive a replica requirement with headroom, define a dashboard, and inject a failure to check that the service fails safely and observably.

inference-capacity-pack/
├── workload-manifest.yaml
├── requests.csv
├── slo-report.json
└── failure-test.md

Freeze the Serving System

The worked example uses these hypothetical planning assumptions:

Pin the engine and tokenizer revisions, because metric behavior and token counts can change between versions. vLLM's benchmark documentation also notes that metric terminology is not standardized across benchmarking tools, so report formulas and measurement points alongside metric names (vLLM Benchmark CLI).

Freeze the Workload Distribution

A single average input length erases the difference between prefill-heavy and decode-heavy traffic. Define explicit classes instead:

For this fixed mix:

weighted input tokens
  = 0.70×1,024 + 0.20×8,192 + 0.10×512
  = 2,406.4 tokens/request

weighted output tokens
  = 0.70×128 + 0.20×256 + 0.10×1,024
  = 243.2 tokens/request

The weighted values help validate aggregate counters, but SLO attainment must still be evaluated independently for every class. An aggregate percentile or attainment ratio can look healthy while a lower-volume class, such as long-context, misses its target.

The manifest should also freeze sampling parameters, stop behavior, chat template, prefix distribution, request cancellation policy, benchmark duration, client location, exact model/engine configuration, and the random seed or request schedule. The arrival model is part of the experiment, not a hidden client detail. NVIDIA's AIPerf, for example, documents constant, Poisson, and gamma arrivals for rate-based runs, a concurrency-burst mode when no rate is set, and fixed-schedule trace replay (AIPerf load-generator options).

Closed Loop and Open Loop Answer Different Questions

Closed-Loop Concurrency Test

A closed-loop test keeps up to N requests in flight. When one completes, the client sends another. As latency rises, completions slow down, and so do new arrivals. The test is useful for finding saturation and the throughput-latency frontier, but because it self-throttles, it cannot show that the service absorbs an independent production arrival rate.

AIPerf calls concurrency-only operation burst mode: it dispatches requests with zero delay, limited only by the concurrency ceiling, and lists saturation testing among its uses (AIPerf arrival patterns).

Open-Loop Rate Test

An open-loop test schedules arrivals at a target rate independent of completion, often using a Poisson or trace-derived process. If offered load exceeds service capacity, the queue grows or bounded admission rejects work, and TTFT together with rejection and timeout counters exposes the overload. This is the right test for an external arrival-rate SLO, provided the load generator does not silently hold scheduled arrivals behind a concurrency cap. AIPerf, for instance, blocks rate-scheduled requests whenever a configured concurrency ceiling is reached (AIPerf load-generator options).

Use both, and add trace replay when you have production data:

1. a closed-loop concurrency sweep to discover saturation candidates;

2. an open-loop rate sweep to validate queue stability and SLO attainment at realistic arrival rates; and

3. timestamped trace replay when production arrival and length distributions are available.

Define the Metrics Before Running the Test

For request i, with client-send time s_i, first non-empty token time f_i, final token time e_i, and O_i output tokens:

TTFT_i = f_i - s_i
E2E_i  = e_i - s_i
TPOT_i = (E2E_i - TTFT_i) / (O_i - 1), for O_i > 1

If token arrival timestamps are t_i,1 ... t_i,O_i, the individual inter-token latencies are:

ITL_i,j = t_i,j - t_i,j-1, for j = 2 ... O_i

These token-gap formulas assume that each visible token has its own observed arrival time. When a chunk contains multiple tokens, report the observed chunk gaps instead of inventing per-token timestamps. E2E here ends at the final visible token, excluding trailing protocol metadata, and all timestamps come from one monotonic client clock. TPOT is undefined for outputs of zero or one token; every class in this fixed-length fixture produces more than one.

TPOT is one average per request; ITL records the individual gaps. When every token arrives as its own timestamped event, TPOT equals the arithmetic mean of that request's ITLs, but the mean can hide a long stall. When one streamed chunk carries several tokens, as it can under speculative decoding, chunk gaps and per-token averages use different denominators. Compare formulas and observation points before comparing metrics (vLLM Benchmark CLI).

For a measurement window of duration T:

request_throughput = successful_requests / T
output_throughput  = sum(successful_output_tokens) / T
error_rate         = failed_requests / attempted_requests
observed_goodput   = successful_requests_meeting_their_class_request_cutoffs / T
class_attainment_c = passing_successful_requests_in_class_c
                     / successful_requests_in_class_c
all_class_SLOs_pass = AND over every class c of:
                      class_attainment_c >= 0.95
                      AND successful_requests_in_class_c >= 2,000

This tutorial uses SLO-compliant offered capacity to mean the highest tested offered request rate at which the service simultaneously has:

• at least 95% joint request-level latency attainment among successful requests in every workload class independently;

• at least 2,000 successful observations in every class;

• no more than 1% errors across all attempted requests; and

• no sustained queue growth during the steady-state window.

Overall observed goodput is still useful, but it cannot rescue a point at which one class fails its latency gate or its sample-size gate. The latency denominator excludes errors while the error cap is global, and that combination has a blind spot. A fixed 70%/20%/10% workload with 24,000 attempts and all 240 errors in the smallest class has a 1% global error rate but only 90% success in that class. If its 2,160 successful requests all meet their latency cutoffs, the point passes this declared gate. When the requirement is per-class availability, or latency attainment over all attempted requests, add a per-class error gate or an all-attempt attainment gate.

DistServe defines goodput as the maximum request rate per provisioned GPU that meets an SLO attainment goal covering both TTFT and TPOT. That is a capacity measure, closer to this tutorial's SLO-compliant capacity normalized per GPU than to the per-window observed goodput above, so state which definition a reported goodput uses. DistServe's reported gains remain specific to its evaluated models, workloads, and SLOs (Zhong et al., OSDI 2024).

Control Cache State Instead of Averaging It Away

Run two named experiment families:

Cold Cache

• restart the engine or explicitly clear its cache before each point;

• for a sustained cold run, prevent prefix reuse within the run as well as across runs; unique early prefixes or supported per-request isolation can do this;

• exclude compilation and model-load warmup from the steady-state window; and

• report a prefix-cache hit rate near zero.

Warm Cache

• prewarm a versioned set of repeated prefixes;

• preserve the same prefix-frequency distribution across configurations;

• begin measurement only after the intended cache state exists; and

• report prefix-cache hit rate and eviction rate.

If prompts share a system prefix, clearing the cache once produces a cold start, not a persistently cold workload; report that warm-up trajectory separately. Never combine cold and warm requests into one unlabeled average. vLLM's benchmark guide warns that rerunning a reproducible dataset against the same server can reuse prompts left in the prefix cache and inflate throughput (vLLM Benchmark CLI).

Worked Example: Find SLO-Compliant Capacity

Assume a fixed-seed Poisson-arrival schedule of 22,000 requests at each point, stratified into exactly 15,400 interactive, 4,400 long-context, and 2,200 generation-heavy requests before shuffling. Measure for at least five minutes, and continue the precommitted schedule if any class finishes with fewer than 2,000 successful observations. Stop at a declared maximum duration and label a point that is still under-sampled inconclusive, so a class that fails completely cannot make the test run forever. Keep the cache policy fixed. The following results are parameterized tutorial data, not measurements from OptiVerse infrastructure:

At 28 RPS, 10% errors would leave the smallest class with only 1,980 successful requests from the initial 22,000-request schedule, so the run continues to 24,000 attempts and retains 2,160. Rates and output counters are rounded illustrations that assume roughly proportional class failures; they are not a recorded run.

The 28 RPS point has the highest raw output throughput but lower observed goodput than the 24 RPS point. Its weakest class reaches only 78.0% attainment, its interactive P95 TTFT and P95 TPOT both exceed the class cutoffs, its error rate is too high, and its queue grows. Choosing by raw throughput alone would pick the wrong operating point.

With four replicas, 24 RPS is the highest tested offered rate at which every class has at least 95% latency attainment among successful requests and at least 2,000 successful observations, while the global error rate stays at or below 1% and the queue stays flat. Its weakest class reaches 95.1% latency attainment over 2,191 successful requests. Therefore:

illustrative SLO-compliant capacity per replica
  = 24 RPS / 4 replicas
  = 6 RPS/replica

This division is valid only for the same workload mix, placement, cache state, engine configuration, and scaling regime. It is not a universal per-replica constant.

Worked Capacity Plan With Headroom

Suppose forecast peak load is 36 RPS and policy reserves 25% capacity, so planned utilization is at most 75%:

required_replicas
  = ceil(peak_RPS / (capacity_per_replica × target_utilization))
  = ceil(36 / (6 × 0.75))
  = ceil(8)
  = 8 replicas

required_GPUs = 8 replicas × 2 GPUs/replica = 16 GPUs

The current eight-GPU cluster runs four replicas, enough for 18 RPS under the 75% policy, so it cannot carry the 36 RPS peak with the requested reserve. The calculation does not authorize procurement by itself: validate the eight-replica topology first, because load-balancer behavior, network paths, cache locality, and shared bottlenecks can prevent linear scaling. Dynamo's sizing guide draws the same boundary: AIConfigurator, an analytical performance model, proposes configurations but does not simulate request-by-request scheduler or KV-cache behavior, so a live benchmark is still required (Dynamo sizing with AIConfigurator).

Little's Law supplies a consistency check for a stable window:

average_in_flight ≈ completion_rate × average_E2E_seconds

As a standalone illustration, separate from the worked example's workload, take a stable window with 23.9 terminal completions per second (all statuses) and a mean time to terminal completion of 1.8 seconds. The expected average in-flight count is 23.9 × 1.8 = 43.02. The completion counter, mean duration, and in-flight gauge must cover the same population, including timeouts and failures; successful-request E2E alone cannot explain a gauge that also counts failed requests. A large gap between this estimate and the observed average gauge suggests a boundary mismatch, dropped telemetry, retries, or an unstable queue. Little's Law cannot rescue an overloaded, ever-growing queue, because the stable-system assumption no longer holds.

Build the Workload Manifest

Create inference-capacity-pack/workload-manifest.yaml:

cutoff: 2026-09-11
model:
  id: dense-70b-class
  revision: dense-70b-fixture-r1
  tokenizer_revision: dense-70b-tokenizer-fixture-r1
  precision: bf16
engine:
  name: tutorial-engine-fixture
  version: "1.0"
  placement: {tp: 2, pp: 1, dp: 4, replicas: 4, gpus: 8}
  prefix_cache: enabled
  speculative_decoding: disabled
  disaggregated_prefill: disabled
client:
  location: loadgen-fixture-a/zone-a
  streaming: true
  arrival_pattern: poisson
  minimum_duration_seconds: 300
  initial_scheduled_requests: 22000
  request_schedule_seed: 20260911
  class_schedule_counts: {interactive: 15400, long_context: 4400, generation_heavy: 2200}
  warmup_seconds: 60
  maximum_duration_seconds: 3600
  request_timeout_seconds: 120
  automatic_retries: false
workload:
  - {name: interactive, mix: 0.70, input_tokens: 1024, output_tokens: 128,
     request_ttft_cutoff_ms: 500, request_tpot_cutoff_ms: 40}
  - {name: long_context, mix: 0.20, input_tokens: 8192, output_tokens: 256,
     request_ttft_cutoff_ms: 2500, request_tpot_cutoff_ms: 50}
  - {name: generation_heavy, mix: 0.10, input_tokens: 512, output_tokens: 1024,
     request_ttft_cutoff_ms: 800, request_tpot_cutoff_ms: 45}
acceptance:
  class_slo_attainment_min: 0.95
  min_successful_samples_per_class: 2000
  error_rate_max: 0.01
  queue_trend_required: flat
  target_utilization: 0.75

The request_*_cutoff_ms fields apply to individual requests; P50/P95/P99 are reported distribution statistics, not substitutes for those cutoffs. Generate the exact class counts, shuffle them with the recorded seed, then draw exponentially distributed inter-arrival times for the Poisson process. If any class has fewer than 2,000 successful rows after the initial schedule, append requests from the same versioned schedule before making a capacity decision.

In AIPerf, express a rate sweep with --request-rate, --arrival-pattern, --streaming, --benchmark-duration, and the frozen dataset, and keep any concurrency-only sweep as a separately named saturation experiment. AIPerf's CLI reference documents both controls, and its default export level writes per-request records for later analysis (AIPerf command-line options).

Build the SLO Analyzer

Normalize your load tool's per-request output into requests.csv with these columns:

request_id,class,ttft_ms,tpot_ms,e2e_ms,output_tokens,status
r-0001,interactive,182.0,25.1,3369.7,128,ok
r-0002,long_context,1940.0,44.2,13211.0,256,ok
r-0003,generation_heavy,910.0,47.0,48991.0,1024,ok
r-0004,interactive,0,0,0,0,error

Save this as analyze_slo.py:

import csv
import json
import math
import sys

REQUEST_CUTOFFS = {
    "interactive": {"ttft_ms": 500, "tpot_ms": 40},
    "long_context": {"ttft_ms": 2500, "tpot_ms": 50},
    "generation_heavy": {"ttft_ms": 800, "tpot_ms": 45},
}
CLASS_ATTAINMENT_MIN = 0.95
MIN_SUCCESSFUL_PER_CLASS = 2000
ERROR_RATE_MAX = 0.01

def percentile(values, probability):
    if not values:
        return None
    ordered = sorted(values)
    index = max(0, math.ceil(probability * len(ordered)) - 1)
    return ordered[index]

if len(sys.argv) != 4:
    raise SystemExit("usage: analyze_slo.py REQUESTS_CSV DURATION_SECONDS QUEUE_TREND")

path = sys.argv[1]
duration_s = float(sys.argv[2])
queue_trend = sys.argv[3]
if not math.isfinite(duration_s) or duration_s <= 0:
    raise ValueError("duration must be positive")
if queue_trend not in {"flat", "growing"}:
    raise ValueError("queue trend must be flat or growing")

with open(path, newline="", encoding="utf-8") as handle:
    rows = list(csv.DictReader(handle))

expected_outputs = {"interactive": 128, "long_context": 256, "generation_heavy": 1024}
seen_ids = set()
for row in rows:
    rid = row["request_id"]
    if not rid or rid in seen_ids:
        raise ValueError("request IDs must be nonempty and unique")
    seen_ids.add(rid)
    if row["class"] not in REQUEST_CUTOFFS:
        raise ValueError("unknown workload class")
    if row["status"] not in {"ok", "error", "timeout", "rejected", "cancelled"}:
        raise ValueError("every scheduled request needs a recognized terminal status")

attempted = len(rows)
successful = [row for row in rows if row["status"] == "ok"]
failed = attempted - len(successful)
passed_total = 0
by_class = {
    name: {"ttft": [], "tpot": [], "e2e": [], "tokens": 0, "passed": 0}
    for name in REQUEST_CUTOFFS
}

for row in successful:
    name = row["class"]
    if name not in REQUEST_CUTOFFS:
        raise ValueError(f"unknown workload class: {name}")
    ttft = float(row["ttft_ms"])
    tpot = float(row["tpot_ms"])
    e2e = float(row["e2e_ms"])
    output_tokens = int(row["output_tokens"])
    if not all(math.isfinite(x) and x >= 0 for x in (ttft, tpot, e2e)):
        raise ValueError("latencies must be finite and nonnegative")
    if output_tokens != expected_outputs[name]:
        raise ValueError("successful output length differs from the fixed workload")
    if abs(e2e - (ttft + tpot * (output_tokens - 1))) > 0.5:
        raise ValueError("TTFT/TPOT/E2E disagree under this fixture's metric convention")
    bucket = by_class[name]
    bucket["ttft"].append(ttft)
    bucket["tpot"].append(tpot)
    bucket["e2e"].append(e2e)
    bucket["tokens"] += output_tokens
    cutoff = REQUEST_CUTOFFS[name]
    if ttft <= cutoff["ttft_ms"] and tpot <= cutoff["tpot_ms"]:
        bucket["passed"] += 1
        passed_total += 1

classes = {}
for name, cutoff in sorted(REQUEST_CUTOFFS.items()):
    bucket = by_class[name]
    count = len(bucket["ttft"])
    attainment = bucket["passed"] / count if count else None
    sample_adequate = count >= MIN_SUCCESSFUL_PER_CLASS
    p99_sample_adequate = count >= 10000
    class_slo_pass = sample_adequate and attainment >= CLASS_ATTAINMENT_MIN
    classes[name] = {
        "successful_count": count,
        "passing_count": bucket["passed"],
        "request_ttft_cutoff_ms": cutoff["ttft_ms"],
        "request_tpot_cutoff_ms": cutoff["tpot_ms"],
        "attainment": attainment,
        "sample_adequate": sample_adequate,
        "p99_sample_adequate": p99_sample_adequate,
        "class_slo_pass": class_slo_pass,
        "p50_ttft_ms": percentile(bucket["ttft"], 0.50),
        "p95_ttft_ms": percentile(bucket["ttft"], 0.95),
        "p99_ttft_ms": percentile(bucket["ttft"], 0.99) if p99_sample_adequate else None,
        "p50_tpot_ms": percentile(bucket["tpot"], 0.50),
        "p95_tpot_ms": percentile(bucket["tpot"], 0.95),
        "p99_tpot_ms": percentile(bucket["tpot"], 0.99) if p99_sample_adequate else None,
        "p95_e2e_ms": percentile(bucket["e2e"], 0.95),
        "output_tokens": bucket["tokens"],
    }

error_rate = failed / attempted if attempted else None
overall_attainment = passed_total / len(successful) if successful else None
all_class_slos_pass = all(item["class_slo_pass"] for item in classes.values())
capacity_gate_pass = (
    all_class_slos_pass
    and error_rate is not None
    and error_rate <= ERROR_RATE_MAX
    and queue_trend == "flat"
)

report = {
    "attempted": attempted,
    "successful": len(successful),
    "failed": failed,
    "error_rate": error_rate,
    "overall_attainment": overall_attainment,
    "observed_goodput_rps": passed_total / duration_s,
    "request_throughput_rps": len(successful) / duration_s,
    "output_throughput_tps": sum(int(row["output_tokens"]) for row in successful) / duration_s,
    "queue_trend": queue_trend,
    "all_class_slos_pass": all_class_slos_pass,
    "capacity_gate_pass": capacity_gate_pass,
    "classes": classes,
}
print(json.dumps(report, indent=2, sort_keys=True))

Run:

python analyze_slo.py inference-capacity-pack/requests.csv 300 flat \
  > inference-capacity-pack/slo-report.json

Use a closed cohort: include every scheduled request, resolve each one to a terminal status, and include the final drain or timeout interval in DURATION_SECONDS. The 300 in the command above fits only the four-row smoke test; for a real point, pass the full cohort duration. Disable automatic retries for this fixture; otherwise record logical requests and attempts separately. Compare the configured arrival rate with actual dispatch timestamps and report client scheduling delay. The analyzer computes rates only from the rows and duration you supply. It cannot verify that your export contains every attempt or that the queue label you pass is true, so preserve the raw trace and queue time series.

The four-row CSV above is a schema smoke test, so its class samples are deliberately inadequate and capacity_gate_pass must be false. On a real point, the analyzer passes its preliminary capacity gate only when every named class has at least 2,000 successful rows and at least 95% joint request-level latency attainment among successful requests, the global error rate is at most 1%, and the queue trend is flat. Supply flat only when both the client scheduling backlog and the scheduler queue-depth time series show no sustained growth. Before calling a point capacity, reconcile CSV IDs against the schedule, validate the mix and run configuration, and review repeated runs. The script uses nearest-rank percentiles so the results are reproducible. Production tools may interpolate instead, so record the algorithm rather than comparing percentile labels blindly.

Observability: Measure the Work, State, and Failure

A production dashboard needs enough signals to explain why an SLO changed. Use bounded-cardinality dimensions such as model revision, engine version, workload class, replica, and result. Keep request IDs in traces/logs, not metric labels.

Request and SLO Signals

• attempted, successful, failed, timed-out, rejected, and cancelled requests;

• TTFT, TPOT, ITL, and E2E histograms by workload class;

• input/output tokens and request/output throughput;

• observed goodput and per-class SLO attainment; and

• retry count and client disconnects.

Scheduler and Queue Signals

• waiting and running requests;

• queue-wait histogram;

• batched prompt and generation tokens;

• preemption, eviction, and admission rejection counts; and

• per-iteration batch composition when available.

KV and GPU Signals

• KV blocks used/total and maximum token capacity;

• prefix-cache hit and eviction rates;

• HBM used, compute utilization, memory bandwidth, power, and OOM events; and

• model load and warmup duration.

Distributed Signals

• collective duration and errors by operation/message bucket;

• KV-transfer bytes, latency, throughput, and failures for disaggregated serving;

• per-replica inflight work and router selection; and

• speculative acceptance length/rate if speculative decoding is enabled.

NVIDIA Dynamo's engine metrics comparison lists the Prometheus metrics that vLLM, SGLang, and TensorRT-LLM expose and warns that metric names and counts change with engine versions. Inspect the deployed /metrics endpoint instead of copying an old dashboard unchanged (Dynamo Engine Metrics Comparison).

Deliberately Broken Benchmark

This configuration produces an impressive number and almost no operational evidence:

load_mode: concurrency_only
concurrency: 128
request_count: 100
prompts: repeat_the_same_prompt
prefix_cache: enabled
clear_cache_before_run: false
warmup_requests: 0
report: [mean_latency, total_tokens_per_second]
model_revision: latest
engine_version: latest

It is broken because:

1. concurrency-only traffic self-throttles and does not validate an independent arrival rate;

2. 100 requests provide almost no tail evidence;

3. repeated prompts and uncleared cache mix cold and warm states;

4. means hide tail latency and class failures;

5. errors, queue growth, cancellations, and cache state are absent; and

6. mutable latest versions prevent reproduction.

Adding one more percentile does not repair it. Replace it with the versioned manifest, separate cache states, both load modes, per-request records, class SLOs, and a failure test.

Failure Injection: Prove the Dashboard

In an isolated test deployment, run the service at the 20 RPS passing point, then terminate one of the four TP2 replicas. Record in failure-test.md:

injection_time
detection_time
router_removal_time
first_error_time and last_error_time
TTFT/TPOT/SLO-attainment during incident
queue peak and recovery time
requests retried, rejected, or lost
which alerts fired
whether capacity returned without manual data repair

Do not expect "no visible change": losing 25% of the replicas should reduce capacity. The system passes when it detects and removes the failed replica, stops routing new work to it, exposes errors or backpressure honestly, and returns to a stable state. By the linear estimate, one lost replica leaves only 18 RPS of capacity, or 13.5 RPS under the 75% utilization policy. If the remaining three replicas cannot sustain 20 RPS within the SLO, as that estimate predicts, admission control should reject or shed load rather than let the queue grow without bound.

Later, repeat the drill with a cold autoscaled replica, reduced KV capacity, request cancellation, and a degraded cross-node link. Each targets a different failure mechanism, so keep them as separate tests instead of collapsing them into one chaos score.

Hands-On Exercise

1. Generate or capture a workload matching the three-class manifest.

2. Run a closed-loop concurrency sweep to locate saturation.

3. Run cold-cache and warm-cache Poisson rate sweeps at 16, 20, 24, and 28 RPS with the fixed-seed 22,000-request schedule and at least five minutes per point.

4. Normalize every attempted request into requests.csv; continue the schedule until every class has at least 2,000 successful rows.

5. Produce slo-report.json and a plot of offered RPS versus raw throughput, observed goodput, per-class attainment, P95 TTFT, and P95 TPOT.

6. After reviewing schedule completeness, workload fidelity, cache state, queue evidence, and run-to-run uncertainty, select the highest tested rate at which the analyzer passes.

7. Calculate replica/GPU need for 36 RPS at 75% target utilization.

8. Inject the one-replica failure at a passing point and complete failure-test.md.

If a class has fewer than 2,000 successful samples, label its tail result exploratory and do not select that point as capacity. Stratification guarantees 2,200 initial attempts for the 10% class, and the analyzer still checks the successful count after errors. The 2,000 and 10,000 sample thresholds are tutorial reporting policies, not statistical guarantees. At 10,000 observations, only about 100 fall in the top 1%, and autocorrelation and bursty arrivals shrink the effective sample size further. Report run-to-run variation and a justified uncertainty estimate. The script withholds P99 below 10,000 successful observations, but it does not certify P99 precision above that threshold.

For the calculator-only path, run the four-row smoke test and expect attempted=4, successful=3, failed=1, error_rate=0.25, passing=2, observed_goodput=2/300 RPS, and capacity_gate_pass=false. A hardware-free exercise can validate the analysis and produce a failure-test plan, but it cannot claim measured capacity or a successful recovery.

Measurable Exit Criteria

The capacity pack passes when:

• model, tokenizer, engine, precision, scheduler, and workload revisions are immutable;

• cold and warm cache runs are separately named and report hit/eviction state;

• closed-loop saturation and open-loop arrival-rate sweeps are both present;

• every point reports attempted/successful/failed requests, error rate, P50/P95 latency, throughput, goodput, queue trend, and successful count per class;

• every class is evaluated separately against request-level TTFT/TPOT cutoffs and has at least 2,000 successful observations;

• the selected capacity point has at least 95% joint request-level latency attainment among successful requests in every class, at most 1% errors globally, and no sustained queue growth;

• the replica calculation includes the 75% utilization policy and returns 8 replicas/16 GPUs for the worked inputs;

• the failure test records detection and recovery times, and at least one configured alert fires;

• the dashboard can distinguish queue saturation, KV pressure, GPU failure, and network/transfer degradation; and

• repeated runs disclose variance instead of publishing only the best run.

Retrieval Practice

1. Why can a closed-loop test hide overload?

2. What is the difference between TPOT and ITL?

3. Why must cold-cache and warm-cache results be separate?

4. What makes observed goodput different from raw request throughput?

5. In the worked example, why is 24 offered RPS selected instead of 28?

6. How many replicas and GPUs are required for 36 RPS with the stated reserve?

7. Which signals distinguish a queue problem from KV-memory pressure?

8. What does a failure test prove that a healthy benchmark does not?

Answer Key

1. New work is issued only after prior work completes, so rising latency slows arrivals and self-throttles the client.

2. TPOT is one average decode time per request; ITL contains each observed gap between streamed outputs.

3. Prefix reuse can lower prefill work and inflate throughput; combining states without reporting their policy and proportions obscures the workload. A fixed, labeled mixture can be evaluated as its own experiment.

4. It counts only successful requests meeting their class request-level cutoffs per unit time; capacity additionally requires every class to pass its own attainment and sample-size gates.

5. The 28 RPS point has higher raw throughput, but its weakest class has only 78.0% attainment, its error rate is 10.0%, and its queue grows. At 24 RPS, every class has at least 95.1% attainment over at least 2,191 successful requests, and the other gates pass.

6. ceil(36/(6×0.75)) = 8 replicas; at TP2 that is 16 GPUs.

7. Queue depth/wait and admission metrics expose saturation; KV utilization, free blocks, eviction, and preemption expose KV pressure. GPU and request metrics provide corroboration.

8. It proves detection, routing removal, load shedding/retry behavior, alerting, and recovery under a known fault.

What This Post Does Not Cover

This tutorial defines an inference-operations method, not a universal SLO. Product teams must derive objectives from user needs and cost/risk constraints. The parameterized results are not OptiVerse measurements, and the linear replica calculation must be validated after scale-out. Quality evaluation, model accuracy, safety, and application correctness remain separate gates even when infrastructure SLOs pass.

Source Notes

• vLLM Project. "Benchmark CLI." Current documentation, accessed 2026-11-02. First-party definitions for client-observed TTFT, ITL, TPOT, throughput, and cache-warmth warnings. docs.vllm.ai

• NVIDIA. "AIPerf Load Generator Options Reference." Current documentation, accessed 2026-11-02. First-party definitions for rate-, concurrency-, trace-, and user-driven load schedules. docs.nvidia.com/aiperf

• NVIDIA. "AIPerf Command Line Options." Current documentation, accessed 2026-11-02. First-party controls for request-rate sweeps, Poisson/gamma/constant arrivals, concurrency, warmup, duration, and cancellation. docs.nvidia.com/aiperf

• Zhong, Y., Liu, S., Chen, J., et al. "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." OSDI, 2024. Primary source for SLO-conditioned goodput in LLM serving. usenix.org

• NVIDIA Dynamo. "Engine Metrics Comparison." Current documentation, accessed 2026-11-02. First-party cross-engine inventory for request, queue, KV, routing, transfer, and speculation metrics. docs.nvidia.com/dynamo

• NVIDIA Dynamo. "Size a Local Deployment with AIConfigurator." Current documentation, accessed 2026-11-02. First-party guidance to estimate candidate layouts and then validate them against a live endpoint. docs.nvidia.com/dynamo

• NVIDIA. "LLM Inference Benchmarking: Fundamental Concepts." April 2025. First-party discussion of metric definitions, workload lengths, and throughput-latency trade-offs. developer.nvidia.com