Inference Benchmarking, Capacity, SLOs, and Observability

An inference benchmark is not a race to print the largest tokens-per-second number. It is an experiment that asks whether a versioned system can sustain a stated workload while enough requests meet explicit latency and reliability objectives. Without the workload, arrival process, cache state, denominator, and percentile, a throughput number cannot answer a capacity question.

This tutorial turns the placement from I-06 into an operations artifact. You will freeze a mixed workload, run both saturation and rate-based tests, calculate request metrics and goodput, derive a replica requirement with headroom, define a dashboard, and inject a failure that should make the service fail safely and observably.

Prerequisites and Learning Outcomes

You should understand TTFT, ITL, TPOT, end-to-end latency, continuous batching, KV caching, and the TP2 × DP4 placement built in I-06.

By the end, you will be able to:

1. distinguish closed-loop concurrency tests from open-loop rate tests;

2. freeze a workload distribution instead of benchmarking one convenient prompt shape;

3. separate cold-cache and warm-cache results;

4. calculate TTFT, ITL, TPOT, end-to-end latency, throughput, observed goodput, and SLO-conditioned capacity;

5. derive replicas and GPUs from measured capacity plus reserve;

6. choose request, queue, KV, GPU, network, and failure metrics; and

7. prove that an alert and recovery path work under an injected failure.

What You Will Build

You will produce a versioned inference-capacity-pack/ with four inspectable files:

inference-capacity-pack/
├── workload-manifest.yaml
├── requests.csv
├── slo-report.json
└── failure-test.md

workload-manifest.yaml freezes the system and traffic. requests.csv holds one row per completed or failed request. slo-report.json contains percentiles, error rate, SLO attainment, and goodput. failure-test.md records one deliberately injected failure, the alerts it triggered, and recovery timing.

Freeze the Serving System

The worked example uses these hypothetical, fully specified assumptions:

Input — Assumption

--- — ---:

Model — dense-70b-fixture-r1, the dense BF16 model from I-06

Placement — 4 replicas, each TP2, on 8 GPUs

Engine and tokenizer — tutorial-engine-fixture-1.0 and dense-70b-tokenizer-fixture-r1

Streaming — enabled

Prefix caching — enabled, but cold and warm runs are separate

Speculative decoding — disabled

P/D disaggregation — disabled

Measurement point — client send to client receive

Primary SLO attainment target — in every class independently, at least 95% of successful requests meet both request-level TTFT and TPOT cutoffs

Maximum error rate — 1%

Capacity reserve — operate at no more than 75% of measured SLO-compliant capacity

The engine and tokenizer revisions matter because metric behavior and token counts can change across versions. Current vLLM documentation also warns that metric terminology is not standardized across tools, so formulas and measurement points must accompany names (vLLM Benchmark CLI).

Freeze the Workload Distribution

A single average input length destroys the distinction between prefill-heavy and decode-heavy traffic. Use explicit classes:

Class — Mix — Input tokens — Output tokens — Request TTFT cutoff — Request TPOT cutoff

--- — ---: — ---: — ---: — ---: — ---:

interactive — 70% — 1,024 — 128 — 500 ms — 40 ms/token

long_context — 20% — 8,192 — 256 — 2,500 ms — 50 ms/token

generation_heavy — 10% — 512 — 1,024 — 800 ms — 45 ms/token

For this fixed mix:

weighted input tokens
  = 0.70×1,024 + 0.20×8,192 + 0.10×512
  = 2,406.4 tokens/request

weighted output tokens
  = 0.70×128 + 0.20×256 + 0.10×1,024
  = 243.2 tokens/request

Those weighted values help validate aggregate counters, but SLO attainment must still be evaluated independently for every class. An aggregate percentile or attainment ratio can look healthy while the lower-volume long-context class is unhealthy.

The manifest should also freeze sampling parameters, stop behavior, chat template, prefix distribution, request cancellation policy, benchmark duration, client location, exact model/engine configuration, and the random seed or request schedule. NVIDIA's current AIPerf documentation supports explicit constant, Poisson, gamma, fixed-schedule, and concurrency-burst arrival patterns; the chosen arrival model is part of the experiment, not a hidden client detail (AIPerf load-generator options).

Closed Loop and Open Loop Answer Different Questions

Closed-Loop Concurrency Test

A closed-loop test keeps up to N requests in flight. When one completes, the client sends another. As latency rises, completion slows and therefore new arrivals slow too. This is useful for finding saturation and the throughput-latency frontier, but it self-throttles and does not prove that a service can absorb an independent production arrival rate.

Current AIPerf calls concurrency-only mode a burst or saturation mode: requests are issued as fast as possible within a concurrency ceiling (AIPerf arrival patterns).

Open-Loop Rate Test

An open-loop test schedules arrivals at a target rate independent of completion, often using a Poisson or trace-derived process. If offered load exceeds service capacity, the queue grows and TTFT exposes the overload. This is the appropriate test for an external arrival-rate SLO, provided the load generator does not silently block scheduled arrivals behind a low concurrency cap.

Use both:

1. closed-loop concurrency sweep to discover saturation candidates;

2. open-loop rate sweep to validate queue stability and SLO attainment at realistic arrival rates; and

3. timestamped trace replay when production arrival and length distributions are available.

Define the Metrics Before Running the Test

For request i, with client-send time s_i, first non-empty token time f_i, final token time e_i, and O_i output tokens:

TTFT_i = f_i - s_i
E2E_i  = e_i - s_i
TPOT_i = (E2E_i - TTFT_i) / (O_i - 1), for O_i > 1

If token arrival timestamps are t_i,1 ... t_i,O, individual inter-token latencies are:

ITL_i,j = t_i,j - t_i,j-1, for j = 2 ... O_i

TPOT is one per-request average; ITL is a distribution of individual gaps. They are similar during ordinary one-token streaming but can diverge when one streamed chunk contains multiple tokens, including under speculative decoding. Current vLLM benchmarking documentation makes that distinction explicit (vLLM Benchmark CLI).

For a measurement window of duration T:

request_throughput = successful_requests / T
output_throughput  = sum(successful_output_tokens) / T
error_rate         = failed_requests / attempted_requests
observed_goodput   = successful_requests_meeting_their_class_request_cutoffs / T
class_attainment_c = passing_successful_requests_in_class_c
                     / successful_requests_in_class_c
all_class_SLOs_pass = AND over every class c of:
                      class_attainment_c >= 0.95
                      AND successful_requests_in_class_c >= 2,000

This tutorial uses capacity goodput to mean the highest offered request rate at which the service simultaneously has:

• at least 95% request-level SLO attainment in every workload class independently;

• at least 2,000 successful observations in every class;

• no more than 1% errors; and

• no sustained queue growth during the steady-state window.

Overall observed goodput remains useful as a throughput counter, but it cannot make a capacity point pass when one class fails. The per-class gate prevents the 70% interactive class from hiding a regression in the 20% or 10% classes.

DistServe uses goodput to connect arrival rate with simultaneous TTFT and TPOT constraints rather than counting raw throughput alone; its reported gains remain specific to its evaluated models, workloads, and SLOs (Zhong et al., OSDI 2024).

Control Cache State Instead of Averaging It Away

Run two named experiment families:

Cold Cache

• restart or explicitly clear the engine cache before the point;

• use unique prefixes when clearing is unavailable;

• exclude compilation and model-load warmup from the steady-state window; and

• report cache hit rate near zero.

Warm Cache

• prewarm a versioned set of repeated prefixes;

• preserve the same prefix-frequency distribution across configurations;

• begin measurement only after the intended cache state exists; and

• report prefix-cache hit rate and eviction rate.

Never combine cold and warm requests into one unlabeled average. vLLM's current benchmark guide warns that rerunning a reproducible dataset against the same server can reuse prefixes left in cache and inflate throughput (vLLM Benchmark CLI).

Worked Example: Find SLO-Compliant Capacity

Assume a fixed-seed Poisson-arrival schedule of 22,000 requests at each point, stratified into exactly 15,400 interactive, 4,400 long-context, and 2,200 generation-heavy requests before shuffling. Measure for at least five minutes and continue the precommitted schedule if any class finishes with fewer than 2,000 successful observations. Keep the cache policy stable. The following results are parameterized tutorial data, not measurements from OptiVerse infrastructure:

Offered RPS — Completed RPS — Error rate — Overall attainment — Minimum class attainment — Minimum class successful n — Observed goodput RPS — Output tok/s — Interactive P95 TTFT — Interactive P95 TPOT — Queue trend

---: — ---: — ---: — ---: — ---: — ---: — ---: — ---: — ---: — ---: — ---

16 — 16.0 — 0.0% — 99.6% — 99.1% — 2,200 — 15.94 — 3,891 — 180 ms — 24 ms — flat

20 — 20.0 — 0.1% — 99.1% — 98.2% — 2,197 — 19.82 — 4,864 — 260 ms — 29 ms — flat

24 — 23.9 — 0.4% — 95.4% — 95.1% — 2,191 — 22.80 — 5,812 — 420 ms — 38 ms — flat

28 — 25.2 — 2.0% — 86.8% — 78.0% — 2,156 — 21.87 — 6,129 — 980 ms — 62 ms — growing

The 28 RPS point has the largest raw output throughput but lower observed goodput than the 24 RPS point. Its weakest class has only 78.0% attainment, it violates both interactive latency objectives, its error rate is too high, and its queue grows. Raw throughput alone would select the wrong operating point.

The highest offered rate for which every class has at least 95% attainment and 2,000 successful observations, while the overall error rate remains at most 1% and the queue stays flat, is 24 RPS for four replicas. Its weakest class has 95.1% attainment over 2,191 successful requests. Therefore:

measured SLO-compliant capacity per replica
  = 24 RPS / 4 replicas
  = 6 RPS/replica

This division is valid only for the same workload mix, placement, cache state, engine configuration, and scaling regime. It is not a universal per-replica constant.

Worked Capacity Plan With Headroom

Suppose forecast peak load is 36 RPS and policy reserves 25% capacity, so planned utilization is at most 75%:

required_replicas
  = ceil(peak_RPS / (capacity_per_replica × target_utilization))
  = ceil(36 / (6 × 0.75))
  = ceil(8)
  = 8 replicas

required_GPUs = 8 replicas × 2 GPUs/replica = 16 GPUs

The current eight-GPU cluster has four replicas, so it cannot carry the forecast peak with the requested reserve. The calculation does not authorize procurement by itself: validate the eight-replica topology because load-balancer behavior, network paths, cache locality, and shared bottlenecks can prevent linear scaling. Current Dynamo sizing documentation makes the same boundary explicit: its analytical optimizer proposes configurations but does not simulate request-by-request scheduler or KV-cache behavior, so a live benchmark is still required (Dynamo sizing with AIConfigurator).

Little's Law supplies a consistency check for a stable window:

average_in_flight ≈ completion_rate × average_E2E_seconds

At 23.9 completed RPS and 1.8 seconds mean E2E, the expected average in-flight count is 23.9 × 1.8 = 43.02. A wildly different client/server in-flight gauge signals a boundary mismatch, dropped telemetry, retries, or an unstable queue. Little's Law does not rescue an overloaded, ever-growing queue because the stable-system assumption has failed.

Build the Workload Manifest

Create inference-capacity-pack/workload-manifest.yaml:

cutoff: 2026-08-29
model:
  id: dense-70b-class
  revision: dense-70b-fixture-r1
  tokenizer_revision: dense-70b-tokenizer-fixture-r1
  precision: bf16
engine:
  name: tutorial-engine-fixture
  version: "1.0"
  placement: {tp: 2, pp: 1, dp: 4, replicas: 4, gpus: 8}
  prefix_cache: enabled
  speculative_decoding: disabled
  disaggregated_prefill: disabled
client:
  location: loadgen-fixture-a/zone-a
  streaming: true
  arrival_pattern: poisson
  minimum_duration_seconds: 300
  initial_scheduled_requests: 22000
  request_schedule_seed: 20260829
  class_schedule_counts: {interactive: 15400, long_context: 4400, generation_heavy: 2200}
  warmup_seconds: 60
workload:
  - {name: interactive, mix: 0.70, input_tokens: 1024, output_tokens: 128,
     request_ttft_cutoff_ms: 500, request_tpot_cutoff_ms: 40}
  - {name: long_context, mix: 0.20, input_tokens: 8192, output_tokens: 256,
     request_ttft_cutoff_ms: 2500, request_tpot_cutoff_ms: 50}
  - {name: generation_heavy, mix: 0.10, input_tokens: 512, output_tokens: 1024,
     request_ttft_cutoff_ms: 800, request_tpot_cutoff_ms: 45}
acceptance:
  class_slo_attainment_min: 0.95
  min_successful_samples_per_class: 2000
  error_rate_max: 0.01
  queue_trend_required: flat
  target_utilization: 0.75

The request_*_cutoff_ms fields apply to individual requests; P50/P95/P99 remain reported distribution statistics and are not interchangeable with those cutoffs. Generate the exact class counts, shuffle them with the recorded seed, then assign Poisson arrival intervals. If any class has fewer than 2,000 successful rows after the initial schedule, append from the same versioned schedule before making a capacity decision.

If you use AIPerf, a rate sweep can be expressed with --request-rate, a chosen arrival pattern, streaming, duration, and the frozen dataset. A concurrency-only sweep should be a separately named saturation experiment. Current AIPerf documentation exposes both controls and writes artifacts for later analysis (AIPerf command-line options).

Build the SLO Analyzer

Normalize your load tool's per-request output into requests.csv with these columns:

request_id,class,ttft_ms,tpot_ms,e2e_ms,output_tokens,status
r-0001,interactive,182.0,25.1,3370.0,128,ok
r-0002,long_context,1940.0,44.2,13211.2,256,ok
r-0003,generation_heavy,910.0,47.0,48991.0,1024,ok
r-0004,interactive,0,0,0,0,error

Save this as analyze_slo.py:

import csv
import json
import math
import sys

REQUEST_CUTOFFS = {
    "interactive": {"ttft_ms": 500, "tpot_ms": 40},
    "long_context": {"ttft_ms": 2500, "tpot_ms": 50},
    "generation_heavy": {"ttft_ms": 800, "tpot_ms": 45},
}
CLASS_ATTAINMENT_MIN = 0.95
MIN_SUCCESSFUL_PER_CLASS = 2000
ERROR_RATE_MAX = 0.01

def percentile(values, probability):
    if not values:
        return None
    ordered = sorted(values)
    index = max(0, math.ceil(probability * len(ordered)) - 1)
    return ordered[index]

if len(sys.argv) != 4:
    raise SystemExit("usage: analyze_slo.py REQUESTS_CSV DURATION_SECONDS QUEUE_TREND")

path = sys.argv[1]
duration_s = float(sys.argv[2])
queue_trend = sys.argv[3]
if duration_s <= 0:
    raise ValueError("duration must be positive")
if queue_trend not in {"flat", "growing"}:
    raise ValueError("queue trend must be flat or growing")

with open(path, newline="", encoding="utf-8") as handle:
    rows = list(csv.DictReader(handle))

attempted = len(rows)
successful = [row for row in rows if row["status"] == "ok"]
failed = attempted - len(successful)
passed_total = 0
by_class = {
    name: {"ttft": [], "tpot": [], "e2e": [], "tokens": 0, "passed": 0}
    for name in REQUEST_CUTOFFS
}

for row in successful:
    name = row["class"]
    if name not in REQUEST_CUTOFFS:
        raise ValueError(f"unknown workload class: {name}")
    ttft = float(row["ttft_ms"])
    tpot = float(row["tpot_ms"])
    e2e = float(row["e2e_ms"])
    output_tokens = int(row["output_tokens"])
    bucket = by_class[name]
    bucket["ttft"].append(ttft)
    bucket["tpot"].append(tpot)
    bucket["e2e"].append(e2e)
    bucket["tokens"] += output_tokens
    cutoff = REQUEST_CUTOFFS[name]
    if ttft <= cutoff["ttft_ms"] and tpot <= cutoff["tpot_ms"]:
        bucket["passed"] += 1
        passed_total += 1

classes = {}
for name, cutoff in sorted(REQUEST_CUTOFFS.items()):
    bucket = by_class[name]
    count = len(bucket["ttft"])
    attainment = bucket["passed"] / count if count else None
    sample_adequate = count >= MIN_SUCCESSFUL_PER_CLASS
    p99_sample_adequate = count >= 10000
    class_slo_pass = sample_adequate and attainment >= CLASS_ATTAINMENT_MIN
    classes[name] = {
        "successful_count": count,
        "passing_count": bucket["passed"],
        "request_ttft_cutoff_ms": cutoff["ttft_ms"],
        "request_tpot_cutoff_ms": cutoff["tpot_ms"],
        "attainment": attainment,
        "sample_adequate": sample_adequate,
        "p99_sample_adequate": p99_sample_adequate,
        "class_slo_pass": class_slo_pass,
        "p50_ttft_ms": percentile(bucket["ttft"], 0.50),
        "p95_ttft_ms": percentile(bucket["ttft"], 0.95),
        "p99_ttft_ms": percentile(bucket["ttft"], 0.99) if p99_sample_adequate else None,
        "p50_tpot_ms": percentile(bucket["tpot"], 0.50),
        "p95_tpot_ms": percentile(bucket["tpot"], 0.95),
        "p99_tpot_ms": percentile(bucket["tpot"], 0.99) if p99_sample_adequate else None,
        "p95_e2e_ms": percentile(bucket["e2e"], 0.95),
        "output_tokens": bucket["tokens"],
    }

error_rate = failed / attempted if attempted else None
overall_attainment = passed_total / len(successful) if successful else None
all_class_slos_pass = all(item["class_slo_pass"] for item in classes.values())
capacity_gate_pass = (
    all_class_slos_pass
    and error_rate is not None
    and error_rate <= ERROR_RATE_MAX
    and queue_trend == "flat"
)

report = {
    "attempted": attempted,
    "successful": len(successful),
    "failed": failed,
    "error_rate": error_rate,
    "overall_attainment": overall_attainment,
    "observed_goodput_rps": passed_total / duration_s,
    "request_throughput_rps": len(successful) / duration_s,
    "output_throughput_tps": sum(int(row["output_tokens"]) for row in successful) / duration_s,
    "queue_trend": queue_trend,
    "all_class_slos_pass": all_class_slos_pass,
    "capacity_gate_pass": capacity_gate_pass,
    "classes": classes,
}
print(json.dumps(report, indent=2, sort_keys=True))

Run:

python analyze_slo.py inference-capacity-pack/requests.csv 300 flat \
  > inference-capacity-pack/slo-report.json

The four-row CSV above is a schema smoke test, so its class samples are deliberately inadequate and capacity_gate_pass must be false. On a real point, the analyzer will not pass capacity unless every named class has at least 2,000 successful rows and at least 95% joint request-level attainment, the error rate is at most 1%, and the queue trend is flat. Supply flat only when a scheduler queue-depth time series shows no sustained growth. The script uses a nearest-rank percentile so the result is reproducible. Production tools may interpolate differently; record the algorithm rather than comparing percentile labels blindly.

Observability: Measure the Work, State, and Failure

A production dashboard needs enough signals to explain why an SLO changed. Use bounded-cardinality dimensions such as model revision, engine version, workload class, replica, and result. Keep request IDs in traces/logs, not metric labels.

Request and SLO Signals

• attempted, successful, failed, timed-out, rejected, and cancelled requests;

• TTFT, TPOT, ITL, and E2E histograms by workload class;

• input/output tokens and request/output throughput;

• observed goodput and per-class SLO attainment; and

• retry count and client disconnects.

Scheduler and Queue Signals

• waiting and running requests;

• queue-wait histogram;

• batched prompt and generation tokens;

• preemption, eviction, and admission rejection counts; and

• per-iteration batch composition when available.

KV and GPU Signals

• KV blocks used/total and maximum token capacity;

• prefix-cache hit and eviction rates;

• HBM used, compute utilization, memory bandwidth, power, and OOM events; and

• model load and warmup duration.

Distributed Signals

• collective duration and errors by operation/message bucket;

• KV-transfer bytes, latency, throughput, and failures for disaggregated serving;

• per-replica inflight work and router selection; and

• speculative acceptance length/rate if speculative decoding is enabled.

Current NVIDIA Dynamo documentation compares the Prometheus signals available from vLLM, SGLang, and TensorRT-LLM and warns that metric names vary by engine version; inspect the deployed /metrics endpoint rather than copying an old dashboard unmodified (Dynamo Engine Metrics Comparison).

Deliberately Broken Benchmark

This configuration produces an impressive number and almost no operational evidence:

load_mode: concurrency_only
concurrency: 128
request_count: 100
prompts: repeat_the_same_prompt
prefix_cache: enabled
clear_cache_before_run: false
warmup_requests: 0
report: [mean_latency, total_tokens_per_second]
model_revision: latest
engine_version: latest

It is broken because:

1. concurrency-only traffic self-throttles and does not validate an independent arrival rate;

2. 100 requests provide almost no tail evidence;

3. repeated prompts and uncleared cache mix cold and warm states;

4. means hide tail latency and class failures;

5. errors, queue growth, cancellations, and cache state are absent; and

6. mutable latest versions prevent reproduction.

The repair is not to add one more percentile. Replace it with the versioned manifest, separate cache states, both load modes, per-request records, class SLOs, and a failure test.

Failure Injection: Prove the Dashboard

Run the service at the 20 RPS passing point, then terminate one of the four TP2 replicas. Record in failure-test.md:

injection_time
detection_time
router_removal_time
first_error_time and last_error_time
TTFT/TPOT/SLO-attainment during incident
queue peak and recovery time
requests retried, rejected, or lost
which alerts fired
whether capacity returned without manual data repair

Expected behavior is not “no visible change.” Losing 25% of replicas should reduce capacity. The system passes when it detects and removes the replica, does not route new work to it, exposes errors or backpressure honestly, and returns to a stable state. If the remaining three replicas cannot sustain 20 RPS under the SLO, admission control should reject or shed load rather than let the queue grow without bound.

Repeat later with a cold autoscaled replica, reduced KV capacity, request cancellation, and a degraded cross-node link. Those tests target different failure mechanisms and should not be collapsed into one chaos score.

Hands-On Exercise

1. Generate or capture a workload matching the three-class manifest.

2. Run a closed-loop concurrency sweep to locate saturation.

3. Run cold-cache and warm-cache Poisson rate sweeps at 16, 20, 24, and 28 RPS with the fixed-seed 22,000-request schedule and at least five minutes per point.

4. Normalize every attempted request into requests.csv; continue the schedule until every class has at least 2,000 successful rows.

5. Produce slo-report.json and a plot of offered RPS versus raw throughput, observed goodput, per-class attainment, P95 TTFT, and P95 TPOT.

6. Select the highest rate whose analyzer reports capacity_gate_pass: true.

7. Calculate replica/GPU need for 36 RPS at 75% target utilization.

8. Inject the one-replica failure at a passing point and complete failure-test.md.

If a class has fewer than 2,000 successful samples, label its tail result exploratory and do not select that point as capacity. Stratification guarantees 2,200 initial attempts for the 10% class, while the analyzer still checks the successful count after errors. Do not report a precise P99 for a class with fewer than 10,000 successful observations; that would leave fewer than about 100 observations in the top 1%.

Measurable Exit Criteria

The capacity pack passes when:

• model, tokenizer, engine, precision, scheduler, and workload revisions are immutable;

• cold and warm cache runs are separately named and report hit/eviction state;

• closed-loop saturation and open-loop arrival-rate sweeps are both present;

• every point reports attempted/completed/failed requests, error rate, P50/P95 latency, throughput, goodput, queue trend, and successful count per class;

• every class is evaluated separately against request-level TTFT/TPOT cutoffs and has at least 2,000 successful observations;

• the selected capacity point has at least 95% joint request-level attainment in every class, at most 1% errors, and no sustained queue growth;

• the replica calculation includes the 75% utilization policy and returns 8 replicas/16 GPUs for the worked inputs;

• the failure test records detection and recovery times, and at least one configured alert fires;

• the dashboard can distinguish queue saturation, KV pressure, GPU failure, and network/transfer degradation; and

• repeated runs disclose variance instead of publishing only the best run.

Retrieval Practice

1. Why can a closed-loop test hide overload?

2. What is the difference between TPOT and ITL?

3. Why must cold-cache and warm-cache results be separate?

4. What makes observed goodput different from raw request throughput?

5. In the worked example, why is 24 offered RPS selected instead of 28?

6. How many replicas and GPUs are required for 36 RPS with the stated reserve?

7. Which signals distinguish a queue problem from KV-memory pressure?

8. What does a failure test prove that a healthy benchmark does not?

Answer Key

1. New work is issued only after prior work completes, so rising latency slows arrivals and self-throttles the client.

2. TPOT is one average decode time per request; ITL contains each observed gap between streamed outputs.

3. Prefix reuse can lower prefill work and inflate throughput; combining states makes the result irreproducible and workload meaning unclear.

4. It counts only successful requests meeting their class request-level cutoffs per unit time; capacity additionally requires every class to pass its own attainment and sample-size gates.

5. The 28 RPS point has higher raw throughput, but its weakest class has only 78.0% attainment, its error rate is 2.0%, and its queue grows. At 24 RPS, every class has at least 95.1% attainment over at least 2,191 successful requests, and the other gates pass.

6. ceil(36/(6×0.75)) = 8 replicas; at TP2 that is 16 GPUs.

7. Queue depth/wait and admission metrics expose saturation; KV utilization, free blocks, eviction, and preemption expose KV pressure. GPU and request metrics provide corroboration.

8. It proves detection, routing removal, load shedding/retry behavior, alerting, and recovery under a known fault.

What This Post Does Not Cover

This tutorial defines an inference-operations method, not a universal SLO. Product teams must derive objectives from user needs and cost/risk constraints. The parameterized results are not OptiVerse measurements, and the linear replica calculation must be validated after scale-out. Quality evaluation, model accuracy, safety, and application correctness remain separate gates even when infrastructure SLOs pass.

Source Notes

• vLLM Project. “Benchmark CLI.” Current documentation, accessed 2026-08-29. First-party definitions for client-observed TTFT, ITL, TPOT, throughput, and cache-warmth warnings. docs.vllm.ai

• NVIDIA. “AIPerf Load Generator Options Reference.” Current documentation, accessed 2026-08-29. First-party definitions for rate-, concurrency-, trace-, and user-driven load schedules. docs.nvidia.com/aiperf

• NVIDIA. “AIPerf Command Line Options.” Current documentation, accessed 2026-08-29. First-party controls for request-rate sweeps, Poisson/gamma/constant arrivals, concurrency, warmup, duration, and cancellation. docs.nvidia.com/aiperf

• Zhong, Y., Liu, S., Chen, J., et al. “DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving.” OSDI, 2024. Primary source for SLO-conditioned goodput in LLM serving. usenix.org

• NVIDIA Dynamo. “Engine Metrics Comparison.” Current documentation, accessed 2026-08-29. First-party cross-engine inventory for request, queue, KV, routing, transfer, and speculation metrics. docs.nvidia.com/dynamo

• NVIDIA Dynamo. “Size a Local Deployment with AIConfigurator.” Current documentation, accessed 2026-08-29. First-party guidance to estimate candidate layouts and then validate them against a live endpoint. docs.nvidia.com/dynamo

• NVIDIA. “LLM Inference Benchmarking: Fundamental Concepts.” April 2025. First-party discussion of metric definitions, workload lengths, and throughput-latency trade-offs. developer.nvidia.com