Inside an LLM Inference Request

One API call hides an entire serving pipeline: request validation, tokenization, admission control, scheduling, model execution, sampling, streaming, and telemetry. This tutorial traces that pipeline for a decoder-only Transformer using full causal attention. That scope matters because multi-query attention (MQA), grouped-query attention (GQA), multi-head latent attention (MLA), sliding or sparse attention, hybrid state-space layers, and speculative multi-token decoding each change parts of the memory or timing math below.

Rather than memorizing a single provider's latency numbers, you will build one request timeline whose KV memory, TTFT, TPOT, ITL, end-to-end latency, and token rates all reconcile from explicit assumptions.

Scope and Running Request

The travel copilot submits this request:

Show me wheelchair-accessible hotels in Kyoto under $200/night with onsen access.

The prompt contains:

The model assumptions are:

Separating query heads from KV heads is essential here. GQA shares each key/value head across a group of eight query heads, so plugging query_heads=64 into the KV formula would overstate cache size by 8×. Inference engineering guides likewise separate MHA, MQA, and GQA because they carry very different KV footprints (NVIDIA inference optimization).

The Request Lifecycle

1. Gateway and Request Normalization

The serving edge authenticates the caller, enforces rate and payload limits, validates model and sampling parameters, assigns a request ID, and records an arrival timestamp. All of that work adds to client-observed TTFT before the model ever runs.

2. Chat Template and Tokenization

The server renders the API messages through a pinned chat template and tokenizer. Token IDs fix the text sequence, but valid KV reuse also requires matching model weights, adapters, positions, masks, and any multimodal inputs. Billing follows the provider's own token-accounting categories, so equal token IDs alone establish neither equal price nor reusable state.

3. Admission, Routing, and Queueing

A global router selects a compatible replica, and the engine scheduler admits the request only when token, sequence, and KV-memory budgets allow it. Otherwise the request waits in a queue or fails fast under backpressure. Any queue wait sits directly inside TTFT.

4. Prefill

During prefill, the model processes all 1,025 input tokens and materializes K and V states for every attention layer. Causal masking preserves autoregressive semantics even though prompt positions can be processed in parallel. In this text-only fixture, with no hidden reasoning tokens or tool calls, the final prompt position produces the logits for the first visible output token. Other generation modes, such as hidden reasoning or tool calling, can do additional work before the first visible output.

Prefill typically has high arithmetic intensity and leans compute-bound, whereas low-batch decode usually has low arithmetic intensity and leans memory-bandwidth-bound. Treat those as roofline tendencies rather than fixed laws: prompt length, batch size, model architecture, kernel implementation, and hardware can shift either phase across the compute-bandwidth boundary.

5. Decode, Sampling, and Streaming

On each subsequent decode step, the engine feeds in the previously selected token, reads the retained KV state, computes new logits, samples or selects the next token, and streams the decoded text chunk. Full causal attention reads a prefix that grows by one token per step; sliding-window, sparse, recurrent, or hybrid layers change that access pattern.

When generation stops immediately on the final sampled token, the engine normally does not need that last token's KV state for another step, which makes exact end-of-request cache counts implementation-dependent. Our calculation below sizes the cache after 149 output tokens have been processed to emit 150 visible output tokens.

6. Completion and Telemetry

At completion, the server emits token-usage and finish-reason metadata, releases request-owned KV blocks unless a prefix-cache policy retains them, and logs latency, token counts, queue wait, cache hits, errors, and cancellations. When a client disconnects mid-stream, the serving stack should propagate cancellation, where the engine supports it, so the GPU stops spending decode steps on abandoned output.

Attention and the KV Cache

At each token position, an attention layer projects the hidden state into query, key, and value vectors. Under full causal self-attention, a query attends to the keys at every earlier position and its own position, then takes a weighted sum of the corresponding values, following the scaled dot-product Q/K/V formulation in the original Transformer paper (Vaswani et al., 2017).

Without a KV cache, every decode step would recompute K and V projections across all preceding tokens. Caching those K/V tensors lets each decode step project only the newly fed token. Keep the two costs distinct: caching eliminates repeated K/V projection work on prior tokens, while causal attention still reads the cached prefix and computes attention scores that grow with sequence length.

Reproducible KV Memory Formula

For standard MHA or GQA storage with one K tensor and one V tensor per KV head:

KV_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element
KV_bytes(sequence)  = KV_bytes_per_token × sequence_tokens

For our 80-layer GQA model:

KV bytes/token = 2 × 80 × 8 × 128 × 2
               = 327,680 bytes
               = 0.3125 MiB

Prompt KV       = 327,680 × 1,025
               = 335,872,000 bytes
               = 320.3125 MiB

KV after 149 processed output tokens
               = 327,680 × (1,025 + 149)
               = 384,696,320 bytes
               = 366.875 MiB

These figures measure the unsharded logical K/V payload rather than total GPU memory allocation. Physical memory also depends on KV dtype, quantization scales, padding and block rounding, workspace buffers, and how tensor parallelism shards or replicates KV heads and pipeline parallelism splits layers across devices. Architectures with different retained state, including MLA, sliding-window, recurrent, or hybrid layers, require deriving their own state formulas rather than substituting into kv_heads.

Metric Definitions

TTFT is time to first token, TPOT is average time per output token after the first, ITL is inter-token latency, and E2E is end-to-end request latency. For this fixture, let s be the client send timestamp, f be the arrival of the first non-empty visible output, e be the arrival of the final visible output, and O be the visible output token count:

TTFT = f - s
E2E  = e - s
TPOT = (E2E - TTFT) / (O - 1), for O > 1

ITL measures each individual gap between consecutive streamed outputs, whereas TPOT collapses post-first-token generation into one request-level average. When each streamed event carries exactly one token, TPOT equals the arithmetic mean of the request's ITLs, even if individual gaps jitter widely around that mean. Speculative decoding can emit multiple accepted tokens in one step, causing per-chunk ITL and per-token TPOT to diverge. vLLM's benchmarking documentation therefore stresses checking exact formulas and measurement points instead of comparing metric names at face value (vLLM Benchmark CLI).

Two per-request token rates answer different questions:

decode-only output rate = (O - 1) / (E2E - TTFT)
whole-request output rate = O / E2E

System output throughput divides total counted output tokens by a measurement-window duration. A valid report states whether it counts token events inside a steady-state window or a closed cohort of completed requests, how it treats requests crossing window boundaries, and whether it excludes tokens from failed requests. Goodput counts only requests that meet stated SLOs per unit time. Rates computed under different counting rules are not interchangeable.

In a separate synthetic 10-second closed-cohort window, suppose all 100 attempted requests finish inside the window: 98 succeed, 2 fail, and 95 successful requests meet the latency SLOs, with no boundary-crossing work:

request throughput = 98 / 10 = 9.8 requests/s
observed goodput   = 95 / 10 = 9.5 requests/s
error rate         = 2 / 100 = 2%

Worked Example: Reconcile the Timeline

Create request-timeline.csv:

event,time_ms,detail
client_send,0,request leaves application
server_arrival,5,gateway has request
scheduler_admit,45,prefill begins
first_visible_token,190,token 1 of 150
last_visible_token,3915,token 150 of 150

Because this endpoint CSV records only the first and last token events, it determines average TPOT without revealing per-step ITL variance. To build a closed-form teaching trace, assume one visible token per event and 149 uniform gaps of 25 ms on a single synthetic reference clock (in production, subtracting client and server timestamps across hosts requires clock synchronization). E2E ends at the final visible token, before any trailing finish metadata:

last token time = 190 + 149 × 25
                = 3,915 ms

TTFT            = 190 - 0
                = 190 ms

E2E             = 3,915 - 0
                = 3,915 ms

TPOT            = (3,915 - 190) / (150 - 1)
                = 3,725 / 149
                = 25 ms/token

decode-only rate = 149 / 3.725
                 = 40.00 tokens/s

whole-request rate = 150 / 3.915
                   = 38.31 tokens/s

The 40.00 tokens/s rate isolates post-first-token streaming cadence, whereas 38.31 tokens/s folds in network transit, queueing, and prefill before the first token arrives. NVIDIA's benchmarking guide draws the same boundary between TTFT, ITL/TPOT, per-user output rate, and aggregate system throughput, although it defines ITL as the average gap between consecutive tokens and treats it as another name for TPOT (NVIDIA benchmarking definitions).

Build the Metric Calculator

Save the script below as request_metrics.py alongside request-timeline.csv, then run python request_metrics.py > request-metrics.json. The script computes metrics directly from the endpoint timestamps, leaving uniform 25 ms ITLs as an external assumption of the synthetic trace:

import csv
import json
from pathlib import Path

layers, kv_heads, head_dim, bytes_per_element = 80, 8, 128, 2
input_tokens, output_tokens = 1025, 150
window_seconds, attempted_requests = 10, 100
successful_requests, slo_passing_requests = 98, 95

with Path("request-timeline.csv").open(newline="") as source:
    rows = list(csv.DictReader(source))
if len({row["event"] for row in rows}) != len(rows):
    raise ValueError("duplicate event")
times = {row["event"]: float(row["time_ms"]) for row in rows}
s, f, e = (times[key] for key in
           ("client_send", "first_visible_token", "last_visible_token"))
if not (s <= f <= e) or output_tokens < 1:
    raise ValueError("invalid visible-output timeline")
if not (0 <= slo_passing_requests <= successful_requests <= attempted_requests):
    raise ValueError("inconsistent request counts")
if window_seconds <= 0 or attempted_requests <= 0:
    raise ValueError("invalid system-window denominator")
gaps = output_tokens - 1
decode_ms = e - f
if gaps > 0 and decode_ms <= 0:
    raise ValueError("positive decode duration required")
kv_bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_element
artifact = {
    "scope": "full causal GQA; no hidden output; logical unsharded KV",
    "kv_bytes_per_token": kv_bytes_per_token,
    "prompt_kv_mib": kv_bytes_per_token * input_tokens / 2**20,
    "stop_kv_mib": kv_bytes_per_token * (input_tokens + gaps) / 2**20,
    "ttft_ms": f - s,
    "e2e_ms": e - s,
    "tpot_ms": decode_ms / gaps if gaps else None,
    "decode_output_tokens_per_second": gaps / (decode_ms / 1000) if gaps else None,
    "whole_request_output_tokens_per_second": output_tokens / ((e-s)/1000) if e > s else None,
    "request_throughput_rps": successful_requests / window_seconds,
    "observed_goodput_rps": slo_passing_requests / window_seconds,
    "error_rate": (attempted_requests-successful_requests) / attempted_requests,
}
print(json.dumps(artifact, indent=2))

Expected values:

kv_bytes_per_token = 327680
prompt_kv_mib = 320.3125
stop_kv_mib = 366.875
e2e_ms = 3915
tpot_ms = 25
decode_output_tokens_per_second = 40
whole_request_output_tokens_per_second ≈ 38.31
request_throughput_rps = 9.8
observed_goodput_rps = 9.5
error_rate = 0.02

Deliberately Broken Cases

Broken Timing Denominator

Writing 190 + 150 × 25 = 3,940 ms charges 150 inter-token gaps for 150 visible tokens, forgetting that token 1 arrives at TTFT = 190 ms and only 149 transitions follow. Dividing all 150 tokens by decode-only duration commits the mirror-image error by crediting prefill's first token to post-prefill decode time.

Broken Architecture Assumption

Substituting query_heads=64 for kv_heads=8 yields 2.5 MiB/token, 8× the actual GQA footprint. That figure belongs to an otherwise identical MHA model. Architectures such as MLA or hybrid state-space models store compressed latent vectors or recurrent state, so neither head count applies without re-deriving the state layout.

Hands-On Exercise

Modify the calculator for this alternative hypothetical model and trace:

• 48 layers;

• 32 query heads and 4 KV heads;

• head dimension 128;

• FP8 KV storage at 1 byte per element, ignoring scale and alignment overhead for this exercise;

• 4,096 input tokens and 256 visible output tokens;

• TTFT of 420 ms; and

• constant 18 ms inter-token gaps.

Produce request-metrics-alt.json. Then answer:

1. What is KV MiB per token?

2. What is prompt KV GiB?

3. What is E2E latency?

4. What are the decode-only and whole-request output rates?

5. Which physical-memory costs did the simplified FP8 formula omit?

Expected core results:

KV bytes/token = 2 × 48 × 4 × 128 × 1 = 49,152 bytes = 0.046875 MiB
prompt KV      = 192 MiB = 0.1875 GiB
E2E            = 420 + 255 × 18 = 5,010 ms
decode rate    = 255 / 4.590 ≈ 55.56 tokens/s
whole rate     = 256 / 5.010 ≈ 51.10 tokens/s

Measurable Exit Criteria

The tutorial artifact passes when:

• architecture scope, KV layout, dtype, and token counts are explicit;

• input components sum to 1,025 tokens;

• KV memory derives from kv_heads rather than total parameters or query heads;

• the first and last visible-token events span exactly 149 ITL gaps;

• the final timestamp is 3,915 ms;

• TTFT, E2E, TPOT, decode-only rate, and whole-request rate reproduce from the CSV;

• request throughput, observed goodput, and error rate reproduce from the separate 10-second system window;

• synthetic figures are not presented as provider measurements;

• the broken MHA-for-GQA substitution is rejected; and

• the alternative exercise records omitted quantization scales and alignment as a physical-memory limitation.

Retrieval Practice

1. Which stages before model execution contribute to client-observed TTFT?

2. Why does GQA reduce KV memory compared with an otherwise matching MHA model?

3. What is the difference between ITL and TPOT?

4. Why are there 149 gaps for 150 visible tokens?

5. Why does the trace yield 40.00 tokens/s on one rate metric and 38.31 tokens/s on the other?

6. When does the standard kv_heads formula stop applying?

Answer Key

1. Gateway authentication and validation, chat-template rendering and tokenization, routing, admission control, and scheduler queueing.

2. Groups of query heads share a smaller set of key/value heads, storing fewer K and V vectors per token at every layer.

3. ITL is each individual time gap between consecutive streamed outputs; TPOT is the single per-request average time per output token after the first token.

4. Token 1 marks the end of TTFT; only the transitions into tokens 2 through 150 create post-first-token intervals (150 - 1 = 149).

5. 40.00 tokens/s divides the 149 post-first-token outputs by decode duration (3.725 s), whereas 38.31 tokens/s divides all 150 visible tokens by full E2E latency (3.915 s).

6. Whenever the retained state is not one K and one V vector per KV head per token, such as in MLA or recurrent/hybrid layers. Sliding-window layers need the retained length the engine actually keeps, and quantization scales, alignment, and block rounding add physical memory even when the logical formula still applies.

What This Post Does Not Cover

This post follows a single request in isolation. I-01 adds token-budget scheduling and overload handling across concurrent requests; I-02 turns per-token KV size into paged block capacity; I-04 separates local prefix reuse from distributed cache-aware routing; I-03 evaluates transferring KV state between disaggregated prefill and decode pools; and I-06 covers placing a dense model across multiple GPUs and the communication cost of that placement.

Source Notes

• Vaswani, A., Shazeer, N., Parmar, N., et al. "Attention Is All You Need." NeurIPS, 2017. Primary source for Transformer attention and Q/K/V projections. arxiv.org/abs/1706.03762

• Hugging Face. "Cache Strategies." Current Transformers documentation, accessed 2026-11-02. First-party cache behavior and current cache variants. huggingface.co

• vLLM Project. "Benchmark CLI." Current documentation, accessed 2026-11-02. First-party TTFT, ITL, and TPOT measurement points and formulas. docs.vllm.ai

• NVIDIA. "LLM Inference Benchmarking: Fundamental Concepts." 2025-04. First-party explanation of client metrics and throughput denominators. developer.nvidia.com

• NVIDIA. "Mastering LLM Techniques: Inference Optimization." 2023-11. First-party overview of prefill/decode phases, KV cache, MQA/GQA, and inference memory. developer.nvidia.com