Making Each Token Cheaper

At small batch sizes, a decode step spends most of its time moving bytes rather than doing arithmetic: it reads the model's weights and every active sequence's KV cache out of GPU memory to produce one token per sequence. That gives three levers on the cost of a token. Quantization moves fewer bytes per weight, speculative decoding produces several tokens per pass over the weights, and IO-aware attention kernels such as FlashAttention cut memory traffic inside attention. None of them is free, and each changes what you have to measure.

This tutorial makes one GPU efficient before the next lesson spreads the model across many. It derives a bandwidth lower bound on decode time for the 70-billion-parameter fixture model at BF16, FP8, and INT4, computes one speculative-decoding speedup from an explicit acceptance rate, draft length, and draft cost, and shows where each saving stops applying.

Freeze the Model and Hardware

The model is the 80-layer GQA model from Inside an LLM Inference Request, with the 70-billion-parameter count, 8,192 hidden size, and model revision dense-70b-fixture-r1 from Distributing a Model Across GPUs, so the numbers carry across lessons:

The bandwidth and compute rows are hypothetical round numbers, not a product specification. For scale, NVIDIA's H100 SXM product page lists 80 GB of GPU memory and 3.35 TB/s of memory bandwidth; its BF16 Tensor Core figure of 1,979 teraFLOPS is marked "with sparsity," so it is not a dense matmul peak (NVIDIA H100). Replace every row with your own datasheet values before reusing the worksheet.

Why Decode Is Usually Bandwidth-Bound

Two costs bound every forward pass. Weights and the KV cache live in high-bandwidth memory (HBM) and must reach the compute cores once per forward pass, which Pope et al. call memory time; an N-parameter decoder-only model also needs about 2N matmul FLOPs per token processed, one multiply and one add per parameter (Pope et al., 2022). A step cannot finish before both are done, so the larger of the two is a lower bound:

memory_time  = (weight_bytes + batch × KV_bytes_per_sequence) / HBM_bandwidth
compute_time = 2 × parameters × positions_in_step / peak_FLOPs
step_time   >= max(memory_time, compute_time)

Pope et al. note that attention matmuls add comparatively few FLOPs for large models and can often be excluded; attention still matters for memory, because each sequence's KV cache is its own. At batch 1 under BF16, the weights alone set the floor:

weight bytes   = 70,000,000,000 × 2 = 140,000,000,000 bytes
memory_time    = 140 GB / 3.0 TB/s = 46.667 ms
compute_time   = 2 × 70e9 × 1 / 1,000e12 = 0.140 ms
intensity      = 140 GFLOP / 140 GB = 1 FLOP/byte
hardware ratio = 1,000 TFLOP/s / 3.0 TB/s ≈ 333 FLOP/byte

The arithmetic units need about 333 FLOPs per byte delivered to stay busy; batch-1 decode offers 1. Batching amortizes the weight bytes across sequences but not the KV bytes, because every sequence brings its own cache. At the 32-sequence snapshot:

KV per sequence = 327,680 × 2,048 = 671,088,640 bytes
memory_time     = (140e9 + 32 × 671,088,640) / 3.0e12 = 53.825 ms
compute_time    = 2 × 70e9 × 32 / 1,000e12 = 4.480 ms

The step is still memory-bound by about 12×. Each extra 2,048-token sequence adds 0.224 ms of KV reads but only 0.140 ms of matmul time, so in this fixture batching alone never makes a plain decode step compute-bound. Pope et al. describe the same shift: weight loading dominates memory time at small batch sizes and short sequences, KV loading at large ones (Pope et al., 2022).

These are bounds, not measurements. Chen et al. state the weights-only version directly: model size divided by the total memory bandwidth across all chips gives a hard ceiling on autoregressive sampling speed (Chen et al., 2023). The bound assumes every byte moves at peak bandwidth with nothing else on the critical path, so a measured step can only be slower.

Weight Quantization

Formats

Quantization stores each value in fewer bits, plus scale factors that map the low-precision values back to their real range.

• FP8 has two encodings. E4M3 (4 exponent bits, 3 mantissa bits) has a maximum normal value of 448 and is recommended for weights and activations; E5M2 reaches 57,344 and is recommended for gradients. Values are multiplied by a scale factor before the cast so they fit the narrow range (Micikevicius et al., 2022).

• INT8 stores 8-bit integers with a scale per tensor, per channel, or per vector; Micikevicius et al. describe per-channel weight scales and per-tensor activation scales as common for INT8. With both weights and activations in INT8 (W8A8), SmoothQuant's authors note that matmuls can then run on INT8 kernels, which can halve GPU memory use and nearly double matmul throughput compared with FP16; their own end-to-end result is up to a 1.56× speedup with half the memory (Xiao et al., 2023).

• Weight-only INT4 (W4A16) stores 4-bit integer weights in groups that share a scale; AWQ uses a group size of 128 throughout its paper. Because the hardware has no INT4 × FP16 multiply instruction, the kernel dequantizes the integers to FP16 before the matrix computation, so arithmetic stays at 16 bits (Lin et al., 2024).

Why Outliers Make Naive INT8 Hard

A per-tensor scale is set by the largest magnitude it must represent. Dettmers et al. found that at around 6.7B parameters, hidden-state features with magnitudes up to 20× larger than other dimensions appear in all transformer layers, concentrated in only 6 feature dimensions at that scale, and regular 8-bit methods then fail. LLM.int8() keeps 16-bit accuracy by multiplying those outlier dimensions in 16-bit while more than 99.9% of values are still multiplied in 8-bit (Dettmers et al., 2022). To see the problem, take a symmetric INT8 grid with 127 positive levels: if one feature at 20× the typical magnitude sets the scale, typical values land on only about 127 / 20 ≈ 6 levels. SmoothQuant's authors summarize the asymmetry: weights are easy to quantize, activations are not (Xiao et al., 2023).

GPTQ, AWQ, and SmoothQuant

• GPTQ is a one-shot, post-training weight quantization method that uses approximate second-order information to choose quantized values layer by layer. It reduces weights to 3 or 4 bits with negligible accuracy loss in the authors' evaluations and quantized 175B-parameter models in about four GPU hours (Frantar et al., 2023).

• AWQ observes that protecting only about 1% of salient weights greatly reduces quantization error, and that salient channels should be found from the activation distribution rather than from weight magnitudes. It scales those channels up through an equivalent transformation before quantizing, without backpropagation or reconstruction (Lin et al., 2024).

• SmoothQuant targets W8A8. Offline, it divides activations by a per-channel smoothing factor and folds that factor into the weights, a mathematically equivalent transformation that migrates quantization difficulty from activations to weights so both can use INT8 matmuls (Xiao et al., 2023).

What Quantization Saves, and What It Does Not

Lower-precision weights cut the memory a replica needs and the bytes each decode step reads, which is the term that dominates the bound above. GPTQ's decode kernel shows the mechanism: it multiplies a quantized matrix by a full-precision vector, dequantizing weights on the fly without quantizing activations. Dequantization costs extra compute, but the kernel reads far less memory, and the authors attribute almost all of their speedup to those kernels (Frantar et al., 2023).

What quantization does not save automatically:

• Prefill compute. Weight-only formats dequantize to 16 bits before the matmul, so compute-bound prefill still does 16-bit arithmetic. SmoothQuant's authors note that weight-only methods speed up generation through reduced data loading, especially at batch size 1, while W8A8 may serve batching and the context stage better (Xiao et al., 2023); Pope et al. likewise describe int8 weights as saving weight-loading time, especially at low batch sizes (Pope et al., 2022).

• Kernel support. As of November 2026, vLLM's FP8 W8A8 guide states that FP8 computation needs NVIDIA compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell), while Turing and Ampere run FP8 checkpoints as weight-only W8A16 through Marlin kernels (vLLM FP8 W8A8); its overview keeps a per-hardware compatibility table (vLLM Quantization). Dettmers et al. also report that LLM.int8()'s overhead can slow inference below an FP16 baseline for models under 6.7B parameters (Dettmers et al., 2022).

• KV bytes. Weight quantization leaves the KV cache in its own dtype. KV-cache quantization is a separate decision with its own accuracy risks, covered in Managing the KV Cache.

• Quality. Every format, group size, and calibration set changes the model's numerics. Re-measure task quality on your own versioned golden set and regression gate, as in Evaluating AI Systems, before trusting a published accuracy table; vLLM's FP8 guide also makes accuracy evaluation the final step of its quantization process.

One reported result shows the scope to keep. GPTQ measured average per-token latency at batch size 1 while generating 128-token sequences with OPT-175B: 71 ms for the 3-bit model on one A100 80GB versus 230 ms for FP16 on five A100 80GB GPUs, a 3.24× reduction (Frantar et al., 2023). That comparison changes both precision and GPU count, and it covers batch-1 decode only.

Worked Example: Weight Bytes and Bounds by Format

GPTQ estimates about 0.15 extra bits per weight at group size 128 (one 16-bit scale per 128 weights alone adds 0.125 bits), so the worksheet charges INT4 at 4.15 bits per weight. FP8 with one 32-bit scale per 8,192-weight row adds under 0.004 bits per weight, which the worksheet ignores. All 70 billion parameters are treated as quantized; if a checkpoint keeps layers such as embeddings at higher precision, count them at their own width.

BF16      70e9 × 16   / 8 = 140.0000 GB = 130.39 GiB
FP8 E4M3  70e9 × 8    / 8 =  70.0000 GB =  65.19 GiB
INT4 g128 70e9 × 4.15 / 8 =  36.3125 GB =  33.82 GiB

batch-1 bound  = weight_bytes / 3.0e12
batch-32 bound = (weight_bytes + 32 × 671,088,640) / 3.0e12

Three readings matter. First, the BF16 weights alone (130.39 GiB) exceed one 80 GiB GPU, which is why the next lesson splits them; FP8 (65.19 GiB) fits under the 68 GiB left after the 12 GiB reserve, with about 2.8 GiB to spare before quantization metadata and activations. That column counts weights only. The batch-32 snapshot also needs 32 × 640 MiB = 20 GiB of KV, more than the 12 GiB reserve, so on one 80 GiB GPU only INT4 holds it (33.82 + 20 = 53.82 GiB); read the BF16 and FP8 batch-32 rows as per-step bandwidth bounds, not as single-GPU configurations. Second, the bit ratio from BF16 to INT4 is 16 / 4.15 = 3.86×, yet the batch-32 bound improves only 2.79×, because the 21.47 GB of BF16 KV reads do not shrink. Third, hitting the synthetic 25 ms TPOT with FP8 at batch 1, counting weights only, would require sustaining 23.333 / 25 = 93% of peak bandwidth; whether a real kernel gets there is a measurement question.

Speculative Decoding

Draft, Then Verify

Speculative decoding pairs the target model with a cheaper draft model. The draft proposes γ tokens autoregressively; the target scores all of them in one parallel pass, accepts a prefix, and always adds one token of its own, so each target pass yields between 1 and γ + 1 tokens (Leviathan et al., 2023). It pays because, when decode is memory-bound, scoring a few extra positions reads the same weights and KV cache once. Chen et al. argue this component by component: linear layers stay memory-bound for a short draft, the KV cache does not grow with draft length, and all-reduces stay latency-bound (Chen et al., 2023).

Why the Acceptance Rule Preserves the Target Distribution

Write p(x) for the target's next-token distribution and q(x) for the draft's, both after temperature, top-k, or nucleus adjustments. For a drafted token x drawn from q:

accept x with probability min(1, p(x) / q(x))
on rejection, sample from p'(x) = max(0, p(x) - q(x)) / Σ max(0, p(x) - q(x))

Recompute why the output follows p. Drafting x and accepting it has probability q(x) × min(1, p(x) / q(x)) = min(p(x), q(x)). Summed over all x, acceptance has probability β = Σ min(p, q), so rejection has probability 1 - β. Because Σ max(0, p - q) = Σ (p - min(p, q)) = 1 - β, rejecting and then sampling x has probability (1 - β) × max(0, p(x) - q(x)) / (1 - β) = p(x) - min(p(x), q(x)). The two paths add to exactly p(x). Leviathan et al. prove the same result with p as the target and q as the draft; Chen et al. use the same scheme with the letters swapped (Leviathan et al., 2023; Chen et al., 2023).

The same distribution is not the same tokens. Chen et al. note that differing numerics and seed handling mean outputs will not be identical, and they claim to recover the target distribution "within hardware numerics" (Chen et al., 2023). As of November 2026, vLLM's guide calls its speculative sampling "theoretically lossless up to the precision limits of hardware numerics" and warns that changes in batch size can vary logprobs and output probabilities (vLLM Speculative Decoding).

Expected Tokens per Target Step

Leviathan et al. define α as the expected probability that a drafted token is accepted. Assuming acceptances are independent and identically distributed, each target pass yields a capped geometric number of tokens:

E[tokens per target step] = (1 - α^(γ+1)) / (1 - α)

Recompute it: a pass always emits one token, a second if the first draft is accepted (probability α), a third if the first two are accepted (α²), and so on up to γ + 1. The expectation is 1 + α + α² + ... + α^γ, a geometric sum equal to the formula above.

Draft Cost and Speedup

Let c be the time of one draft run divided by the time of one target run. One iteration costs γc + 1 target-step units, and the paper's expected walltime improvement is:

speedup = (1 - α^(γ+1)) / ((1 - α) × (γc + 1))

The theorem rests on an assumption the authors state: γ + 1 concurrent target evaluations run in parallel without increasing walltime. That holds while the target step is memory-bound with compute to spare. The paper also notes the result assumes long enough generations (Leviathan et al., 2023).

Why Gains Shrink at Large Batch

At larger batch sizes, the verification pass processes batch × (γ + 1) positions, and the spare compute runs out. EAGLE's authors describe this mechanism and measure it: on MT-bench at temperature 0, their reported speedup ratio over vanilla autoregressive decoding for LLaMA2-Chat 70B fell from 3.01× at batch size 1 to 2.81×, 2.50×, and 2.40× at batch sizes 2, 3, and 4, and the same section states that its 70B throughput tests ran on four A100 40GB GPUs at FP16 (Li et al., 2024). As of November 2026, vLLM's guide presents speculative decoding as a way to reduce inter-token latency for medium-to-low QPS, memory-bound workloads (vLLM Speculative Decoding), and TurboSpec's authors, working in vLLM, observe that it may degrade serving performance if added without tuning to the incoming requests and speculation method (Liu et al., 2024).

Variants: EAGLE and Medusa

• EAGLE drafts at the feature level: a small head autoregresses on the target's second-to-top-layer features, using a token sequence advanced by one step, and the target verifies the draft tree with tree attention while keeping its output distribution (Li et al., 2024).

• Medusa adds extra decoding heads to the target model and verifies several candidate continuations at once with tree-based attention. Medusa-1 trains the heads on a frozen backbone for lossless acceleration; Medusa-2 also fine-tunes the backbone, which changes the model, and the optional typical-acceptance scheme deliberately stops matching the original distribution (Cai et al., 2024).

What Speculation Does to ITL

Tokens now arrive in bursts. Each iteration emits up to γ + 1 tokens, so per-chunk ITL tracks iteration time, which is longer than a plain decode step, while per-token TPOT falls. Benchmarking Inference Capacity explains why chunk gaps and per-token averages use different denominators and lists acceptance rate and acceptance length as signals to record when speculation is enabled. Never compare a speculative run's chunk ITL with a plain run's ITL as if they measured the same thing.

Worked Example: One Speculative Calculation

Take α = 0.8, γ = 4, and c = 0.1, so one draft step costs a tenth of a target step. These are teaching values, not measurements of any model pair.

E[tokens/step] = (1 - 0.8^5) / (1 - 0.8)
               = (1 - 0.32768) / 0.2
               = 3.3616 tokens per target step

speedup        = 3.3616 / (4 × 0.1 + 1)
               = 3.3616 / 1.4
               = 2.4011×

Apply it to the travel copilot's Kyoto hotel request from Inside an LLM Inference Request, where a plain decode step took the synthetic 25 ms:

iteration time = 25 × (4 × 0.1 + 1) = 35 ms
average TPOT   = 25 / 2.4011 = 10.41 ms/token
149 post-first-token outputs ≈ 149 × 10.41 ≈ 1,551 ms (versus 3,725 ms)

The average improves 2.40×, but the gap between streamed chunks grows from 25 ms to 35 ms, and each chunk carries 1 to 5 tokens. α is an average, and the independence assumption is a modeling simplification, so sensitivity matters more than the headline: at α = 0.6 the same γ and c give 2.3056 tokens per step and 1.65×; a slower draft with c = 0.5 at α = 0.8 gives 1.12×; and at α = 0.2 speculation is slower than plain decoding, at 0.89×.

Now test the free-verification assumption with the bound model. At the 32-sequence snapshot, verification processes 32 × 5 = 160 positions, so its compute time is 2 × 70e9 × 160 / 1,000e12 = 22.4 ms. Under BF16 (53.825 ms) and FP8 (30.492 ms), memory time still dominates and verification stays nearly free. Under INT4, memory time is only 19.262 ms, so verification becomes compute-bound at 22.4 ms, about 1.163× a plain step, and replacing the "+ 1" in the denominator gives 3.3616 / (0.4 + 1.163) = 2.15× instead of 2.40×. This is our extension of the paper's model with c held at 0.1, not a result from the paper, but it shows a real interaction: removing weight bytes lowers the batch size at which speculation stops being free.

Attention Kernels

FlashAttention: Exact Attention, Fewer HBM Trips

Standard attention writes the N × N score and probability matrices to HBM and reads them back. FlashAttention is an IO-aware exact attention algorithm: it tiles Q, K, and V into blocks that fit in on-chip SRAM, computes the softmax incrementally across blocks, and fuses the steps into one kernel, so the N × N matrix never travels to HBM. The paper proves O(N²d²/M) HBM accesses, where d is the head dimension and M the SRAM size, against Ω(Nd + N²) for standard attention, and shows up to 9× fewer accesses for typical values (Dao et al., 2022). Only the block-sparse extension in the same paper is approximate.

FlashAttention-2 and FlashAttention-3

FlashAttention-2 keeps the algorithm and repartitions the work: fewer non-matmul FLOPs, parallelism across sequence length even for a single head, and better work splitting between warps, for about 2× over FlashAttention and 50%–73% of theoretical maximum FLOPs/s on A100 (Dao, 2023). FlashAttention-3 targets Hopper: warp specialization overlaps computation with data movement, block-wise matmul and softmax are interleaved, and block quantization with incoherent processing makes FP8 usable. On an H100 80GB SXM5 with FP16 inputs, sequence lengths from 512 to 16k, and 16k total tokens per run, the authors report a 1.5–2.0× forward-pass speedup over FlashAttention-2, up to 740 TFLOPs/s; the FP8 path reaches close to 1.2 PFLOPs/s with, in the authors' test with outlier features, 2.6× lower numerical error than standard FP8 attention with per-tensor quantization (Shah et al., 2024).

Why Kernels Matter Most for Long Prefill

The score matrix has one entry per query position per key position. Prefill processes every prompt position at once, so without tiling a 32,768-token prompt would build a 32,768 × 32,768 score matrix per head, 2 GiB in BF16; decode has one query per sequence, so its score row is only as long as the context. That is why FlashAttention's savings show up mostly in long prefill. Decode attention is dominated instead by reading the KV cache, which Chen et al. say accounts for most of attention's memory-bandwidth use during sampling (Chen et al., 2023). The Flash-Decoding authors add that FlashAttention parallelizes across batch size and query length, so with one query at batch size 1 it uses less than 1% of the GPU; Flash-Decoding instead splits keys and values into chunks processed in parallel (Dao et al., 2023). FlashAttention-3's paper lists optimizing for LLM inference as future work (Shah et al., 2024). Choose kernels by which phase dominates your workload, and benchmark the attention backend your engine actually selects.

Build the Token-Cost Calculator

Save as token_cost.py and run python token_cost.py > token-cost.json:

import json

PARAMETERS = 70_000_000_000
LAYERS, KV_HEADS, HEAD_DIM, KV_BYTES_PER_ELEMENT = 80, 8, 128, 2
HBM_BYTES_PER_SECOND = 3.0e12
PEAK_DENSE_FLOPS = 1.0e15
CONTEXT_TOKENS, BATCH = 2048, 32
BITS_PER_WEIGHT = {"bf16": 16.0, "fp8_e4m3": 8.0, "int4_g128": 4.15}
ALPHA, GAMMA, DRAFT_COST = 0.8, 4, 0.1
PLAIN_STEP_MS = 25.0

if not (0 <= ALPHA < 1 and GAMMA >= 1 and DRAFT_COST >= 0 and BATCH >= 1):
    raise ValueError("invalid speculation or batch setting")

kv_bytes_per_sequence = (2 * LAYERS * KV_HEADS * HEAD_DIM
                         * KV_BYTES_PER_ELEMENT * CONTEXT_TOKENS)

def step_bound_ms(weight_bytes, batch, positions_per_sequence):
    memory = (weight_bytes + batch * kv_bytes_per_sequence) / HBM_BYTES_PER_SECOND
    compute = 2 * PARAMETERS * batch * positions_per_sequence / PEAK_DENSE_FLOPS
    return 1000 * max(memory, compute)

expected_tokens = (1 - ALPHA ** (GAMMA + 1)) / (1 - ALPHA)
speedup = expected_tokens / (GAMMA * DRAFT_COST + 1)
formats = {}
for name, bits in BITS_PER_WEIGHT.items():
    weight_bytes = PARAMETERS * bits / 8
    plain_ms = step_bound_ms(weight_bytes, BATCH, 1)
    verify_ratio = step_bound_ms(weight_bytes, BATCH, GAMMA + 1) / plain_ms
    formats[name] = {
        "bits_per_weight": bits,
        "weight_gb": weight_bytes / 1e9,
        "weight_gib": weight_bytes / 2**30,
        "batch1_weight_bound_ms": 1000 * weight_bytes / HBM_BYTES_PER_SECOND,
        "batch_step_bound_ms": plain_ms,
        "verify_cost_ratio": verify_ratio,
        "speculative_speedup_at_batch": expected_tokens / (GAMMA * DRAFT_COST + verify_ratio),
    }

artifact = {
    "scope": "lower bounds from peak figures; not measurements",
    "parameters": PARAMETERS,
    "hbm_bytes_per_second": HBM_BYTES_PER_SECOND,
    "peak_dense_flops": PEAK_DENSE_FLOPS,
    "batch": BATCH,
    "context_tokens": CONTEXT_TOKENS,
    "kv_bytes_per_sequence": kv_bytes_per_sequence,
    "formats": formats,
    "speculation": {
        "alpha": ALPHA,
        "gamma": GAMMA,
        "draft_cost": DRAFT_COST,
        "expected_tokens_per_target_step": expected_tokens,
        "speedup_free_verification": speedup,
        "iteration_ms": PLAIN_STEP_MS * (GAMMA * DRAFT_COST + 1),
        "average_tpot_ms": PLAIN_STEP_MS / speedup,
    },
}
print(json.dumps(artifact, indent=2))

Expected core values:

kv_bytes_per_sequence = 671088640
bf16:      weight_gb = 140.0,   batch1 ≈ 46.667 ms, batch32 ≈ 53.825 ms, verify_cost_ratio = 1.0
fp8_e4m3:  weight_gb = 70.0,    batch1 ≈ 23.333 ms, batch32 ≈ 30.492 ms, verify_cost_ratio = 1.0
int4_g128: weight_gb = 36.3125, batch1 ≈ 12.104 ms, batch32 ≈ 19.262 ms, verify_cost_ratio ≈ 1.1629
expected_tokens_per_target_step = 3.3616
speedup_free_verification ≈ 2.4011
int4_g128 speculative_speedup_at_batch ≈ 2.1509
iteration_ms = 35.0
average_tpot_ms ≈ 10.4117

Deliberately Broken Cases

"INT4 Is Always 4× Faster"

Four bits against sixteen is a 4× ratio for the raw weight payload only. Group scales raise INT4 to about 4.15 bits (3.86×), the 21.47 GB of BF16 KV reads at the batch-32 snapshot do not shrink (2.79× on the bound), and a bound is not a measurement. Prefill under weight-only INT4 still runs 16-bit matmuls, and a format without a fast kernel on your GPU can be slower than BF16.

"Speculative Decoding Changes the Outputs"

Under the rejection-sampling rule, the output distribution equals the target's up to hardware numerics, as recomputed above; what differs is which sample you get, because seeds and numerics differ. The claim becomes true only for lossy variants: Medusa's typical acceptance, a fine-tuned backbone as in Medusa-2, or vLLM's FLy verification, which the vLLM guide marks as not distribution-preserving (vLLM Speculative Decoding).

"Speculation Always Helps"

At α = 0.2, γ = 4, and c = 0.1, a target step yields 1.2496 tokens on average against an iteration cost of 1.4 target steps, so speculation runs at 0.89× of plain decoding. Break-even for this γ and c sits near α ≈ 0.287, and at large batch the "+ 1" verification cost grows, as the INT4 batch-32 check showed.

"FlashAttention Approximates Attention"

FlashAttention computes exact attention and saves time by avoiding HBM round trips for the N × N matrix; only the block-sparse variant in the same paper is approximate (Dao et al., 2022). Beyond small floating-point rounding differences from a different summation order, a quality change after switching kernels points to a different precision path, such as FP8 attention, or to a bug, not to tiling.

"Quantization Speeds Up Prefill and Decode Equally"

Small-batch decode is bound by bytes, so fewer weight bytes help directly. Prefill is usually compute-bound, and weight-only formats dequantize to 16 bits before the matmul, so they leave prefill arithmetic unchanged and add dequantization work. Only formats whose matmuls run in lower precision, such as FP8 or INT8 W8A8 on supported hardware, can cut prefill compute, and those still need a quality check.

Hands-On Exercise

Extend the calculator to:

1. sweep α over 0.3, 0.6, 0.8, and 0.9 at γ = 4 and c = 0.1;

2. find the γ from 1 to 8 that maximizes speedup at α = 0.8 and c = 0.1;

3. recompute α = 0.8 and γ = 4 with a slow draft, c = 0.5;

4. find, for each format, the smallest batch at which verifying γ + 1 = 5 positions per sequence makes the bound compute-bound at 2,048-token contexts; and

5. recompute the INT4 batch-32 step bound with idealized 1-byte FP8 KV, ignoring KV scales.

Reference values for the speculative cases, all with free verification:

Verification becomes compute-bound at batch 98 for BF16, 49 for FP8, and 26 for INT4. With 1-byte KV, the INT4 batch-32 step bound falls from 19.262 ms to 15.683 ms. Record these as bound-model outputs, then write the benchmark plan that would test them: acceptance rate on your own prompts, draft cost on your hardware, and chunk ITL, TPOT, and goodput at your real batch sizes.

Measurable Exit Criteria

The worksheet passes when:

• model, parameter count, bits per weight, KV dtype, batch size, context length, peak bandwidth, and peak compute are explicit, and hardware figures are labeled hypothetical or cited to a datasheet;

• weight bytes equal 140.0, 70.0, and 36.3125 GB for BF16, FP8, and INT4 at 4.15 bits;

• batch-1 weight bounds equal 46.667, 23.333, and 12.104 ms, and batch-32 step bounds equal 53.825, 30.492, and 19.262 ms;

• every time value is labeled a lower bound, not a measurement;

• the INT4 gain at batch 32 is reported as 2.79× on the bound, not 4×;

• α = 0.8, γ = 4, and c = 0.1 yield 3.3616 expected tokens per target step and a 2.4011× speedup, with the independence and free-verification assumptions stated;

• the INT4 batch-32 verification ratio (1.163) and adjusted speedup (2.15×) are labeled as an extension of the paper's model;

• speculative runs report acceptance rate, chunk ITL, and TPOT separately;

• every quantized configuration carries a quality result from your own evaluation set; and

• every cited paper speedup names its model, hardware, workload, baseline, and metric.

Retrieval Practice

1. Why does a batch-1 BF16 decode step on the fixture take at least 46.667 ms when its matmuls need only 0.140 ms at peak?

2. Why does batching amortize weight reads but not KV reads?

3. Why do activation outliers make per-tensor INT8 hard, and how does SmoothQuant respond?

4. Why does weight-only INT4 help decode more than prefill?

5. Show that the speculative acceptance rule emits token x with probability p(x).

6. Why can speculation lower TPOT while raising chunk ITL, and why do its gains shrink at large batch?

7. What does FlashAttention save, and why does that matter more for a 32k-token prefill than for decode?

Answer Key

1. Each step must read all 140 GB of BF16 weights from HBM, which takes 140 GB / 3.0 TB/s = 46.667 ms at peak, while 2 × 70e9 FLOPs take 0.140 ms at 1,000 TFLOP/s; the step offers 1 FLOP per byte against a hardware ratio of about 333.

2. All sequences in a batch share one copy of the weights, read once per step, but each sequence's KV cache is its own and must be read separately; at 2,048 tokens each sequence adds 671,088,640 bytes, or 0.224 ms, per step.

3. A per-tensor scale is set by the largest magnitude, so a feature at 20× the typical magnitude leaves typical values only about 6 of the 127 positive INT8 levels. SmoothQuant divides activations by a per-channel smoothing factor and folds it into the weights, an equivalent transformation that lets both quantize to INT8.

4. Small-batch decode is limited by bytes moved, which weight-only INT4 cuts; prefill is usually compute-bound, and W4A16 kernels dequantize weights to 16 bits before the matmul, so the arithmetic does not shrink.

5. Drafting and accepting x has probability q(x) × min(1, p(x) / q(x)) = min(p(x), q(x)); rejection has probability 1 - β with β = Σ min(p, q); the residual distribution then contributes p(x) - min(p(x), q(x)); the two paths sum to p(x).

6. Each iteration costs γc + 1 plain steps (35 ms in the example) but emits 3.3616 tokens on average, so TPOT falls to 10.41 ms while the gap between streamed chunks grows to 35 ms. At large batch, the batch × (γ + 1) verification positions use up the spare compute the formula assumes, so verification costs more than one plain step and the speedup shrinks.

7. It avoids writing and reading the N × N score matrix in HBM by tiling into on-chip SRAM, while still computing exact attention. A 32k-token prefill would otherwise build a 32,768 × 32,768 matrix per head (2 GiB in BF16); decode has one query row per sequence and is dominated by KV reads instead.

What This Post Does Not Cover

Distributing a Model Across GPUs splits the BF16 model that does not fit on one GPU and adds the collective cost that tensor parallelism puts on every step. Benchmarking Inference Capacity measures TTFT, ITL, TPOT, and goodput under load, which is where every bound in this lesson has to be checked. KV-cache quantization belongs to Managing the KV Cache, and quality gates belong to Evaluating AI Systems. Pruning, distillation, and mixture-of-experts sparsity are out of scope.

Source Notes

• Pope, R., Douglas, S., Chowdhery, A., et al. "Efficiently Scaling Transformer Inference." arXiv:2211.05102, 2022. Primary source for the memory-time and compute-time cost model, the 2N matmul FLOPs per token approximation, and KV loading that grows with batch size. arxiv.org/abs/2211.05102

• NVIDIA. "NVIDIA H100 Tensor Core GPU." Product page, accessed 2026-11-03. First-party memory capacity, memory bandwidth, and sparsity-marked Tensor Core figures. nvidia.com

• Micikevicius, P., Stosic, D., Burgess, N., et al. "FP8 Formats for Deep Learning." arXiv:2209.05433, 2022. Primary source for the E4M3 and E5M2 encodings, their ranges and recommended uses, and scale factors. arxiv.org/abs/2209.05433

• Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale." NeurIPS, 2022. Primary source for emergent activation outliers and mixed-precision decomposition. arxiv.org/abs/2208.07339

• Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." ICLR, 2023. Primary source for one-shot second-order weight quantization, group-size overhead, and batch-1 decode kernel results. arxiv.org/abs/2210.17323

• Lin, J., Tang, J., Tang, H., et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys, 2024. Primary source for activation-aware salient-channel scaling, group size 128, and W4A16 dequantization. arxiv.org/abs/2306.00978

• Xiao, G., Lin, J., Seznec, M., et al. "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models." ICML, 2023. Primary source for W8A8 smoothing and the weight-only versus W8A8 trade-off. arxiv.org/abs/2211.10438

• vLLM Project. "Quantization." Current documentation, accessed 2026-11-03. First-party list of supported methods and per-hardware compatibility table. docs.vllm.ai

• vLLM Project. "FP8 W8A8." Current documentation, accessed 2026-11-03. First-party FP8 compute-capability requirements, weight-only Marlin fallback, and post-quantization accuracy evaluation. docs.vllm.ai

• Leviathan, Y., Kalman, M., and Matias, Y. "Fast Inference from Transformers via Speculative Decoding." ICML, 2023. Primary source for the acceptance rule, expected tokens per target step, and the walltime improvement formula. arxiv.org/abs/2211.17192

• Chen, C., Borgeaud, S., Irving, G., et al. "Accelerating Large Language Model Decoding with Speculative Sampling." arXiv:2302.01318, 2023. Primary source for modified rejection sampling, the memory-bandwidth ceiling, and why scoring a short draft costs about one step. arxiv.org/abs/2302.01318

• Li, Y., Wei, F., Zhang, C., and Zhang, H. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty." arXiv:2401.15077, 2024. Primary source for feature-level drafting and speedup versus batch size. arxiv.org/abs/2401.15077

• Cai, T., Li, Y., Geng, Z., et al. "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads." ICML, 2024. Primary source for decoding heads, tree-based attention, and typical acceptance. arxiv.org/abs/2401.10774

• Liu, X., Park, J., Hu, L., et al. "TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput." arXiv:2406.14066, 2024, revised 2025. Primary source for workload-dependent degradation from untuned speculative decoding in vLLM. arxiv.org/abs/2406.14066

• vLLM Project. "Speculative Decoding." Current documentation, accessed 2026-11-03. First-party scope, numerics caveats, and supported speculation methods. docs.vllm.ai

• Dao, T., Fu, D. Y., Ermon, S., et al. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." arXiv:2205.14135, 2022. Primary source for tiled, IO-aware exact attention and its HBM-access analysis. arxiv.org/abs/2205.14135

• Dao, T. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." arXiv:2307.08691, 2023. Primary source for the FlashAttention-2 work-partitioning changes and A100 utilization. arxiv.org/abs/2307.08691

• Shah, J., Bikshandi, G., Zhang, Y., et al. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision." arXiv:2407.08608, 2024. Primary source for Hopper asynchrony, FP8 attention, and H100 benchmark settings. arxiv.org/abs/2407.08608

• Dao, T., Haziza, D., Massa, F., and Sizov, G. "Flash-Decoding for Long-Context Inference." Stanford CRFM, 2023-10. Authors' engineering post on splitting keys and values for decode attention. crfm.stanford.edu