Separating Prefill and Decode

Prefill-decode disaggregation assigns prompt processing and token generation to separate worker pools, then transfers each request's KV state between them. The split can isolate decode from long-prefill interference and let each pool scale independently. Both pools still need compatible model weights; whether that raises total weight replication relative to an aggregated baseline depends on placement. Disaggregation also adds routing and KV transfer, introduces new failure modes, and can degrade latency or capacity when the workload or network does not justify it.

This tutorial treats disaggregation as a measured engineering trade-off rather than a default upgrade. You will size one KV transfer, compare topology budgets, define an aggregated-versus-disaggregated benchmark, and reject a short-prompt case where remote prefill adds cost without repairing an SLO.

Scope the Two Phases Correctly

In a decoder-only Transformer, prefill processes many known prompt positions in parallel and often has high arithmetic intensity. Low-batch decode processes one new position per active sequence and often has lower arithmetic intensity, so weight and KV memory traffic can dominate. Prompt length, batch size, attention variant, quantization, kernel choice, and hardware can shift either regime.

The operational problem is cross-phase interference: a large prefill scheduled alongside or ahead of active decodes can widen the gaps users see between streamed tokens. Chunked prefill limits that interruption on a shared worker. Disaggregation instead isolates the two phases on separate worker pools.

Phase isolation has its own costs. Each pool needs compatible model weights and runtime capacity, every prefill result must reach a decoder, and requests can now fail during routing, KV buffer allocation, transfer, handoff, or decode admission.

Architecture and Request Flow

1. The frontend tokenizes and admits the request.

2. A prefill router selects a compatible prefill worker based on uncached prompt tokens, queue depth, and prefix-cache locality.

3. The prefill worker computes the prompt KV state and handoff metadata, including any first output token already sampled.

4. Before a direct transfer, a compatible decoder is selected and its receive buffers are reserved. A shared-store design may instead publish KV blocks first and assign a reader afterward.

5. The selected transport moves the required KV blocks, and identity, block layout, token sequence, readiness, and buffer ownership are validated at handoff.

6. The decoder continues token generation. Document whether the first token streams before or after handoff, because that choice shifts transfer latency between TTFT and the early ITL distribution.

Current NVIDIA Dynamo guidance treats prefill and decode routing differently: the decode-stage router disables prefix-overlap scoring because decode routing should not chase prefix reuse, assumes no KV reuse unless the backend can truly deduplicate transferred blocks, and stops tracking prefill tokens so that decode-side load reflects decode work. Optional topology-aware KV transfer can also constrain or bias decode routing toward the selected prefill worker's transfer domain (Dynamo disaggregated routing).

Current Implementation Status Matters

As checked on 2026-11-02, vLLM marks disaggregated prefilling experimental and states directly that the feature does not improve throughput. Its documented reasons are tuning TTFT and ITL separately and controlling tail ITL. Chunked prefill with a proper chunk size can reach the same tail-ITL goal, but the docs note that the right size is hard to find in practice and call disaggregated prefilling a much more reliable way to control tail ITL (vLLM Disaggregated Prefilling).

That scope note does not contradict research systems that raise SLO-conditioned capacity through phase-specific placement and hardware provisioning. It keeps a published research result from being read as a promise about one current feature.

Reproducible KV-Transfer Formula

For standard GQA KV tensors:

KV_bytes = 2 × layers × kv_heads × head_dim × bytes_per_element × prompt_tokens

bandwidth_floor_ms = KV_bytes / (effective_GB_per_second × 10^9) × 1000

handoff_floor_ms = setup_ms + bandwidth_floor_ms

Here effective_GB_per_second is sustained payload bandwidth, excluding the separately modeled setup term; if a measured rate already includes setup, do not add setup_ms again. The formula models the full logical KV payload crossing one equivalent link. Sharded parallel transfers need per-rank byte accounting, shared-link contention checks, and critical-path analysis. Peak link specifications are not measured application bandwidth.

Freeze the Worked Transfer

Use this hypothetical architecture and prompt:

KV size:

KV bytes/token = 2 × 80 × 8 × 128 × 2
               = 327,680 bytes

KV bytes       = 327,680 × 2,000
               = 655,360,000 bytes
               = 625 MiB

Worked Example: Compare Three Links

The 20 ms handoff budget is a tutorial constraint rather than a universal SLO. Passing this simplified floor only keeps the 100 GB/s link in contention, because queueing, buffer allocation, and synchronization can still push real handoff latency past 20 ms.

DistServe gives an approximate aggregate-bandwidth illustration: about 1.13 GB per 512-token OPT-66B request and roughly 90 Gbps at ten requests/s, in the paper's units. That figure sizes network traffic rather than guaranteeing invisible per-request transfer latency. DistServe's evaluation compared it with vLLM and DeepSpeed-MII and reported up to 7.4× higher per-GPU request rate at more than 90% simultaneous TTFT/TPOT attainment, with the 7.4× maximum measured against DeepSpeed-MII (Zhong et al., OSDI 2024). That is a measured result under DistServe's evaluated setup, distinct from the hypothetical numbers in our table.

For sizing with explicit units, OPT-66B's published configuration has 64 layers and hidden size 9,216. Assuming full multi-head K/V in FP16, 512 cached tokens require 2 × 64 × 9,216 × 2 × 512 = 1,207,959,552 bytes, or 1.125 GiB ≈ 1.208 GB; ten such payloads per second come to about 96.64 Gbit/s before protocol overhead. Use this unit-explicit calculation rather than treating the paper's rounded 90 Gbps illustration as a network provisioning figure (OPT-66B configuration).

Transfer Strategies

Serialized Point-to-Point Handoff

Prefill completes, transfers the full KV state, and then hands generation to decode. If the first output token streams before handoff, that transfer delay lands directly between the first and second visible tokens.

Layer-Wise Asynchronous Transfer

Splitwise streams each layer's KV state asynchronously while later layers continue prefill, hiding part of the transfer behind ongoing compute. For smaller prompts, where fine-grained overlap overhead outweighs the gain, Splitwise uses serialized transfer instead (Patel et al., ISCA 2024). Whether the first token precedes or follows handoff depends on the runtime and must be observed at the client.

Shared or Tiered KV Store

Prefill writes KV blocks into a distributed cache or memory tier that decode workers read on demand. Mooncake centers its long-context serving architecture on distributed KV storage and KV-aware scheduling (Qin et al., 2024). Instead of a single point-to-point handoff, the operator now manages tier hit rate, block placement, aggregate read/write bandwidth, eviction, and metadata consistency.

Build the Transfer Model

Save as model_pd_transfer.py and run python model_pd_transfer.py > pd-transfer-model.json:

import json

LAYERS = 80
KV_HEADS = 8
HEAD_DIM = 128
BYTES_PER_ELEMENT = 2
PROMPT_TOKENS = 2000
SETUP_MS = 1.0
HANDOFF_BUDGET_MS = 20.0
LINKS_GB_S = [100, 25, 10]

bytes_per_token = 2 * LAYERS * KV_HEADS * HEAD_DIM * BYTES_PER_ELEMENT
kv_bytes = bytes_per_token * PROMPT_TOKENS

links = []
for bandwidth in LINKS_GB_S:
    bandwidth_floor_ms = kv_bytes / (bandwidth * 1e9) * 1000
    handoff_floor_ms = SETUP_MS + bandwidth_floor_ms
    links.append({
        "effective_bandwidth_gb_s": bandwidth,
        "bandwidth_floor_ms": bandwidth_floor_ms,
        "handoff_floor_ms": handoff_floor_ms,
        "floor_within_20ms_budget": handoff_floor_ms <= HANDOFF_BUDGET_MS,
    })

short_prompt_tokens = 300
short_kv_bytes = bytes_per_token * short_prompt_tokens
short_handoff_10gb_s_ms = SETUP_MS + short_kv_bytes / (10 * 1e9) * 1000

artifact = {
    "scope": "hypothetical GQA transfer model",
    "bytes_per_token": bytes_per_token,
    "prompt_tokens": PROMPT_TOKENS,
    "kv_bytes": kv_bytes,
    "kv_mib": kv_bytes / 2**20,
    "links": links,
    "broken_short_prompt_case": {
        "prompt_tokens": short_prompt_tokens,
        "effective_bandwidth_gb_s": 10,
        "handoff_floor_ms": short_handoff_10gb_s_ms,
    },
}
print(json.dumps(artifact, indent=2))

Expected core values (floats rounded to four decimal places):

bytes_per_token = 327680
kv_bytes = 655360000
kv_mib = 625
100 GB/s handoff floor = 7.5536 ms, floor_within_20ms_budget=true
25 GB/s handoff floor = 27.2144 ms, floor_within_20ms_budget=false
10 GB/s handoff floor = 66.5360 ms, floor_within_20ms_budget=false
300-token/10 GB/s handoff floor = 10.8304 ms

Benchmark Primer Before Choosing a Layout

Use I-06's placement worksheet to count the actual GPUs in both designs. TP8 means eight tensor-parallel ranks cooperate on one replica. Two aggregated TP8 replicas use 16 GPUs; a TP8 prefill pool plus a TP8 decode pool also uses 16 GPUs. Equal GPU count is a comparison constraint, not a prediction of equal cost or performance.

Keep four benchmark quantities distinct:

• Offered rate: how many requests per second the client schedules, independent of server completion. Verify actual dispatch delay so client backpressure does not mask server overload.

• Throughput: how many successful requests finish per second over a declared measurement window, while failed requests stay in the reliability denominator.

• Latency and attainment: TTFT measures time to the first output token, and TPOT averages time across subsequent output tokens. A request counts as a joint pass only when it meets every required cutoff; a mean latency is neither a tail percentile nor a joint attainment rate.

• SLO-compliant capacity: the highest tested offered rate that satisfies the declared attainment, error-rate, and queue-stability gates on the fixed workload. A fast transfer floor alone never establishes capacity.

Consider a synthetic 100-request cohort where 95 requests succeed and 90 meet both latency cutoffs. The error rate is 5%, joint attainment among successful requests is 90/95 (about 94.7%), and joint attainment across all attempts is 90%. Always declare the denominator before comparing deployments; under this lesson's 95% all-attempts gate, that synthetic point fails.

Hold the request schedule, model revision, sequence lengths, cache policy, hardware, and measurement boundaries constant while changing placement. Client metric conventions follow the vLLM benchmark guide. This primer covers what you need to interpret the transfer experiment below; I-07 develops the full SLO analyzer, sample-size and uncertainty policy, headroom calculation, and failure test.

Build the Equal-Budget Benchmark Plan

Create pd-benchmark-plan.yaml:

comparison:
  model_revision: "<same immutable revision>"
  engine_version: "<same pinned version>"
  total_gpus: 16
  gpu_type: "<same pinned SKU>"
  network_and_host_budget: "<declare and match>"
  warmup_requests: 100
  offered_rates_rps: [1, 2, 4, 8]
  output_tokens_minimum_for_tpot: 2
  request_deadline_seconds: 120
  precision: bf16
  workload_trace: "<same timestamped input/output/prefix distribution>"
  cache_state: [cold, warm]
  duration_seconds: 600
deployments:
  aggregated:
    description: "same workers perform prefill and decode"
    replicas: 2
    tensor_parallel: 8
    pipeline_parallel: 1
  disaggregated:
    prefill_replicas: 1
    decode_replicas: 1
    tensor_parallel_per_replica: 8
    pipeline_parallel_per_replica: 1
    kv_transport: "<declared backend and topology>"
metrics:
  client: [ttft, tpot, itl, e2e, errors, goodput]
  scheduler: [prefill_queue, decode_queue, running, rejected]
  kv_transfer: [bytes, setup_ms, transfer_ms, failures]
acceptance:
  ttft_max_ms: 2000
  tpot_max_ms: 50
  max_itl_per_request_ms: 200
  numerator: successful_requests_meeting_all_three_latency_conditions
  denominator: all_offered_requests_after_warmup_including_rejections_and_timeouts
  slo_attainment_min: 0.95
  error_rate_max: 0.01
  require_no_sustained_queue_growth: true

Compare both layouts at equal GPU count under an identical open-loop workload. Report raw throughput, but choose a layout by the maximum arrival rate that satisfies both TTFT and TPOT/ITL objectives. Changing GPU count, accelerator SKU, power or network budget, or the prompt set confounds a phase-placement comparison. Cross-hardware or unequal-GPU designs can still be compared on explicitly normalized cost or capacity, but they answer a different question.

Deliberately Broken Case: Short Prompts on a Slow Link

Assume a separate synthetic baseline fixture:

• 300-token prompts;

• co-located P95 ITL = 35 ms against a 50 ms objective;

• local prefill interference window = 2 ms; and

• remote link = 10 GB/s effective plus 1 ms setup.

The handoff floor is 10.8304 ms. Under a serialized handoff with no queueing or compute offset, trading 2 ms of local prefill interference for remote transfer adds at least 8.8304 ms to the affected path. That single-request floor does not prove the disaggregated setup breaches the 50 ms P95 ITL target, because a point handoff floor is not a tail percentile and the co-located baseline already meets the 50 ms SLO with 15 ms to spare. The engineering verdict is "not justified by the supplied evidence" until an end-to-end rate sweep shows otherwise.

Keep aggregated serving as the baseline, tune chunked prefill first, and evaluate selective remote prefill for long uncached prompts. Adopt the more complex design only if measured SLO-conditioned capacity or cost justifies it. Treating "disaggregate everything" as a default rule produces the broken configuration.

Hands-On Exercise

1. Run the transfer model for prompt lengths of 300, 2,000, and 8,192 tokens.

2. Add FP8 KV as an idealized sensitivity case, then list the scaling-factor, alignment, and accuracy requirements needed in practice.

3. Measure actual effective bandwidth for at least two message sizes on your target topology.

4. Run aggregated and disaggregated rate sweeps under the same 16-GPU budget.

5. Report P50/P95/P99 TTFT and TPOT/ITL, joint SLO attainment, error rate, queue trend, and KV-transfer latency distribution.

6. Inject a failed transfer and verify that the runtime either fails the request cleanly or retries it at most once without emitting duplicate client-visible tokens or leaking KV buffers (a retry may recompute prefill work, so this is not an exactly-once computation guarantee).

7. Test a selective routing policy that keeps short or cache-hot prefills local.

Measurable Exit Criteria

The tutorial passes when:

• phase bottlenecks are stated as workload- and hardware-dependent;

• KV bytes derive from layers, KV heads, head dimension, dtype, and prompt length;

• the 2,000-token example produces exactly 625 MiB;

• every link result separates bandwidth floor from setup and runtime overhead;

• current vLLM experimental status and no-throughput-gain scope are stated;

• the DistServe 7.4× result retains its compared baselines and >90% simultaneous TTFT/TPOT attainment condition;

• hypothetical values are labeled and kept distinct from published paper measurements;

• aggregated and disaggregated tests use equal resources and identical workload traces;

• the short-prompt adoption proposal is rejected as unsupported without claiming an unmeasured SLO breach; and

• transfer failures, duplicate-output prevention, and recovery are tested.

Retrieval Practice

1. What problem does P/D disaggregation primarily target in current vLLM documentation?

2. Why is peak interconnect bandwidth insufficient for a transfer model?

3. Where should prefix locality normally influence routing in a disaggregated design?

4. What does layer-wise transfer attempt to hide?

5. Why is raw throughput insufficient to select a deployment?

6. Why does the 300-token broken case stay aggregated?

Answer Key

1. Independent TTFT/ITL tuning and tail-ITL control; the vLLM docs state explicitly that the feature does not improve throughput.

2. Effective bandwidth depends on topology, message size, transport protocol, link contention, memory registration, and synchronization.

3. Primarily in prefill-worker or shared-cache selection; decoder selection normally prioritizes compatible transfer topology and active decode load.

4. KV-transfer time behind the ongoing prefill computation of later model layers.

5. A deployment can raise raw throughput while breaching TTFT or TPOT/ITL cutoffs; SLO-conditioned capacity requires meeting latency, error, and queue-stability gates together.

6. Under the serialized no-overlap fixture, the 10.8304 ms handoff floor exceeds the 2 ms local interference it replaces, and the co-located baseline already meets its 50 ms P95 ITL objective, so no end-to-end benefit is established. An end-to-end rate sweep would still be needed to determine the actual SLO and cost result.

What This Post Does Not Cover

This post does not prescribe one KV connector or cluster topology. I-06 covers dense parallelism and collective placement, and I-07 covers the full capacity, SLO, and observability workflow needed for production selection.

Source Notes

• Zhong, Y., Liu, S., Chen, J., et al. "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." OSDI, 2024. Primary source for disaggregation, SLO-conditioned goodput, placement, and KV-transfer sizing. usenix.org

• Patel, P., Choukse, E., Zhang, C., et al. "Splitwise: Efficient Generative LLM Inference Using Phase Splitting." ISCA, 2024. Primary source for phase-specific pools and asynchronous layer-wise KV transfer. microsoft.com

• vLLM Project. "Disaggregated Prefilling (Experimental)." Current documentation, accessed 2026-11-02. First-party feature maturity, TTFT/ITL rationale, connectors, and no-throughput statement. docs.vllm.ai

• NVIDIA Dynamo. "Disaggregated Serving." Current documentation, accessed 2026-11-02. First-party prefill/decode routing and topology behavior. docs.nvidia.com

• Qin, R., Li, Z., He, W., et al. "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving." arXiv, 2024. A conference version appeared at FAST 2025 as "Mooncake: Trading More Storage for Less Computation." Primary source for distributed KV storage and long-context serving. arxiv.org/abs/2407.00079