Dense Distributed Parallelism and Networking: A Placement Tutorial

An inference server becomes distributed for one of two reasons: the model does not fit on one accelerator, or one accelerator cannot meet the required latency and throughput. Those reasons sound similar, but they lead to different designs. Splitting one request across more GPUs can make the model fit while adding communication to every forward pass. Replicating a model can increase request throughput without reducing the memory needed by any one replica.

This tutorial is designed to be read before the revised I-05 tutorial introduces mixture-of-experts deployment. The teaching job is deliberately narrow: place a dense decoder-only model on a multi-GPU, multi-node cluster, estimate its memory and collective-communication floor, and reject a configuration that fits but is operationally poor.

Prerequisites and Learning Outcomes

Before starting, you should understand the request lifecycle, prefill and decode, KV-cache growth, and continuous batching from Posts I-00 through I-02. You do not need prior distributed-systems or CUDA experience.

By the end, you will be able to:

1. distinguish data, tensor, pipeline, and context parallelism;

2. explain all-reduce, all-gather, reduce-scatter, and all-to-all without treating them as interchangeable;

3. calculate whether a model fits under an explicit per-GPU memory reserve;

4. estimate a topology-aware communication lower bound;

5. choose tensor-parallel groups that stay inside the fastest failure domain; and

6. produce a placement artifact with measurable acceptance criteria.

What You Will Build

You will build a parallelism-plan.json artifact containing:

• the model, engine/runtime revisions, precision, layer count, hidden size, and workload assumptions;

• the cluster topology and effective link assumptions;

• candidate TP × PP × DP layouts;

• per-rank weight memory;

• an estimated decode collective floor; and

• the selected placement and its failure-domain rationale.

The artifact is a plan, not a benchmark. Its estimates decide which configurations deserve an actual benchmark; measurements still decide production capacity.

Freeze the Architecture and Workload

All calculations in this tutorial use the following hypothetical but fully specified setup. Do not transplant the result to another model or cluster without changing the inputs.

Input — Assumption

--- — ---:

Model — dense decoder-only Transformer, 70 billion parameters

Model revision — dense-70b-fixture-r1

Serving engine version — tutorial-engine-fixture-1.0

Accelerator/collective runtime — tutorial-runtime-fixture-1.0

Weight precision — BF16, 2 bytes per parameter

Transformer blocks — 80

Hidden size — 8,192

Input-context distribution — 100% of requests have 2,048 input tokens

Decode batch — 32 active sequences, one token each

Cluster — 2 nodes × 4 GPUs = 8 GPUs

GPU memory — 80 GiB per GPU

Runtime + KV reserve — 12 GiB per GPU

Effective intra-node bandwidth — 100 GB/s

Effective inter-node bandwidth — 25 GB/s

Intra-node step latency — 5 microseconds

Inter-node step latency — 20 microseconds

Service objective — maximize online request capacity while keeping decode collective floor below 5 ms

The three immutable fixture identifiers are tutorial labels, not product releases. Replace them together with the workload and topology before using the plan elsewhere. The bandwidth numbers are planning assumptions, not product specifications. Measure the actual message sizes and topology with the communication library and hardware you deploy. Current vLLM guidance likewise recommends keeping tensor parallelism within a node and using pipeline parallelism across nodes when a model must span them, because the interconnect changes the best strategy (vLLM Parallelism and Scaling).

Four Parallelism Strategies, Four Different Jobs

Data Parallelism

Data parallelism creates independent model replicas and sends different requests to them. During inference, ordinary data-parallel replicas do not synchronize gradients because there is no training step. Data parallelism increases aggregate request capacity and creates failure isolation, but every replica must still hold a complete model, possibly using tensor or pipeline parallelism inside that replica.

If a replica uses TP=2 and the cluster has 8 GPUs, DP=4 means four two-GPU replicas. A failed TP rank takes down its two-GPU replica, but the other three replicas can continue after the router removes the failed one.

Tensor Parallelism

Tensor parallelism shards matrices inside a layer. Ranks collaborate on the same tokens and exchange partial activations at repeated points in every transformer block. Megatron-LM established an influential intra-layer tensor-parallel design, and current vLLM states that its tensor parallelism includes Megatron-LM's algorithm (Shoeybi et al., 2019; vLLM Parallelism and Scaling).

Tensor parallelism reduces per-rank weight memory, but it puts collective communication on the latency path. It should not be increased merely because more GPUs are available.

Pipeline Parallelism

Pipeline parallelism assigns consecutive layer ranges to stages. A request's activations move from stage 0 to stage 1 rather than requiring every rank to cooperate inside every layer. Microbatches can overlap across stages, but an empty stage at pipeline startup or drain is a pipeline bubble. Under the simplified assumptions of equal stage times and m microbatches over p stages, idealized pipeline utilization is:

pipeline_utilization = m / (m + p - 1)
bubble_fraction       = (p - 1) / (m + p - 1)

For p=2 and m=8, utilization is 8/9 = 88.9% and the bubble fraction is 1/9 = 11.1%. Real inference also includes imbalanced layers, queueing, activation transfers, and decode dependencies. GPipe supplies the microbatch-pipeline foundation; modern inference engines implement their own schedules (Huang et al., 2019).

Context Parallelism

Context parallelism partitions a long sequence across ranks so attention state and computation do not reside entirely on one device. Ring Attention, for example, distributes sequence blocks and circulates key-value blocks while overlapping communication with blockwise attention (Liu, Zaharia, and Abbeel, 2023).

Context parallelism solves a long-sequence memory or attention-compute problem. It is not a default way to increase ordinary short-context request throughput. Our fixed distribution of 2,048-token input contexts does not justify it, so the selected plan will use CP=1.

The Collective-Communication Vocabulary

The operations below describe where data ends up; they do not prescribe a single ring, tree, or fused implementation. NCCL defines the current operation semantics directly (NVIDIA NCCL collective API).

Collective — Result after the operation — Typical inference use

All-reduce — reduce corresponding values, then leave the full reduced result on every rank — combine tensor-parallel partial sums

Reduce-scatter — reduce corresponding values, then leave one result shard on each rank — keep the reduced activation sharded

All-gather — concatenate shards and leave the full result on every rank — reconstruct an activation needed by every rank

All-to-all — every rank sends a distinct shard to every other rank — token dispatch/return in many expert-parallel layouts

An all-reduce can be implemented conceptually as reduce-scatter followed by all-gather, but an optimized library may choose another algorithm. Similarly, expert-parallel systems do not universally use one collective; I-05 will show that the engine, DP/TP layout, message size, and hardware decide the implementation.

Two Reproducible Planning Formulas

Formula 1: Per-Rank Weight Memory

For a dense model whose weights are evenly tensor-sharded:

weight_bytes_total    = parameters × bytes_per_parameter
weight_gib_total      = weight_bytes_total / 2^30
weight_gib_per_rank   = weight_gib_total / TP / PP
usable_gib_per_gpu    = gpu_gib - runtime_and_kv_reserve_gib
fits                  = weight_gib_per_rank <= usable_gib_per_gpu

This is a first filter. Uneven layers, embeddings, the output head, allocator behavior, CUDA graphs, activations, and quantization metadata can break an apparently exact fit. That is why the 12 GiB reserve is visible rather than silently assumed.

Formula 2: Ring All-Reduce Planning Floor

For a ring all-reduce over p ranks and a logical message of M bytes per rank, a useful planning approximation is:

wire_bytes_per_rank ≈ 2 × (p - 1) / p × M
time_per_collective ≈ wire_bytes_per_rank / effective_bandwidth
                      + 2 × (p - 1) × link_step_latency

This is not a performance prediction. It ignores protocol selection, chunking, contention, topology-aware trees, overlap, kernel launch cost, and congestion. Its purpose is to expose configurations whose communication floor is already unacceptable.

Worked Example: Place the 70B-Class Model

Step 1: Calculate Weight Memory

total weight bytes = 70,000,000,000 × 2
                   = 140,000,000,000 bytes

total weight GiB   = 140,000,000,000 / 1,073,741,824
                   = 130.39 GiB

usable per GPU     = 80 - 12
                   = 68 GiB

Candidate memory results:

Layout — Weight GiB per GPU — Fits the 68 GiB budget? — Independent replicas

--- — ---: — ---: — ---:

TP1 × PP1 × DP8 — 130.39 — No — 8

TP2 × PP1 × DP4 — 65.19 — Yes — 4

TP4 × PP1 × DP2 — 32.60 — Yes — 2

TP8 × PP1 × DP1 — 16.30 — Yes — 1

TP4 × PP2 × DP1 — 16.30 — Yes — 1

The smallest tensor-parallel group that passes the visible memory reserve is TP=2. Choosing TP4 or TP8 uses more GPUs per request than the fit constraint requires and reduces the number of independent replicas.

For the TP4 × PP2 candidate, the calculator below includes TP collective traffic but not the activation transfer or bubble cost between PP stages. That row cannot be selected until those additional costs are measured.

Step 2: Calculate the Decode Activation Message

For one BF16 hidden activation per active decode sequence:

M = decode_batch × hidden_size × bytes_per_element
  = 32 × 8,192 × 2
  = 524,288 bytes
  = 0.5 MiB

For this exercise, assume two all-reduces per transformer block, or 2 × 80 = 160 collectives per decode forward pass. Exact collective placement is model- and engine-specific; verify it with traces before using the result operationally.

Step 3: Compare TP2 Inside a Node With TP8 Across Nodes

For TP=2 on an intra-node link:

wire bytes/rank = 2 × 1/2 × 524,288 = 524,288 bytes
transfer time   = 524,288 / 100,000,000,000 = 5.24 microseconds
latency term    = 2 × 1 × 5 = 10 microseconds
one collective  ≈ 15.24 microseconds
160 collectives ≈ 2.44 milliseconds

For TP=8 spanning both nodes:

wire bytes/rank = 2 × 7/8 × 524,288 = 917,504 bytes
transfer time   = 917,504 / 25,000,000,000 = 36.70 microseconds
latency term    = 2 × 7 × 20 = 280 microseconds
one collective  ≈ 316.70 microseconds
160 collectives ≈ 50.67 milliseconds

Both layouts fit. Only TP2 meets the exercise's <5 ms collective-floor criterion, and TP2 also produces four independent replicas. The selected layout is therefore:

Node A: [GPU0, GPU1] replica 0, TP2   [GPU2, GPU3] replica 1, TP2
Node B: [GPU4, GPU5] replica 2, TP2   [GPU6, GPU7] replica 3, TP2
PP=1, DP=4, CP=1

This is a topology-aware placement decision, not a claim that TP2 always wins. A larger KV requirement, a different precision, or a model that does not fit inside one node can change the answer.

Build the Artifact

Save the following as plan_parallelism.py, run it, and redirect the output:

import json
import math

PARAMETERS = 70_000_000_000
MODEL_REVISION = "dense-70b-fixture-r1"
ENGINE_VERSION = "tutorial-engine-fixture-1.0"
RUNTIME_VERSION = "tutorial-runtime-fixture-1.0"
BYTES_PER_PARAMETER = 2
GPU_GIB = 80
RESERVE_GIB = 12
LAYERS = 80
HIDDEN = 8192
INPUT_CONTEXT_DISTRIBUTION = {"2048": 1.0}
DECODE_BATCH = 32
COLLECTIVES_PER_LAYER = 2

CANDIDATES = [
    {"tp": 1, "pp": 1, "dp": 8, "domain": "single_gpu"},
    {"tp": 2, "pp": 1, "dp": 4, "domain": "intra_node"},
    {"tp": 4, "pp": 1, "dp": 2, "domain": "intra_node"},
    {"tp": 8, "pp": 1, "dp": 1, "domain": "inter_node"},
    {"tp": 4, "pp": 2, "dp": 1, "domain": "pipeline_across_nodes"},
]

LINKS = {
    "single_gpu": {"bandwidth_gb_s": math.inf, "step_us": 0},
    "intra_node": {"bandwidth_gb_s": 100, "step_us": 5},
    "inter_node": {"bandwidth_gb_s": 25, "step_us": 20},
    "pipeline_across_nodes": {"bandwidth_gb_s": 100, "step_us": 5},
}

total_weight_gib = PARAMETERS * BYTES_PER_PARAMETER / 2**30
message_bytes = DECODE_BATCH * HIDDEN * 2
usable_gib = GPU_GIB - RESERVE_GIB

rows = []
for candidate in CANDIDATES:
    tp = candidate["tp"]
    pp = candidate["pp"]
    weight_gib = total_weight_gib / tp / pp
    link = LINKS[candidate["domain"]]
    if tp == 1:
        floor_ms = 0.0
    else:
        wire_bytes = 2 * (tp - 1) / tp * message_bytes
        transfer_s = wire_bytes / (link["bandwidth_gb_s"] * 1e9)
        latency_s = 2 * (tp - 1) * link["step_us"] * 1e-6
        floor_ms = (transfer_s + latency_s) * LAYERS * COLLECTIVES_PER_LAYER * 1e3
    rows.append({
        **candidate,
        "weight_gib_per_gpu": round(weight_gib, 2),
        "fits": weight_gib <= usable_gib,
        "decode_collective_floor_ms": round(floor_ms, 2),
        "passes": weight_gib <= usable_gib and floor_ms < 5,
    })

artifact = {
    "assumptions": {
        "parameters": PARAMETERS,
        "model_revision": MODEL_REVISION,
        "engine_version": ENGINE_VERSION,
        "runtime_version": RUNTIME_VERSION,
        "bytes_per_parameter": BYTES_PER_PARAMETER,
        "gpu_gib": GPU_GIB,
        "runtime_and_kv_reserve_gib": RESERVE_GIB,
        "layers": LAYERS,
        "hidden_size": HIDDEN,
        "input_context_distribution": INPUT_CONTEXT_DISTRIBUTION,
        "decode_batch": DECODE_BATCH,
    },
    "candidates": rows,
    "selected": {"tp": 2, "pp": 1, "dp": 4, "cp": 1},
}
print(json.dumps(artifact, indent=2))

Run:

python plan_parallelism.py > parallelism-plan.json

Expected checks:

TP1: fits=false
TP2: weight_gib_per_gpu=65.19, decode_collective_floor_ms=2.44, passes=true
TP8: weight_gib_per_gpu=16.30, decode_collective_floor_ms=50.67, passes=false
selected: TP2 × PP1 × DP4 × CP1

Deliberately Broken Configuration

The following configuration is valid enough to launch on some distributed runtimes, but it is broken for this workload:

world_size: 8
tensor_parallel_size: 8
pipeline_parallel_size: 1
data_parallel_size: 1
placement: spread_across_two_nodes
reason: "use every GPU for the lowest per-rank weight memory"

It fails for three measurable reasons:

1. TP2 already satisfies the 68 GiB fit budget, so TP8 solves no remaining fit problem.

2. The planning floor is about 50.67 ms per decode forward pass, above the 5 ms criterion.

3. One failed rank removes the only replica; the selected TP2/DP4 plan has four replica failure domains.

A second failure may remain invisible until load: distributing ranks without checking GPU-to-NIC and GPU-to-GPU affinity can send traffic through a slower path. Current NCCL is topology-aware, but the deployment still must expose the right devices and network interfaces (NVIDIA NCCL overview).

A Second Broken All-to-All Plan

Suppose a long-context backend uses two sequence-transpose all-to-all operations per block over CP=8, and each rank owns 4,096 BF16 token activations with hidden size 8,192. Unified Sequence Parallelism describes why all-to-all and ring-based sequence parallelism have different topology trade-offs (Fang and Zhao, 2024). Under this exercise's simplified assumptions:

local activation M       = 4,096 × 8,192 × 2
                         = 67,108,864 bytes = 64 MiB
bytes sent to other ranks = 7/8 × M
                         = 58,720,256 bytes
one bandwidth-only floor = 58,720,256 / 25,000,000,000
                         = 2.35 ms
160 all-to-alls           = 2.35 × 160
                         = 375.81 ms

The plan is broken before adding per-peer latency, contention, or synchronization. An implementation may overlap much of this traffic or choose another algorithm, but that possibility is something to measure—not permission to erase the data movement from the plan.

Hands-On Exercise

Change the artifact generator for this new requirement:

• output-heavy workload;

• 24 GiB per GPU reserved for KV cache and runtime;

• same 8 GPUs and link assumptions;

• collective-floor limit remains 5 ms.

Answer these questions in a placement-decision.md file:

1. Does TP2 still fit?

2. What is the smallest TP size that fits?

3. How many independent replicas remain?

4. Does the smallest fitting layout cross a node boundary?

5. If the answer changes, which constraint changed it: model fit, collective floor, or failure isolation?

Expected memory calculation:

usable per GPU = 80 - 24 = 56 GiB
TP2 weight     = 65.19 GiB -> does not fit
TP4 weight     = 32.60 GiB -> fits

TP4 × DP2 is the smallest fitting layout and keeps each TP group inside one four-GPU node. Its collective floor under the same formula is approximately 6.06 ms, so it fails the original 5 ms objective. That result is intentional: the exercise demonstrates that a constraint set can have no passing configuration. The correct response is to revise hardware, precision, reserve, or SLO and then benchmark; it is not to hide the failed criterion.

Measurable Exit Criteria

Your tutorial artifact passes only if all of the following are true:

• every candidate satisfies TP × PP × DP = 8 or explicitly marks unused GPUs;

• weight memory and reserve use the same GiB unit;

• no selected rank exceeds 80 GiB;

• the selected TP groups do not cross nodes unless the model cannot otherwise fit;

• the selected plan reports an estimated collective floor below 5 ms, or explicitly reports that no candidate passes;

• the placement diagram identifies independent replica failure domains;

• assumptions include model revision, precision, engine/runtime versions, input-context distribution, GPU topology, and measured effective bandwidth before production approval; and

• an actual load test replaces the planning floor before capacity is promised.

Retrieval Practice

1. Which parallelism strategy creates independent request-serving replicas?

2. Why can tensor parallelism make a model fit yet worsen decode latency?

3. What is the semantic difference between all-gather and reduce-scatter?

4. When is context parallelism justified?

5. In the worked example, why is TP2 preferred to TP8?

6. What should you do when no candidate satisfies both memory and communication criteria?

Answer Key

1. Data parallelism.

2. It shards weights but adds repeated collectives to the forward-pass critical path.

3. All-gather reconstructs the full tensor from shards on every rank; reduce-scatter reduces values and leaves only one result shard on each rank.

4. When sequence-state memory or long-context attention computation cannot fit or meet the objective on one rank; it is not a default short-context throughput knob.

5. TP2 is the smallest group that fits, stays within each node, gives four replicas, and has a 2.44 ms planning floor versus about 50.67 ms for cross-node TP8.

6. Report that the constraint set is infeasible, then change precision, hardware, reserve, topology, or the objective and remeasure. Do not silently drop a criterion.

What This Post Does Not Cover

This tutorial covers dense-model placement. It does not cover expert routing, expert load balancing, or expert-parallel collectives; those follow in I-05 after it is reorganized. It also does not claim that the simple ring formula predicts production latency. Kernel traces, collective benchmarks, queueing, KV capacity, and end-to-end load tests remain required.

The next P0 tutorial, I-07, turns a deployed layout into a reproducible benchmark, derives SLO-conditioned capacity, and defines the telemetry needed to know when the plan is wrong.

Source Notes

• Shoeybi, M., Patwary, M., Puri, R., et al. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.” 2019. Primary source for the influential intra-layer tensor-parallel design. arxiv.org/abs/1909.08053

• Narayanan, D., Shoeybi, M., Casper, J., et al. “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM.” 2021. Primary source for composing tensor, pipeline, and data parallelism. arxiv.org/abs/2104.04473

• Huang, Y., Cheng, Y., Bapna, A., et al. “GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism.” 2019. Primary source for microbatch pipeline parallelism and bubble amortization. arxiv.org/abs/1811.06965

• Liu, H., Zaharia, M., and Abbeel, P. “Ring Attention with Blockwise Transformers for Near-Infinite Context.” 2023. Primary source for distributing long-sequence attention across devices. arxiv.org/abs/2310.01889

• Fang, J., and Zhao, S. “USP: A Unified Sequence Parallelism Approach for Long Context Generative AI.” 2024. Primary source comparing all-to-all and ring-based sequence-parallel approaches. arxiv.org/abs/2405.07719

• NVIDIA. “Collective Communication Functions.” Current NCCL documentation, accessed 2026-08-29. First-party definitions of all-reduce, all-gather, reduce-scatter, and all-to-all semantics. github.com/NVIDIA/nccl

• vLLM Project. “Parallelism and Scaling.” Current documentation, accessed 2026-08-29. First-party deployment guidance for tensor and pipeline parallelism across single- and multi-node systems. docs.vllm.ai