Distributing a Model Across GPUs

Two common reasons to distribute inference are making a model fit in memory and meeting latency or throughput targets, and they lead to different designs. Splitting one request across more GPUs can make the model fit, but it adds communication to every forward pass. Replicating the model can raise request throughput, but it does not reduce the memory that any one replica needs.

Read this tutorial before I-05, which covers mixture-of-experts deployment. The scope is deliberately narrow: place a dense decoder-only model on a multi-GPU, multi-node cluster, estimate its memory requirements and collective-communication cost, and reject a configuration that fits in memory but is a poor operational choice.

Freeze the Architecture and Workload

All calculations in this tutorial use the hypothetical setup below. Replace these inputs before applying the worksheet to another model or cluster.

The three fixture identifiers are tutorial labels, not product releases; a real plan records the pinned model, engine, and runtime revisions it was computed for. Replace them together with the workload and topology before reusing the plan. The link bandwidth and latency figures are planning assumptions, not product specifications; measure actual message sizes and topology with the communication library and hardware you deploy. Current vLLM guidance points the same way: it recommends tensor parallelism within a node when a model is too large for one GPU but fits on one node, and tensor parallelism within each node combined with pipeline parallelism across nodes when a model is too large for one node. The interconnect changes which strategy is best (vLLM Parallelism and Scaling).

Four Parallelism Strategies, Four Different Jobs

Data Parallelism

Data parallelism creates independent model replicas and sends different requests to them. During inference, ordinary data-parallel replicas do not synchronize gradients because there is no training step. Data parallelism increases aggregate request capacity and creates failure isolation, but every replica must still hold a complete model, possibly using tensor or pipeline parallelism inside that replica.

If a replica uses TP=2 and the cluster has 8 GPUs, DP=4 means four two-GPU replicas. A failed TP rank takes down its two-GPU replica, but the other three can continue if their processes and dependencies remain healthy. A whole-node failure removes both replicas on that node; four replica groups are not four independent node-level failure domains.

Tensor Parallelism

Tensor parallelism shards matrices inside a layer. Ranks collaborate on the same tokens and exchange partial activations at repeated points in every transformer block. Megatron-LM established an influential intra-layer tensor-parallel design, and current vLLM states that its tensor parallelism includes Megatron-LM's algorithm (Shoeybi et al., 2019; vLLM Parallelism and Scaling).

Tensor parallelism reduces per-rank weight memory, but it puts collective communication on the latency path. Do not increase it merely because more GPUs are available.

Pipeline Parallelism

Pipeline parallelism assigns consecutive layer ranges to stages. A request's activations move from stage 0 to stage 1, so the ranks do not all have to cooperate inside every layer. Microbatches can overlap across stages, but a stage left idle while the pipeline fills or drains is a pipeline bubble. Under the simplified assumptions of equal stage times and m microbatches over p stages, idealized pipeline utilization is:

pipeline_utilization = m / (m + p - 1)
bubble_fraction       = (p - 1) / (m + p - 1)

For p=2 and m=8, utilization is 8/9 = 88.9% and the bubble fraction is 1/9 = 11.1%. Real inference also includes imbalanced layers, queueing, activation transfers, and decode dependencies. GPipe supplies the microbatch-pipeline foundation; modern inference engines implement their own schedules (Huang et al., 2019).

Context Parallelism

Context parallelism partitions a long sequence across ranks so attention state and computation do not reside entirely on one device. Ring Attention, for example, distributes sequence blocks and circulates key-value blocks while overlapping communication with blockwise attention (Liu, Zaharia, and Abbeel, 2023).

Context parallelism solves a long-sequence memory or attention-compute problem; it is not a default way to raise ordinary short-context request throughput. Our fixed distribution of 2,048-token input contexts does not justify it, so the selected plan uses CP=1.

The Collective-Communication Vocabulary

The operations below describe where data ends up; they do not prescribe a single ring, tree, or fused implementation. NCCL defines the current operation semantics directly (NVIDIA NCCL collective API).

An all-reduce can be implemented conceptually as reduce-scatter followed by all-gather, but an optimized library may choose another algorithm. Similarly, expert-parallel systems do not universally use one collective; I-05 will show that the engine, DP/TP layout, message size, and hardware decide the implementation.

Two Reproducible Planning Formulas

Formula 1: Per-Rank Weight Memory

For a dense model whose weights are evenly tensor-sharded:

weight_bytes_total    = parameters × bytes_per_parameter
weight_gib_total      = weight_bytes_total / 2^30
weight_gib_per_rank   = weight_gib_total / TP / PP
usable_gib_per_gpu    = gpu_gib - runtime_and_kv_reserve_gib
fits                  = weight_gib_per_rank <= usable_gib_per_gpu

This is a first filter. Uneven layers, embeddings, the output head, allocator behavior, CUDA graphs, activations, and quantization metadata can break an apparently exact fit. That is why the 12 GiB reserve is visible rather than silently assumed.

Formula 2: Ring All-Reduce Planning Estimate

For a ring all-reduce over p ranks and a logical message of M bytes per rank, a useful planning approximation is:

wire_bytes_per_rank ≈ 2 × (p - 1) / p × M
time_per_collective ≈ wire_bytes_per_rank / effective_bandwidth
                      + 2 × (p - 1) × link_step_latency

Treat the result as a ring-specific screening estimate. It is neither a performance prediction nor a lower bound across all valid algorithms, because it ignores protocol selection, chunking, contention, topology-aware trees, overlap, kernel launch cost, and congestion. Charging inter-node latency to every ring step can overestimate a ring that mixes intra-node and inter-node links, and overlap or a different collective algorithm can change the critical path. A configuration that fails this screen is one to investigate; the failure does not prove that no optimized implementation can pass.

Worked Example: Place the 70B-Class Model

Step 1: Calculate Weight Memory

total weight bytes = 70,000,000,000 × 2
                   = 140,000,000,000 bytes

total weight GiB   = 140,000,000,000 / 1,073,741,824
                   = 130.39 GiB

usable per GPU     = 80 - 12
                   = 68 GiB

Candidate memory results:

Among the PP1 candidates, the smallest tensor-parallel group that passes the visible memory reserve is TP=2. Choosing TP4 or TP8 uses more GPUs per request than the fit constraint requires and reduces the number of independent replicas.

For the TP4 × PP2 candidate, the calculator below includes TP collective traffic but not the activation transfer or bubble cost between PP stages. That row cannot be selected until those additional costs are measured.

Step 2: Calculate the Decode Activation Message

For one BF16 hidden activation per active decode sequence:

M = decode_batch × hidden_size × bytes_per_element
  = 32 × 8,192 × 2
  = 524,288 bytes
  = 0.5 MiB

For this exercise, assume two all-reduces per transformer block, or 2 × 80 = 160 collectives per decode forward pass. Exact collective placement depends on the model and engine, so verify it with traces before using the result operationally.

Step 3: Compare TP2 Inside a Node With TP8 Across Nodes

For TP=2 on an intra-node link:

wire bytes/rank = 2 × 1/2 × 524,288 = 524,288 bytes
transfer time   = 524,288 / 100,000,000,000 = 5.24 microseconds
latency term    = 2 × 1 × 5 = 10 microseconds
one collective  ≈ 15.24 microseconds
160 collectives ≈ 2.44 milliseconds

For TP=8 spanning both nodes:

wire bytes/rank = 2 × 7/8 × 524,288 = 917,504 bytes
transfer time   = 917,504 / 25,000,000,000 = 36.70 microseconds
latency term    = 2 × 7 × 20 = 280 microseconds
one collective  ≈ 316.70 microseconds
160 collectives ≈ 50.67 milliseconds

Both layouts fit, but only TP2 meets the exercise's <5 ms collective-estimate criterion, and TP2 also produces four independent replicas. Under this screening policy, the first layout to benchmark is therefore:

Node A: [GPU0, GPU1] replica 0, TP2   [GPU2, GPU3] replica 1, TP2
Node B: [GPU4, GPU5] replica 2, TP2   [GPU6, GPU7] replica 3, TP2
PP=1, DP=4, CP=1

The choice follows from this cluster's topology and memory budget; TP2 does not always win. A larger KV requirement, a different precision, or a model that does not fit inside one node can change the answer.

Build the Artifact

Save the following as plan_parallelism.py, run it, and redirect the output:

import json

PARAMETERS = 70_000_000_000
MODEL_REVISION = "dense-70b-fixture-r1"
ENGINE_VERSION = "tutorial-engine-fixture-1.0"
RUNTIME_VERSION = "tutorial-runtime-fixture-1.0"
BYTES_PER_PARAMETER = 2
GPU_GIB = 80
RESERVE_GIB = 12
LAYERS = 80
HIDDEN = 8192
INPUT_CONTEXT_DISTRIBUTION = {"2048": 1.0}
DECODE_BATCH = 32
COLLECTIVES_PER_LAYER = 2
COLLECTIVE_LIMIT_MS = 5.0
ACTIVATION_BYTES = 2
NODES = 2
GPUS_PER_NODE = 4
TOTAL_GPUS = NODES * GPUS_PER_NODE

CANDIDATES = [
    {"tp": 1, "pp": 1, "dp": 8, "domain": "single_gpu"},
    {"tp": 2, "pp": 1, "dp": 4, "domain": "intra_node"},
    {"tp": 4, "pp": 1, "dp": 2, "domain": "intra_node"},
    {"tp": 8, "pp": 1, "dp": 1, "domain": "inter_node"},
    {"tp": 4, "pp": 2, "dp": 1, "domain": "pipeline_across_nodes"},
]

LINKS = {
    "single_gpu": {"bandwidth_gb_s": None, "step_us": 0},
    "intra_node": {"bandwidth_gb_s": 100, "step_us": 5},
    "inter_node": {"bandwidth_gb_s": 25, "step_us": 20},
    "pipeline_across_nodes": {"bandwidth_gb_s": 100, "step_us": 5},
}

total_weight_gib = PARAMETERS * BYTES_PER_PARAMETER / 2**30
message_bytes = DECODE_BATCH * HIDDEN * ACTIVATION_BYTES
usable_gib = GPU_GIB - RESERVE_GIB

rows = []
for candidate in CANDIDATES:
    tp = candidate["tp"]
    pp = candidate["pp"]
    weight_gib = total_weight_gib / tp / pp
    link = LINKS[candidate["domain"]]
    if tp == 1:
        estimate_ms = 0.0
    else:
        wire_bytes = 2 * (tp - 1) / tp * message_bytes
        transfer_s = wire_bytes / (link["bandwidth_gb_s"] * 1e9)
        latency_s = 2 * (tp - 1) * link["step_us"] * 1e-6
        estimate_ms = (transfer_s + latency_s) * LAYERS * COLLECTIVES_PER_LAYER * 1e3
    assert candidate["tp"] * candidate["pp"] * candidate["dp"] == TOTAL_GPUS
    fully_modeled = pp == 1
    rows.append({
        **candidate,
        "weight_gib_per_gpu": round(weight_gib, 2),
        "fits": weight_gib <= usable_gib,
        "decode_collective_estimate_ms": round(estimate_ms, 2),
        "fully_modeled": fully_modeled,
        "passes": (weight_gib <= usable_gib and estimate_ms < COLLECTIVE_LIMIT_MS) if fully_modeled else None,
    })

eligible = [r for r in rows if r["passes"] is True]
choice = min(eligible, key=lambda r: r["tp"]) if eligible else None
selected = {k: choice[k] for k in ("tp", "pp", "dp")} | {"cp": 1} if choice else None

artifact = {
    "assumptions": {
        "parameters": PARAMETERS,
        "model_revision": MODEL_REVISION,
        "engine_version": ENGINE_VERSION,
        "runtime_version": RUNTIME_VERSION,
        "bytes_per_parameter": BYTES_PER_PARAMETER,
        "gpu_gib": GPU_GIB,
        "runtime_and_kv_reserve_gib": RESERVE_GIB,
        "layers": LAYERS,
        "hidden_size": HIDDEN,
        "input_context_distribution": INPUT_CONTEXT_DISTRIBUTION,
        "decode_batch": DECODE_BATCH,
        "activation_bytes": ACTIVATION_BYTES,
        "collectives_per_layer": COLLECTIVES_PER_LAYER,
        "collective_limit_ms": COLLECTIVE_LIMIT_MS,
        "topology": {"nodes": NODES, "gpus_per_node": GPUS_PER_NODE, "total_gpus": TOTAL_GPUS},
        "effective_links": LINKS,
        "bandwidth_unit": "decimal GB/s; null means no TP communication",
        "link_model_scope": "TP ring collectives only; PP transfers and bubbles are not modeled",
    },
    "candidates": rows,
    "selected": selected,
    "selection_rationale": (
        "Choose the smallest TP among listed fully modeled PP1 candidates passing both criteria; "
        "this is a screening choice, not a measured optimum."
    ) if choice else "No listed fully modeled candidate passes both criteria.",
    "failure_domain_rationale": {
        "selected_domain": choice["domain"] if choice else None,
        "assumption": "Treat loss of any required rank as failure of its replica.",
        "placement": "Keep each TP group within one node when TP fits the node.",
        "node_failure": "A node outage removes every replica with a rank on that node; shared services may create wider failures.",
    },
    "selection_scope": "listed fully-modeled PP1 candidates only",
}
print(json.dumps(artifact, indent=2, allow_nan=False))

Run:

python plan_parallelism.py > parallelism-plan.json

Expected checks:

TP1: fits=false
TP2: weight_gib_per_gpu=65.19, decode_collective_estimate_ms=2.44, passes=true
TP8: weight_gib_per_gpu=16.30, decode_collective_estimate_ms=50.67, passes=false
selected: TP2 × PP1 × DP4 × CP1

Deliberately Broken Configuration

The following configuration is valid enough to launch on some distributed runtimes, but it is broken for this workload:

world_size: 8
tensor_parallel_size: 8
pipeline_parallel_size: 1
data_parallel_size: 1
placement: spread_across_two_nodes
reason: "use every GPU for the lowest per-rank weight memory"

It fails for three measurable reasons:

1. TP2 already satisfies the 68 GiB fit budget, so TP8 solves no remaining fit problem.

2. The planning estimate is about 50.67 ms per decode forward pass, above the 5 ms criterion.

3. One failed rank removes the only replica; the selected TP2/DP4 plan has four replica failure domains.

A second failure may stay invisible until the system is under load: distributing ranks without checking GPU-to-NIC and GPU-to-GPU affinity can send traffic through a slower path. Current NCCL detects topology automatically, but the deployment must still expose the right devices and network interfaces to it (NVIDIA NCCL overview).

A Second Broken All-to-All Plan

Suppose a long-context backend uses two sequence-transpose all-to-all operations per block over CP=8, and each rank owns 4,096 BF16 token activations with hidden size 8,192. Unified Sequence Parallelism describes why all-to-all and ring-based sequence parallelism have different topology trade-offs (Fang and Zhao, 2024). This exercise simplifies the cost: every byte a rank sends to another rank is charged to the 25 GB/s inter-node link, and the 160 operations run one after another:

local activation M       = 4,096 × 8,192 × 2
                         = 67,108,864 bytes = 64 MiB
bytes sent to other ranks = 7/8 × M
                         = 58,720,256 bytes
one bandwidth-only floor = 58,720,256 / 25,000,000,000 seconds
                         = 2.34881024 ms ≈ 2.35 ms
160 all-to-alls           = 2.34881024 × 160 ms
                         = 375.8096384 ms ≈ 375.81 ms

Under these serial-transfer assumptions, the estimate fails the toy communication criterion. Charging all seven peers to the inter-node link is conservative, because three of them share the sending rank's node; charging only the half of M that leaves the node (33,554,432 bytes, about 1.34 ms per operation and 214.75 ms for 160 operations) still fails. Neither figure measures the exposed latency of an overlapped or topology-aware implementation. An implementation may overlap much of this traffic or choose another algorithm, but that possibility is something to measure, not permission to erase the data movement from the plan.

Hands-On Exercise

Change the artifact generator for this new requirement:

• output-heavy workload;

• 24 GiB per GPU reserved for KV cache and runtime;

• same 8 GPUs and link assumptions;

• collective-estimate limit remains 5 ms.

Answer these questions in a placement-decision.md file:

1. Does TP2 still fit?

2. What is the smallest TP size that fits?

3. How many independent replicas remain?

4. Does the smallest fitting layout cross a node boundary?

5. If the answer changes, which constraint changed it: model fit, collective estimate, or failure isolation?

Expected memory calculation:

usable per GPU = 80 - 24 = 56 GiB
TP2 weight     = 65.19 GiB -> does not fit
TP4 weight     = 32.60 GiB -> fits

TP4 × PP1 × DP2 is the smallest fitting TP-only layout in the listed candidates and keeps each TP group inside one four-GPU node. Its collective estimate under the same formula is approximately 6.06 ms, so it fails the original 5 ms objective. That result is intentional: none of the fully modeled PP1 candidates passes this screen. It does not establish global infeasibility, because a layout such as TP2 × PP2 × DP2, absent from this list, still deserves analysis of pipeline transfers, scheduling, and memory. Expand the candidate set or revise the hardware, precision, reserve, or SLO, then benchmark; do not hide the failed criterion.

Measurable Exit Criteria

Your tutorial artifact passes only if all of the following are true:

• every candidate satisfies TP × PP × DP = 8 or explicitly marks unused GPUs;

• weight memory and reserve use the same GiB unit;

• no selected rank exceeds 80 GiB;

• the selected TP groups do not cross nodes unless the model cannot otherwise fit;

• the selected plan reports an estimated collective cost below 5 ms, or explicitly reports that no candidate passes;

• the placement diagram identifies independent replica failure domains;

• assumptions include model revision, precision, engine/runtime versions, input-context distribution, GPU topology, and measured effective bandwidth before production approval; and

• an actual load test replaces the planning estimate before capacity is promised.

Retrieval Practice

1. Which parallelism strategy creates independent request-serving replicas?

2. Why can tensor parallelism make a model fit yet worsen decode latency?

3. What is the semantic difference between all-gather and reduce-scatter?

4. When is context parallelism justified?

5. In the worked example, why is TP2 preferred to TP8?

6. What should you do when no candidate satisfies both memory and communication criteria?

Answer Key

1. Data parallelism.

2. It shards weights but adds repeated collectives to the forward-pass critical path.

3. All-gather reconstructs the full tensor from shards on every rank; reduce-scatter reduces values and leaves only one result shard on each rank.

4. When sequence-state memory or long-context attention computation cannot fit or meet the objective on one rank; it is not a default short-context throughput knob.

5. TP2 is the smallest group that fits, stays within each node, gives four replicas, and has a 2.44 ms planning estimate versus about 50.67 ms for cross-node TP8.

6. Report that none of the evaluated candidates passes; unmodeled layouts remain unresolved. Expand the search or change precision, hardware, reserve, topology, or the objective and remeasure. Do not silently drop a criterion.

What This Post Does Not Cover

This tutorial covers dense-model placement. It does not cover expert routing, expert load balancing, or expert-parallel collectives; those follow in I-05. It also does not claim that the simple ring formula predicts production latency. Kernel traces, collective benchmarks, queueing, KV capacity, and end-to-end load tests remain required.

I-07 turns a deployed layout into a reproducible benchmark, derives SLO-conditioned capacity, and defines the telemetry needed to know when the plan is wrong.

Source Notes

• Shoeybi, M., Patwary, M., Puri, R., et al. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism." 2019. Primary source for the influential intra-layer tensor-parallel design. arxiv.org/abs/1909.08053

• Narayanan, D., Shoeybi, M., Casper, J., et al. "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM." 2021. Primary source for composing tensor, pipeline, and data parallelism. arxiv.org/abs/2104.04473

• Huang, Y., Cheng, Y., Bapna, A., et al. "GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism." 2019. Primary source for microbatch pipeline parallelism and bubble amortization. arxiv.org/abs/1811.06965

• Liu, H., Zaharia, M., and Abbeel, P. "Ring Attention with Blockwise Transformers for Near-Infinite Context." 2023. Primary source for distributing long-sequence attention across devices. arxiv.org/abs/2310.01889

• Fang, J., and Zhao, S. "USP: A Unified Sequence Parallelism Approach for Long Context Generative AI." 2024. Primary source comparing all-to-all and ring-based sequence-parallel approaches. arxiv.org/abs/2405.07719

• NVIDIA. "Collective Communication Functions." Current NCCL documentation, accessed 2026-11-02. First-party definitions of all-reduce, all-gather, reduce-scatter, and all-to-all semantics. github.com/NVIDIA/nccl

• vLLM Project. "Parallelism and Scaling." Current documentation, accessed 2026-11-02. First-party deployment guidance for tensor and pipeline parallelism across single- and multi-node systems. docs.vllm.ai