Deploying Mixture-of-Experts Models

A mixture-of-experts model replaces some dense feed-forward sublayers with several expert feed-forward networks and a learned router. Each token activates only a subset of the routed experts, while shared layers (and, in some models, shared experts) still run for every token. Sparse activation therefore separates total parameters stored from parameters active for one token.

The architecture does not dictate a single deployment. Experts can be tensor-sharded, assigned whole to expert-parallel ranks, or replicated, and each choice can be combined with data or tensor parallelism for attention and with pipeline or context parallelism. The best layout depends on precision, concurrency, batch shape, collectives, topology, and expert skew.

Layer A: The MoE Model

Experts Are Feed-Forward Paths, Not Complete Models

In a common Transformer MoE block, attention remains a shared sublayer while the dense feed-forward network is replaced by N expert feed-forward networks. A router scores each token and activates k routed experts. Some architectures also have one or more always-on shared experts or alternate dense and MoE blocks.

The sparsely gated MoE layer predates modern LLM serving (Shazeer et al., 2017). Switch Transformers later demonstrated top-1 routing and reported reaching T5-baseline quality up to 7× faster in pretraining with the same computational resources. That is a training result, not a serving-latency result (Fedus, Zoph, and Shazeer, 2022).

Total vs. Active Parameters

Mixtral 8×7B has 46.7B total and 12.9B active parameters, eight experts per MoE layer, and top-2 routing. These parameter counts come from the Mistral release. The active-parameter count is only a rough proxy for compute; latency still depends on weight access, utilization, dispatch, and communication. The technical report describes the architecture and routing analysis.

DeepSeek-V3 has 671B total and 37B active parameters, with 256 routed experts plus one shared expert per MoE layer and eight routed experts selected per token (DeepSeek-V3 Technical Report). Saying it "computes like a 37B dense model" is too strong: the cost still includes attention and shared layers, expert dispatch, grouped GEMMs, load imbalance, and communication.

Router Functions Vary

A textbook router computes linear scores, applies softmax, picks the top-k experts, and forms a weighted sum. DeepSeek-V3 shows that this recipe is not universal: it computes sigmoid affinity scores, adds a load-balancing bias only when selecting the top-k experts, and normalizes the selected experts' original (unbiased) scores to form the gating values. Take routing details from each model's own architecture, not from a generic diagram (DeepSeek-V3 Technical Report).

Expert specialization is also not guaranteed to align with human labels such as "travel" or "math." Mixtral's routing analysis found limited topic-level specialization and some syntactic and positional patterns.

Layer B: Deployment Strategies

Full Replica

When a complete MoE model fits at a supported precision, replicate it if independent replicas deliver better request throughput for the target workload than a communication-heavy sharded layout. Each replica holds all the weights and its own KV capacity, but ordinary inference needs no cross-replica collectives. The claim that "replication does not work for MoE" is false; replication is a memory-versus-performance trade-off.

Tensor-Parallel Experts

Each expert's matrices can be split across TP ranks. Every rank holds a slice of every expert and combines partial outputs with TP collectives. Routing destinations stay inside the TP group, but partial activations cross the group repeatedly.

Expert Parallelism

Expert parallelism assigns whole routed experts to EP ranks. Each token's activation is dispatched to the ranks that hold its selected experts, and the results return to be combined. Conceptually the flow is dispatch followed by combine, but an engine may implement it with all-to-all, all-gather plus reduce-scatter, all-reduce, or fused point-to-point kernels.

GShard established large-scale conditional-computation sharding, and DeepSpeed-MoE developed training and inference techniques for large expert models (Lepikhin et al., 2020; Rajbhandari et al., 2022). Neither result makes any single EP degree optimal on current hardware.

Attention DP/TP Plus EP

"Replicate attention, shard experts" is one layout, not a law. Current vLLM behavior depends on TP and DP: with TP=1, attention weights are replicated across all DP ranks; with TP>1, attention is tensor-sharded across the TP ranks inside each DP group. In both cases the experts span one EP group of size EP = TP × DP (vLLM Expert Parallel Deployment).

Replicated attention weights do not mean every token traverses every replica. Different DP ranks normally process different partitions of requests and tokens with the same weights. DP ranks that share an EP group are coupled through expert execution, so they are neither independent complete-model replicas nor independent failure domains.

Collective Flow for One MoE Block

For one token batch under a common attention-DP/EP design:

local tokens
  -> local shared attention
  -> router scores and top-k selection
  -> dispatch selected token activations to expert owners
  -> grouped expert GEMMs
  -> combine/return expert outputs
  -> weighted expert sum + residual

If B tokens are uniformly routed to k of N experts, the mean assignment count is:

expected assignments per expert = B × k / N

The uniform mean is an expectation, not a guarantee. Hot experts can create stragglers, longer tail latency, and idle ranks; the effect depends on expert placement, execution capacity, and overlap. Expert-parallel load balancing (EPLB), redundant copies of hot experts, capacity policies, and training-time load balancing each address a different part of the problem.

Broken Fit Check: Mixtral BF16 on One 80 GiB GPU

The weight-only requirement is:

BF16 bytes = 46.7 billion × 2
           = 93.4 billion bytes

BF16 GiB   = 93,400,000,000 / 2^30
           = 86.99 GiB

The weights alone exceed 80 GiB before counting KV cache, activations, allocator state, and runtime buffers, so all BF16 weights cannot stay resident on that GPU. Weight offloading is a separate design with its own transfer and latency costs.

An idealized one-byte-per-weight checkpoint is:

FP8 weight GiB = 46,700,000,000 / 2^30 = 43.49 GiB

That may fit with a declared runtime and KV reserve, but the actual footprint of quantization metadata, kernel support, output quality, and support for the checkpoint format must all be validated. The BF16 failure does not prove that expert parallelism is required: TP2, a GPU with more memory, or a supported lower-precision replica are all alternatives.

Worked Example: A Transparent Hypothetical MoE

This deliberately simple parameter-accounting fixture makes every placement number derivable. Treat the eight experts as equal-sized banks that span all modeled MoE layers; dispatch is calculated for one layer. Real architectures require summing each layer's expert and shared parameters. "Active parameters" here is a nominal path count, not an exact FLOP or memory-traffic count; an embedding lookup, for example, does not read every embedding weight for each token:

Assume BF16 weights, EP8, and 20 GiB per GPU reserved for KV/runtime. Each EP rank holds all 8B shared parameters plus one 8B expert:

per-rank parameters = 8B + 8B = 16B
per-rank BF16 GiB   = 16,000,000,000 × 2 / 2^30
                    = 29.80 GiB
with reserve        = 29.80 + 20
                    = 49.80 GiB < 80 GiB

For 256 routed tokens, top-2, eight experts:

mean assignments/expert = 256 × 2 / 8 = 64

For one rank starting with 32 tokens, hidden size 4,096, BF16 activations, and top-2:

logical dispatch bytes/rank = 32 × 2 × 4,096 × 2
                            = 524,288 bytes
                            = 0.5 MiB

Some selected experts live on the local rank, and optimized kernels may merge messages, so 0.5 MiB is the logical activation volume before routing locality and protocol overhead are counted. It is not measured wire traffic.

Build the Placement Calculator

Save as model_moe_placement.py:

import json

GIB = 2**30
GPU_GIB = 80
RESERVE_GIB = 20

mixtral_total_parameters = 46_700_000_000
mixtral_bf16_gib = mixtral_total_parameters * 2 / GIB
mixtral_fp8_gib = mixtral_total_parameters * 1 / GIB

shared_parameters = 8_000_000_000
experts = 8
expert_parameters_each = 8_000_000_000
top_k = 2
ep_size = 8
if not 1 <= top_k <= experts:
    raise ValueError("top_k must be between 1 and experts")
if ep_size <= 0 or experts % ep_size:
    raise ValueError("this equal-placement fixture requires experts divisible by ep_size")

total_parameters = shared_parameters + experts * expert_parameters_each
active_parameters = shared_parameters + top_k * expert_parameters_each
experts_per_rank = experts // ep_size
parameters_per_ep_rank = shared_parameters + experts_per_rank * expert_parameters_each
weight_gib_per_ep_rank = parameters_per_ep_rank * 2 / GIB

batch_tokens = 256
expected_assignments_per_expert = batch_tokens * top_k / experts
if batch_tokens % ep_size:
    raise ValueError("this even-token fixture requires batch_tokens divisible by ep_size")
local_tokens = batch_tokens // ep_size
hidden_size = 4096
activation_bytes = 2
logical_dispatch_bytes_per_rank = local_tokens * top_k * hidden_size * activation_bytes

artifact = {
    "mixtral_fit_check": {
        "total_parameters": mixtral_total_parameters,
        "bf16_weight_gib": mixtral_bf16_gib,
        "fp8_weight_gib_idealized": mixtral_fp8_gib,
        "bf16_fits_80gib_before_overhead": mixtral_bf16_gib <= GPU_GIB,
        "reserve_gib": RESERVE_GIB,
        "fp8_fits_with_reserve": mixtral_fp8_gib + RESERVE_GIB <= GPU_GIB,
    },
    "hypothetical_ep_model": {
        "total_parameters": total_parameters,
        "active_parameters_per_token": active_parameters,
        "experts_per_rank": experts_per_rank,
        "bf16_weight_gib_per_rank": weight_gib_per_ep_rank,
        "bf16_weight_plus_reserve_gib": weight_gib_per_ep_rank + RESERVE_GIB,
        "expected_assignments_per_expert": expected_assignments_per_expert,
        "logical_dispatch_bytes_per_rank": logical_dispatch_bytes_per_rank,
    },
}
serialized = json.dumps(artifact, indent=2)
with open("moe-placement.json", "w", encoding="utf-8") as handle:
    handle.write(serialized + "\n")
print(serialized)

Expected checks:

Mixtral BF16 weight ≈ 86.99 GiB, single-80-GiB fit=false
Mixtral idealized FP8 weight ≈ 43.49 GiB, fit with 20-GiB reserve=true
hypothetical total=72B, active=24B
EP8 experts/rank=1
BF16 weight/rank ≈ 29.80 GiB
weight+reserve ≈ 49.80 GiB
expected assignments/expert=64
logical dispatch/rank=524288 bytes

Date-Stamped Open-Model Landscape

These selected published architectures were checked against first-party sources on 2026-11-02. The table is neither an exhaustive inventory nor a performance ranking. Expert counts are per MoE layer or block; parameter counts describe the whole named model:

The table shows several published uses of MoE. A selected sample cannot establish how prevalent MoE is or reveal every frontier architecture; treat a proprietary model's architecture as unknown unless its developer confirms it.

Deliberately Broken Cases

BF16 Mixtral on One 80 GiB GPU

The 86.99 GiB weight-only requirement fails before any runtime memory is counted. Shrinking the batch or paging the KV cache cannot make all BF16 weights fit in GPU memory; weight offloading changes the residency assumption and needs its own benchmark.

Universal Replicate-Attention/Shard-Experts Rule

A plan that forces replicated attention ignores TP groups, context parallelism, and engine behavior. Current vLLM can tensor-shard attention while experts span TP×DP; the plan must name both group structures.

Assuming All-to-All Is Always Best

AMD's vLLM MoE playbook reports a crossover that depends on hardware and workload. In its tested path, DP+EP communicates with all-to-all while TP+EP uses only all-reduce, and the ultra-sparse Llama 4 Maverick performed 7%–12% better without the EP flag than with it on that MI300X setup (AMD ROCm MoE playbook). The vendor test, published 24 November 2025, used Llama-4-Maverick-17B-128E-Instruct-FP8 on 8× MI300X, 8,192 input and 1,024 output tokens per request, 640 requests, AITER enabled, vLLM 0.11.1rc3.dev69+g269c4db0a, and ROCm 7.0.51831. The post does not state cache warmth. Treat the result as a vendor-reported comparison under those conditions, not as an independently reproduced gain or a universal threshold.

Ignoring Expert Skew

If one expert receives 160 assignments while the uniform mean is 64, its rank can become a straggler when the ranks otherwise have comparable expert execution capacity. The average active-parameter count does not reveal this tail.

Hands-On Exercise

1. Reproduce the Mixtral BF16/FP8 fit check with a 12 GiB and then 24 GiB reserve.

2. Change the hypothetical model from EP8 to EP4. Compute experts/rank and weight+reserve GiB.

3. Reset to EP8, then compare top-1, top-2, and top-4 routing for 256 tokens; compute mean assignments/expert and logical dispatch bytes.

4. Create a skewed assignment vector whose total equals 256 × top_k; report max/mean and the overloaded rank.

5. Draw TP2×DP4 attention groups and EP8 expert ownership on eight GPUs.

6. Before a deployment decision, benchmark at least one TP-heavy and one DP+EP layout on the same hardware and workload. Without GPU access, complete the calculator and the benchmark plan, and mark the deployment choice as pending measurement.

Expected EP4 memory for the hypothetical model:

experts/rank = 8 / 4 = 2
parameters/rank = 8B shared + 2×8B experts = 24B
BF16 weight/rank = 24B × 2 / 2^30 ≈ 44.70 GiB
weight + 20 GiB reserve ≈ 64.70 GiB

At EP8, top-1/top-2/top-4 give 32/64/128 mean assignments per expert and 262,144/524,288/1,048,576 logical dispatch bytes per rank. At EP4, an even partition of 256 tokens starts each rank with 64 tokens. A top-2 skew example is [160, 64, 64, 64, 48, 40, 40, 32]: its sum is 512 and max/mean is 2.5; with one expert per rank, rank 0 has the most assignments. For TP2×DP4, attention groups are {0,1}, {2,3}, {4,5}, {6,7} and the EP group spans {0,…,7}. These are fixture calculations, not measured latency.

Measurable Exit Criteria

The tutorial passes when:

• model routing and deployment routing are separate concepts;

• router function is model-specific and DeepSeek sigmoid gating is included as a counterexample;

• Mixtral BF16 weight-only memory is about 86.99 GiB and is rejected on one 80 GiB GPU;

• total and active parameters are never treated as complete memory/latency proxies;

• attention replication/sharding is derived from TP/DP groups;

• collective choice is tied to engine, topology, message size, and workload;

• the hypothetical EP8 calculation yields 29.80 GiB weight/rank and 49.80 GiB with reserve;

• expert assignment skew is included in failure analysis; and

• all model-landscape claims are date-stamped and first-party-confirmed.

Retrieval Practice

1. What is the difference between total and active MoE parameters?

2. Why is softmax top-k not a universal router description?

3. Can full MoE replication ever be correct?

4. Why does replicated attention not mean every token visits every replica?

5. What does expert parallelism trade against memory distribution?

6. Why does a uniform assignments/expert formula not guarantee latency balance?

Answer Key

1. Total parameters determine all stored weights; active parameters count the shared and selected paths used for one token.

2. DeepSeek-V3 uses sigmoid affinities, bias-adjusted selection, and selected-score normalization; other models also vary.

3. Yes, if the checkpoint plus runtime/KV fits and independent replicas outperform communication-heavy sharding for the target workload.

4. DP ranks normally process different request/token partitions with the same weights; a token stays in its assigned replica/group unless another parallel dimension moves it.

5. It reduces and redistributes per-rank expert-weight memory and aggregates memory bandwidth across ranks, but it adds dispatch/combine communication and imbalance risk.

6. Learned routing is data-dependent; hot experts can receive far more than the mean and become stragglers.

What This Post Does Not Cover

This post assumes I-06's dense-parallelism foundation. I-07 supplies the benchmark, SLO, observability, and failure method needed to choose between candidate MoE layouts. Training-time router losses are mentioned only where they affect deployment behavior.

Source Notes

• Shazeer, N., Mirhoseini, A., Maziarz, K., et al. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." ICLR, 2017. Foundational sparsely gated MoE source. arxiv.org/abs/1701.06538

• Fedus, W., Zoph, B., and Shazeer, N. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." JMLR, 2022. Primary top-1 routing and scoped pretraining-speed source. arxiv.org/abs/2101.03961

• Lepikhin, D., Lee, H., Xu, Y., et al. "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding." 2020. Primary conditional-computation sharding source. arxiv.org/abs/2006.16668

• Rajbhandari, S., Li, C., Yao, Z., et al. "DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale." ICML, 2022. Primary expert-model inference/sharding source. arxiv.org/abs/2201.05596

• Jiang, A. Q., Sablayrolles, A., Roux, A., et al. "Mixtral of Experts." 2024. Primary Mixtral architecture, parameter, routing, and utilization caveat source. arxiv.org/abs/2401.04088

• DeepSeek-AI. "DeepSeek-V3 Technical Report." 2024. Primary 671B/37B architecture, sigmoid routing, load balance, and deployment source. arxiv.org/abs/2412.19437

• vLLM Project. "Expert Parallel Deployment." Current documentation, accessed 2026-11-02. First-party current TP/DP/EP attention and expert layout behavior. docs.vllm.ai

• AMD ROCm. "The vLLM MoE Playbook: A Practical Guide to TP, DP, PP and Expert Parallelism." 2025-11. Hardware-specific vendor benchmarks retained as conditional deployment evidence. rocm.blogs.amd.com