Inference Infrastructure

Measure useful capacity with a frozen workload, per-class latency goals, queue evidence, and failure tests.

Compare data, tensor, pipeline, and context parallelism using memory, communication, and failure boundaries.

Separate model routing from expert placement, and calculate the memory and communication each layout needs.

Balance prefix reuse against queue load, compatibility, and tenant isolation when routing inference requests.

Decide when phase separation is worth the cost of KV transfer, extra coordination, and a more complex deployment.

Calculate memory per token, allocate complete blocks, and distinguish snapshot capacity from room to finish.

Explore token-budget scheduling, chunked prefill, preemption, and backpressure in a small runnable simulation.

Trace a request through prefill, decode, caching, and streaming—and reconcile its timing and memory costs.