Inference Infrastructure15 min
Distributing a Model Across GPUs
Compare data, tensor, pipeline, and context parallelism using memory, communication, and failure boundaries.
Huang Tzu Lin
TagBlog
#GPU Memory Management
3 posts
Inference Infrastructure14 min
Deploying Mixture-of-Experts Models
Separate model routing from expert placement, and calculate the memory and communication each layout needs.
Huang Tzu Lin
Inference Infrastructure14 min
Managing the KV Cache
Calculate memory per token, allocate complete blocks, and distinguish snapshot capacity from room to finish.
Huang Tzu Lin