#Distributed Inference

An inference server becomes distributed for one of two reasons: the model does not fit on one accelerator, or one accelerator cannot meet the required latency and throughput. Those reasons sound similar, but they lead to different designs. Splitting one request across more GPUs can make the model...

A mixture-of-experts model replaces some dense feed-forward sublayers with several expert feed-forward networks and a learned router. Each token activates only a subset of routed experts, while shared layers—and sometimes shared experts—still run for every token. Sparse activation separates total...