#Networking

An inference server becomes distributed for one of two reasons: the model does not fit on one accelerator, or one accelerator cannot meet the required latency and throughput. Those reasons sound similar, but they lead to different designs. Splitting one request across more GPUs can make the model...