#Latency & Throughput

Balance prefix reuse against queue load, compatibility, and tenant isolation when routing inference requests.

Decide when phase separation is worth the cost of KV transfer, extra coordination, and a more complex deployment.

Explore token-budget scheduling, chunked prefill, preemption, and backpressure in a small runnable simulation.

Trace a request through prefill, decode, caching, and streaming—and reconcile its timing and memory costs.