#KV Cache

Balance prefix reuse against queue load, compatibility, and tenant isolation when routing inference requests.

Calculate memory per token, allocate complete blocks, and distinguish snapshot capacity from room to finish.

Trace a request through prefill, decode, caching, and streaming—and reconcile its timing and memory costs.