Inference Infrastructure21 min
Prefix-Aware Routing: Cache-Conscious Request Distribution
In Post I-02, we saw that PagedAttention enables different requests to share physical KV cache blocks on the same replica. Two requests with the same system prompt can point to the same physical blocks rather than storing duplicate copies. That sharing mechanism is real and it works -- but only i...
Huang Tzu Lin