Inference Infrastructure16 min
Paged KV Cache: GPU Memory Management for LLM Serving
In Post I-00, we traced a single API call through the inference pipeline and introduced the KV cache: the data structure that stores attention key-value vectors so the model does not recompute them at every decode step. The KV cache grows with every generated token, and it must reside in GPU memo...
Huang Tzu Lin