Inference Infrastructure14 min
Managing the KV Cache
Calculate memory per token, allocate complete blocks, and distinguish snapshot capacity from room to finish.
Huang Tzu Lin
TagBlog
#Paged Attention
1 post
1 post
Calculate memory per token, allocate complete blocks, and distinguish snapshot capacity from room to finish.