Managing the KV Cache

The KV cache is dynamic per-request state: it grows as tokens are processed, varies in length across sequences, and competes with model weights and runtime workspaces for accelerator memory. Paging stores that state in fixed-size, non-contiguous blocks, so no request needs one contiguous reservation. It does not make memory infinite or deliver a universal capacity multiplier, and KV growth, prefix retention, quantization, and multi-tier placement remain separate engineering problems.

This tutorial sizes KV memory for an explicit GQA architecture, walks through an illustrative 825-token allocation example, compares contiguous reservation against paged blocks under a declared 12 GiB budget, and extends the design to prefix reuse, quantized KV, host offload, and distributed tiering.

Freeze the Architecture and Memory Budget

The worked example reuses the hypothetical 80-layer GQA model from I-00:

A 16-token block matches the original PagedAttention default, not a universal rule: block size is an engine, attention-kernel, and hardware-backend choice.

Three Different Kinds of Waste

Reservation Waste

The comparison policy in this tutorial reserves each request's maximum sequence length up front. Contiguous allocation does not require that policy; allocators that grow or compact regions avoid it but pay their own costs, such as copying. When the policy reserves 2,048 token slots for a sequence that currently holds 825 processed tokens, the other 1,223 slots stay unavailable to other requests even if generation stops on the next step.

Internal Fragmentation

Allocating memory in fixed-size chunks leaves unused slots whenever a sequence does not end on a chunk boundary. Under simple single-pool paging without lookahead slots or speculative reservations, only a sequence's final block can be partially empty, bounding internal tail waste to at most block_size - 1 slots per active sequence. Hybrid attention pools, speculative decoding lookahead slots, and retained shared blocks require separate accounting.

External Fragmentation

Variable-length contiguous allocations scatter free regions of uneven sizes across the pool, so a new request can fail admission even when total free bytes exceed its demand. Uniform physical blocks remove this contiguous-region requirement, but block-table metadata, attention-kernel indexing, and placement overhead remain.

What the 2023 PagedAttention Result Means

Figure 2 of Kwon et al., which profiles memory during the paper's §6.2 experiment, shows token states occupying only 20.4%–38.2% of KV-cache memory under the authors' reimplementations of Orca (Max), Orca (Pow2), and Orca (Oracle); the remaining roughly 60%–80% went to reservations, fragmentation, and other overhead. These are specific profiled allocation policies. Separately, the paper reports 2–4× higher whole-system throughput at matched latency against its evaluated baselines (FasterTransformer and those Orca variants), a result that combines PagedAttention kernels, block-table allocation, iteration-level scheduling, and copy-on-write sharing (Kwon et al., SOSP 2023).

Those numbers are historically important, but they do not mean every current engine wastes 60%–80% of KV memory, or that paging alone gives every workload 2–4× throughput. Current serving stacks already start from paged or related allocation and layer GQA/MLA, prefix caching, quantized KV, and tiered storage on top.

Reproduce Bytes per Token

For standard MHA or GQA KV tensors:

KV_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element

GQA example:

2 × 80 × 8 × 128 × 2 = 327,680 bytes/token = 0.3125 MiB/token

An otherwise matching 64-KV-head MHA model requires:

2 × 80 × 64 × 128 × 2 = 2,621,440 bytes/token = 2.5 MiB/token

That 8× footprint gap comes from KV-head sharing, not total parameter count. MLA, sliding-window attention, recurrent layers, and hybrid state-space layers store different per-token or fixed-size states and need their own derivations.

Worked Example: The 825-Token Request

Contiguous Reservation

allocated slots = 2,048
used slots      = 825
unused slots    = 2,048 - 825 = 1,223
unused fraction = 1,223 / 2,048 = 59.72%

Paged Allocation With 16-Token Blocks

blocks required = ceil(825 / 16) = 52
allocated slots = 52 × 16 = 832
used slots      = 825
unused slots    = 832 - 825 = 7
internal waste  = 7 / 832 = 0.8413%

For this 825-token snapshot, the per-request allocation ratio is:

contiguous / paged = 2,048 / 832 = 2.4615×

That is 2.46×, not 4–5×. The ratio tracks how much of the 2,048-slot reservation a request actually fills: shorter active sequences widen it, and sequences running near the reservation ceiling narrow it.

Worked Capacity Under a 12 GiB KV Budget

Treat 12 GiB as a hypothetical HBM slice left for this unsharded logical KV pool after weights, activations, and runtime workspaces have been subtracted; it is not the total VRAM needed to serve the model. In multi-GPU deployments, size each rank from its own KV-head and layer shard and its own free HBM rather than summing raw card memory across the node. Under our GQA assumptions:

12 GiB = 12 × 2^30 = 12,884,901,888 bytes
token slots = floor(12,884,901,888 / 327,680)
            = 39,321 slots

At 16 tokens per block, the pool holds floor(39,321 / 16) = 2,457 complete blocks, or 39,312 block-backed token slots; the leftover 3 MiB (9.6 tokens of KV) cannot form another block. If every active request currently sits at 825 tokens:

contiguous requests = floor(39,321 / 2,048) = 19
paged requests      = floor(39,312 / 832)   = 47
capacity ratio      = 47 / 19               = 2.4737×

The 47-request figure is an instantaneous occupancy ceiling at 825 tokens, not a guarantee that 47 requests can keep decoding concurrently to completion: as outputs grow into block 53 and beyond, the scheduler still needs headroom, conservative admission, or preemption. Even at the 825-token snapshot, a 48th request would need 48 × 832 = 39,936 slots, so no batch of 48 or more such requests fits under these assumptions.

Block Tables and Sharing

A block table maps each sequence's logical blocks to non-contiguous physical KV blocks in HBM. The model still sees one continuous token sequence while the attention kernel gathers K/V vectors through the block table, and the allocator returns a physical block to the free list once no active or retained sequence references it.

Reference counting and copy-on-write let multiple sequences point at the same immutable prefix blocks, allocating a fresh physical block only when two sequences diverge inside a shared tail block. The original vLLM paper applied this to parallel sampling, beam search, and shared prefixes that the service provider predefines and caches in advance. Paging permits sharing, but automatic cross-request reuse also needs a matching, retention, and eviction policy on top; I-04 covers vLLM automatic prefix caching and SGLang RadixAttention.

Quantized KV, Offload, and Tiering

Quantized KV

Storing K/V elements in idealized 1-byte FP8 rather than 2-byte BF16 cuts logical payload bytes in half:

FP8 GQA bytes/token = 2 × 80 × 8 × 128 × 1
                    = 163,840 bytes
ideal token slots   = floor(12 GiB / 163,840)
                    = 78,643

Real FP8 KV caches also store quantization scales, follow kernel alignment rules, and can change accuracy. vLLM's documentation notes that some attention layer types, such as sliding-window attention, are more sensitive to KV-cache quantization, and recommends calibrating scales against a dataset for the highest quality (vLLM Quantized KV Cache). Fitting more sequences into HBM does not by itself lower per-step latency.

Offload

Moving cold or preempted KV blocks to host DRAM or NVMe extends effective cache capacity, but every transfer back costs PCIe or network time and consumes interconnect bandwidth. An offload hit therefore has a different TTFT profile from an HBM hit.

Shared and Tiered KV

Disaggregated architectures such as Mooncake turn the GPU cluster's underused CPU, DRAM, and SSD resources into a distributed KV cache for separate prefill and decode clusters, which makes block placement, admission control, and RDMA transfer one scheduling problem (Qin et al., 2024). Evaluating a tiered cache requires tracking hit rates per storage tier, bytes transferred, transfer tail latency, eviction churn, and prefill compute saved.

Build the Capacity Calculator

Save as kv_capacity.py and run python kv_capacity.py > kv-capacity.json:

import json

LAYERS = 80
KV_HEADS = 8
HEAD_DIM = 128
BYTES_PER_ELEMENT = 2
KV_BUDGET_BYTES = 12 * 2**30
USED_TOKENS = 825
MAX_RESERVED_TOKENS = 2048
BLOCK_SIZE = 16

bytes_per_token = 2 * LAYERS * KV_HEADS * HEAD_DIM * BYTES_PER_ELEMENT
token_slots = KV_BUDGET_BYTES // bytes_per_token
if not (0 < USED_TOKENS <= MAX_RESERVED_TOKENS and BLOCK_SIZE > 0):
    raise ValueError("invalid sequence or block size")
paged_blocks = (USED_TOKENS + BLOCK_SIZE - 1) // BLOCK_SIZE
pool_blocks = KV_BUDGET_BYTES // (bytes_per_token * BLOCK_SIZE)
paged_slots = paged_blocks * BLOCK_SIZE
contiguous_requests = token_slots // MAX_RESERVED_TOKENS
paged_requests = pool_blocks // paged_blocks

artifact = {
    "architecture": {
        "layers": LAYERS,
        "kv_heads": KV_HEADS,
        "head_dim": HEAD_DIM,
        "bytes_per_element": BYTES_PER_ELEMENT,
    },
    "bytes_per_token": bytes_per_token,
    "kv_token_slots": token_slots,
    "whole_pool_blocks": pool_blocks,
    "block_backed_token_slots": pool_blocks * BLOCK_SIZE,
    "request": {
        "used_tokens": USED_TOKENS,
        "contiguous_slots": MAX_RESERVED_TOKENS,
        "paged_blocks": paged_blocks,
        "paged_slots": paged_slots,
        "paged_unused_slots": paged_slots - USED_TOKENS,
        "paged_internal_waste_percent": 100 * (paged_slots - USED_TOKENS) / paged_slots,
    },
    "capacity": {
        "contiguous_requests": contiguous_requests,
        "paged_requests": paged_requests,
        "request_capacity_ratio": paged_requests / contiguous_requests,
    },
    "sensitivity": {
        "mha_64_kv_heads_bytes_per_token": 2 * LAYERS * 64 * HEAD_DIM * 2,
        "fp8_gqa_bytes_per_token": 2 * LAYERS * KV_HEADS * HEAD_DIM * 1,
    },
}
print(json.dumps(artifact, indent=2))

Expected core values:

bytes_per_token = 327680
kv_token_slots = 39321
paged_blocks = 52
paged_slots = 832
paged_unused_slots = 7
paged_internal_waste_percent ≈ 0.8413
contiguous_requests = 19
paged_requests = 47
request_capacity_ratio ≈ 2.4737

Deliberately Broken Cases

Universal 4–5× Claim

Our 825-token request against a 2,048-slot reservation yields a 2,048 / 832 = 2.4615× per-request reduction and 47 versus 19 requests under the declared budget (2.4737×). Quoting a flat 4–5× multiplier substitutes a different historical workload's utilization for this calculation.

Universal 16-Token Block

Hard-coding 16 tokens per block overlooks kernel and hardware constraints. At 825 tokens, a 32-token block happens to land on the same 832-slot multiple (26 × 32 = 832), whereas a 200-token sequence allocates 208 slots under block size 16 and 224 slots under block size 32. Always state the configured block size.

Wrong Head Count

Plugging 64 query heads into the formula cuts the computed capacity of the 12 GiB pool from 39,321 token slots to 4,915. That models an MHA variant with 8× larger KV state, not the 8-KV-head GQA model under test.

Memory Is "Solved"

Paging eliminates contiguous-allocation waste and external fragmentation. Capacity planning still has to account for output growth during decode, prefix-cache eviction, quantization accuracy, offload latency, tier placement, distributed transfer, and admission control under sustained overload.

Hands-On Exercise

Extend the calculator to compare four configurations:

1. BF16 GQA, block size 16;

2. BF16 GQA, block size 32;

3. idealized FP8 GQA, block size 16; and

4. BF16 MHA with 64 KV heads, block size 16.

Pack repeating three-request groups of lengths [200, 825, 1,525] into the 12 GiB KV budget until another complete group no longer fits. Report:

• allocated token slots per three-request group;

• useful token slots (200 + 825 + 1,525 = 2,550);

• tail-block waste per group;

• complete three-request groups fitting in the 12 GiB pool; and

• both useful-state bytes and full allocated-block bytes when offloading the 1,525-token request (wire transfer size also depends on transfer layout, metadata, scales, and padding).

Treat FP8 quality impact and offload transfer latency as separate empirical benchmark gates rather than deriving them from byte counts.

Assuming no cross-request sharing, independent tail blocks, and whole three-request groups (200 + 825 + 1,525 = 2,550 useful slots per group), the reference values are:

Multiply per-group slots by Complete groups to obtain full-group pool usage. Leftover blocks could hold part of another group, which this exercise intentionally does not admit. These are static allocation counts, not throughput results; throughput needs a runtime benchmark.

Measurable Exit Criteria

The KV tutorial passes when:

• architecture, KV heads, dtype, HBM budget, block size, and sequence lengths are explicit;

• BF16 GQA KV state equals 327,680 bytes/token;

• 12 GiB yields 39,321 whole token slots under the simplified formula (39,312 block-backed slots at block size 16);

• the 825-token request allocates 52 blocks (832 slots), leaving 7 unused slots (0.8413% internal waste);

• contiguous and paged snapshot capacities equal 19 and 47 requests under the declared baseline;

• historical 60%–80% waste and 2–4× throughput figures are scoped to their 2023 baselines rather than presented as universal current-engine results;

• block-level paging is distinguished from prefix-cache matching and retention;

• FP8 capacity is marked idealized until scaling overhead, kernel speed, and task accuracy are measured; and

• offload and tiered-KV evaluations include transfer latency, bandwidth, and per-tier hit rates.

Retrieval Practice

1. How does reservation waste differ from internal fragmentation?

2. How does a block table remove the requirement for one contiguous KV buffer per request?

3. Why does our worked example give a 2.47× request-capacity gain rather than 4–5×?

4. Does PagedAttention by itself decide which finished prompts stay cached for future requests?

5. Why can quantizing KV to FP8 double slot capacity without cutting per-token latency?

6. What bottleneck appears as soon as KV blocks spill from GPU HBM to host DRAM or SSD?

Answer Key

1. Reservation waste is future sequence positions held up to a maximum cap before the request reaches them; internal fragmentation is the unused remainder inside an already allocated block, such as the last 7 slots of a 52-block allocation for 825 tokens.

2. Each sequence's block table maps logical fixed-size chunks to arbitrary physical blocks in the KV pool, so growing a sequence only claims one free physical block at a time without relocating earlier blocks.

3. With 825 active tokens against a 2,048-slot reservation (832 paged slots per request) and a 39,321-token-slot budget (2,457 whole blocks), capacity moves from 19 contiguous requests to 47 paged requests (47 / 19 ≈ 2.4737×).

4. No. Paging provides non-contiguous block storage and reference-counted sharing; automatic cross-request reuse additionally requires prefix hashing or radix-tree matching, retention and eviction rules, and tenant isolation.

5. Halving bytes per element lets roughly twice as many token slots fit, but quantize/dequantize work, scale handling, and the larger batches that the extra capacity admits can offset the bandwidth savings on the critical path. Accuracy calibration and hardware and kernel support also matter.

6. Interconnect bandwidth and transfer latency across PCIe or RDMA, along with contention against other GPU transfers and the trade-off between fetching cold blocks and recomputing prefill.

What This Post Does Not Cover

I-04 covers local prefix retention and distributed cache-aware request routing. I-03 analyzes moving KV blocks between disaggregated prefill and decode pools. I-07 benchmarks end-to-end capacity and latency SLOs under realistic prompt and output distributions.

Source Notes

• Kwon, W., Li, Z., Zhuang, S., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP, 2023. Primary source for baseline KV utilization measurements, PagedAttention block tables, copy-on-write sharing, and vLLM system evaluations. arxiv.org/abs/2309.06180

• vLLM Project. "Quantized KV Cache." Current documentation, accessed 2026-11-02. First-party FP8 KV formats, scale calibration, and runtime constraints. docs.vllm.ai

• Qin, R., Li, Z., He, W., et al. "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving." arXiv:2407.00079, 2024. A conference version appeared at USENIX FAST 2025 as "Mooncake: Trading More Storage for Less Computation." Primary source for distributed DRAM/SSD KV tiering across disaggregated prefill and decode clusters. arxiv.org/abs/2407.00079

• Hugging Face. "Cache Strategies." Current Transformers documentation, accessed 2026-11-02. First-party overview of dynamic, static, offloaded, and quantized KV cache layouts. huggingface.co