The vLLM team introduced the vLLM V1 architecture with an O(1) zero-overhead prefix-caching allocator on top of paged KV blocks, making automatic prefix reuse fast enough to enable by default even when cache hit rates are low.
PagedAttention
Also known as: Paged KV Cache
Plain English
A GPU memory technique borrowed from operating systems that splits an LLM's short-term conversation memory into fixed-size pages so servers stop wasting VRAM and can serve far more users simultaneously.
Technical Overview
Traditional serving engines pre-allocated contiguous GPU memory buffers for each sequence's maximum KV cache length, wasting 60–80% of VRAM to fragmentation and over-reservation. PagedAttention partitions each request's KV cache into fixed-size non-contiguous physical blocks tracked by a logical-to-physical block table, enabling near-zero waste, copy-on-write prefix sharing, and higher batch concurrency inside vLLM and other Inference Engines.
Chronological Entries (2)
Kwon et al. published the PagedAttention paper alongside vLLM, adapting virtual-memory paging to allocate non-contiguous KV cache blocks on demand. Eliminating contiguous buffer fragmentation improved serving throughput by 2–4× over prior systems while enabling copy-on-write memory sharing for beam search and parallel sampling.
Direct Relations (4)
- ExtendsKV Cache
- EnablesvLLM
- EnablesContinuous Batching
- Alternative toRadixAttention