Prefix caching (e.g., RadixAttention in SGLang and prompt caching APIs from Anthropic and OpenAI) reuses pre-computed KV blocks across requests sharing long system prompts or RAG context, cutting time-to-first-token and API cost by up to 75–90%.
KV Cache
Also known as: Key-Value Cache · PagedAttention · Prefix Caching
Plain English
A short-term memory buffer stored in GPU VRAM while an LLM generates text. Without it, the model would have to re-read the entire conversation from scratch for every single new word it outputs.
Technical Overview
During autoregressive decoding in a causal Transformer, the Key (K) and Value (V) projection tensors of prior tokens remain invariant. Caching these tensors reduces per-token generation complexity from quadratic to linear in sequence length, but shifts the inference bottleneck from compute to HBM memory bandwidth and capacity—making Inference Engines (vLLM, SGLang, llama.cpp) and Quantization essential.
Chronological Entries (3)
DeepSeek introduced Multi-head Latent Attention (MLA) in DeepSeek-V2, compressing Keys and Values into a low-rank latent vector before caching. This shrank the KV Cache footprint by over 90%, making long context windows drastically cheaper to serve.
Kwon et al. introduced PagedAttention in vLLM, borrowing virtual memory paging from operating systems to store non-contiguous KV Cache blocks. This eliminated severe GPU VRAM fragmentation and boosted concurrent LLM serving throughput by 2–4x.