FIFoundationIndex
IndexMapAbout
ConceptLLM

KV Cache

Also known as: Key-Value Cache · PagedAttention · Prefix Caching

Plain English

A short-term memory buffer stored in GPU VRAM while an LLM generates text. Without it, the model would have to re-read the entire conversation from scratch for every single new word it outputs.

Technical Overview

During autoregressive decoding in a causal Transformer, the Key (K) and Value (V) projection tensors of prior tokens remain invariant. Caching these tensors reduces per-token generation complexity from quadratic to linear in sequence length, but shifts the inference bottleneck from compute to HBM memory bandwidth and capacity—making Inference Engines (vLLM, SGLang, llama.cpp) and Quantization essential.

Chronological Entries (3)