vLLM 0.34.0 enabled the extensible KV cache by default on CUDA and ROCm. The engine measures free GPU memory after warmup to size the cache with the memory available to the worker.
vLLM
Also known as: vLLM Server
Plain English
vLLM serves language and multimodal models for many requests at once. Teams can use it to run a model behind an API on one accelerator or across several machines.
Technical Overview
vLLM combines continuous batching with paged KV cache management. Large deployments can split model computation across devices and use separate prefill and decode workers. The server supports OpenAI-compatible APIs for supported models and tasks.
Chronological Entries (3)
The Kinference team introduced a Kubernetes control plane that uses vLLM workers for generation. Its router considers which worker has the needed prompt prefix cached, giving it a way to reuse work across requests.
The vLLM paper introduced PagedAttention, which allocates attention state in blocks as sequences grow. This allows a server to fit more active requests in GPU memory and reuse cache blocks when generation branches share the same prefix.
Direct Relations (2)
- Part ofInference Engines
- UsesKV Cache