IndexMapAbout
Tool / RuntimeLLMCross-Modal

vLLM

Also known as: vLLM Server

Plain English

vLLM serves language and multimodal models for many requests at once. Teams can use it to run a model behind an API on one accelerator or across several machines.

Technical Overview

vLLM combines continuous batching with paged KV cache management. Large deployments can split model computation across devices and use separate prefill and decode workers. The server supports OpenAI-compatible APIs for supported models and tasks.

Chronological Entries (3)

#3Release
AI-Assisted

vLLM 0.34.0 enabled the extensible KV cache by default on CUDA and ROCm. The engine measures free GPU memory after warmup to size the cache with the memory available to the worker.

#2Ecosystem
AI-Assisted

The Kinference team introduced a Kubernetes control plane that uses vLLM workers for generation. Its router considers which worker has the needed prompt prefix cached, giving it a way to reuse work across requests.

Direct Relations (2)