Serving massive Mixture of Experts (MoE) models like DeepSeek pushed engines toward prefill-decode disaggregation (separating compute-bound prompt ingestion from memory-bound token generation across specialized GPU pools) and wide expert parallelism.
Inference Engines (vLLM, SGLang, llama.cpp)
Also known as: vLLM · SGLang · TensorRT-LLM · llama.cpp · Ollama
Plain English
Specialized runtime servers that actually execute AI models in production or on your laptop—squeezing 5–20× more speed and concurrent users out of the same GPU compared to naive Python scripts.
Technical Overview
Production LLM serving engines such as vLLM, SGLang, and NVIDIA TensorRT-LLM combine continuous batching, PagedAttention/RadixAttention KV Cache management, FlashAttention & GPU Kernels kernels, tensor/expert parallelism for Mixture of Experts (MoE), FP8/INT4 Quantization, and Speculative Decoding. For local machines, llama.cpp (and wrappers like Ollama) executes GGUF weights across Apple Silicon Metal, CUDA, and CPU.
Chronological Entries (2)
vLLM and SGLang emerged as the dominant open-source serving engines for cluster deployments, standardizing on OpenAI-compatible HTTP APIs, structured JSON decoding via finite-state machines, and disaggregated prefill/decode architectures.
Direct Relations (5)
- UsesKV Cache
- UsesFlashAttention & GPU Kernels
- UsesQuantization
- UsesGGUF
- EnablesServerless GPU & Inference Clouds