FIFoundationIndex
IndexMapAbout
Tool / RuntimeLLMCross-Modal

Inference Engines (vLLM, SGLang, llama.cpp)

Also known as: vLLM · SGLang · TensorRT-LLM · llama.cpp · Ollama

Plain English

Specialized runtime servers that actually execute AI models in production or on your laptop—squeezing 5–20× more speed and concurrent users out of the same GPU compared to naive Python scripts.

Technical Overview

Production LLM serving engines such as vLLM, SGLang, and NVIDIA TensorRT-LLM combine continuous batching, PagedAttention/RadixAttention KV Cache management, FlashAttention & GPU Kernels kernels, tensor/expert parallelism for Mixture of Experts (MoE), FP8/INT4 Quantization, and Speculative Decoding. For local machines, llama.cpp (and wrappers like Ollama) executes GGUF weights across Apple Silicon Metal, CUDA, and CPU.

Chronological Entries (2)

#1Ecosystem
Curated

vLLM and SGLang emerged as the dominant open-source serving engines for cluster deployments, standardizing on OpenAI-compatible HTTP APIs, structured JSON decoding via finite-state machines, and disaggregated prefill/decode architectures.