Native Multi-Token Prediction (MTP) modules and EAGLE-2/3 draft heads became standard production features across vLLM and SGLang, especially valuable for long-chain reasoning outputs and code-editing workloads where token acceptance rates exceed 70%.
Speculative Decoding
Also known as: Draft-and-Verify · Multi-Token Prediction · EAGLE
Plain English
A lossless speedup trick for LLMs: a tiny fast 'draft' model guesses the next several words ahead, and the big smart model verifies all of them in a single parallel check—generating 2–3× faster with mathematically identical output quality.
Technical Overview
Because autoregressive decoding is memory-bandwidth-bound on GPUs, verifying K draft tokens in parallel costs almost the same latency as generating 1 token. Modern variants replace separate draft models with lightweight attached prediction heads such as Medusa, EAGLE, or native Multi-Token Prediction (MTP) heads trained directly into models like DeepSeek, supported natively in Inference Engines (vLLM, SGLang, llama.cpp).
Chronological Entries (2)
Leviathan et al. (Google) and Chen et al. (DeepMind) formalized Speculative Decoding with a rejection-sampling scheme that guarantees the exact same output probability distribution as the target LLM while accelerating decoding 2–3×.