FIFoundationIndex
IndexMapAbout
ConceptLLM

Speculative Decoding

Also known as: Draft-and-Verify · Multi-Token Prediction · EAGLE

Plain English

A lossless speedup trick for LLMs: a tiny fast 'draft' model guesses the next several words ahead, and the big smart model verifies all of them in a single parallel check—generating 2–3× faster with mathematically identical output quality.

Technical Overview

Because autoregressive decoding is memory-bandwidth-bound on GPUs, verifying K draft tokens in parallel costs almost the same latency as generating 1 token. Modern variants replace separate draft models with lightweight attached prediction heads such as Medusa, EAGLE, or native Multi-Token Prediction (MTP) heads trained directly into models like DeepSeek, supported natively in Inference Engines (vLLM, SGLang, llama.cpp).

Chronological Entries (2)