DeepSeek demonstrated that distilling ~800k verified reasoning traces from a 671B teacher into small 7B–32B dense models outperformed running RL directly on those small models—cementing synthetic data Distillation & Synthetic Data as the fastest path to high-intelligence small models.
Distillation & Synthetic Data
Also known as: Knowledge Distillation · Synthetic Data · Step Distillation
Plain English
Using a huge, expensive 'teacher' model to generate high-quality training examples or supervision signals so a much smaller, faster, cheaper 'student' model can punch far above its weight.
Technical Overview
Knowledge distillation transfers behavior from a frontier teacher to a smaller student via logit matching or curated synthetic trajectories (reasoning traces, code edits, or captioned images). In Diffusion Models, step distillation collapses 30–50 denoising steps into 1–4 steps (e.g., FLUX.1-schnell), while in LLMs synthetic reasoning traces produced by RLVR teachers dramatically lift 7B–32B models.
Chronological Entries (2)
Black Forest Labs released FLUX.1 [schnell] under Apache 2.0 using latent adversarial diffusion distillation, cutting FLUX generation from 28–50 steps down to 1–4 steps with no CFG overhead.