FIFoundationIndex
IndexMapAbout
ConceptCross-ModalLLMImage

Distillation & Synthetic Data

Also known as: Knowledge Distillation · Synthetic Data · Step Distillation

Plain English

Using a huge, expensive 'teacher' model to generate high-quality training examples or supervision signals so a much smaller, faster, cheaper 'student' model can punch far above its weight.

Technical Overview

Knowledge distillation transfers behavior from a frontier teacher to a smaller student via logit matching or curated synthetic trajectories (reasoning traces, code edits, or captioned images). In Diffusion Models, step distillation collapses 30–50 denoising steps into 1–4 steps (e.g., FLUX.1-schnell), while in LLMs synthetic reasoning traces produced by RLVR teachers dramatically lift 7B–32B models.

Chronological Entries (2)