IndexMapAbout
ConceptLLMCross-Modal

Synthetic Data & Self-Play

Also known as: Synthetic Data · Self-Instruct · Self-Play

Plain English

Using AI models to generate, filter, and verify new training examples—textbooks, code solutions, or reasoning traces—when high-quality human data runs short or needs targeted coverage.

Technical Overview

Synthetic data pipelines generate instruction pairs (Self-Instruct), textbook-grade pre-training corpora (Phi, Cosmopedia), or verified reasoning/code rollouts (Nemotron-4, RLVR) filtered by unit tests, reward models, or deduplication. Paired with distillation and post-training, synthetic generation lets labs scale capability on scarce domains without scraping raw web noise.

Timeline (4)

RSS

2022

2023

  1. Paper

    Gunasekar et al. at Microsoft Research published Textbooks Are All You Need (Phi-1), demonstrating that a 1.3B model trained on curated synthetic textbooks and exercises matched much larger models on Python coding benchmarks.

2024

See also (bkz.) (7)

Open full graph
2 more connections