IndexMapAnti-DoomscrollAbout
ConceptLLMImageVideoCross-Modal

Pre-Training

Also known as: Pretraining · Foundation Model Pre-Training

Plain English

The first and most expensive stage of building a foundation model: training a network from scratch on a huge, mostly unlabeled corpus so it learns general patterns before any fine-tuning.

Technical Overview

Pre-training optimizes a self-supervised objective (next-token prediction for LLMs, denoising or flow objectives for image and video models) over trillions of tokens or billions of samples, using distributed training across thousands of accelerators. Results depend on the compute-to-data balance described by scaling laws, on corpus filtering and mixing, on the optimizer, and on late-stage data schedules such as annealing. The resulting base checkpoint is then adapted by post-training.

Timeline (7)

RSS
  1. 2020

  2. 2022

    Paper

    DeepMind trained over 400 models and concluded that parameters and training tokens should grow in roughly equal proportion. Its 70B-parameter Chinchilla, trained on 1.4T tokens, outperformed the larger 280B Gopher at the same compute budget, shifting later pre-training runs toward more data per parameter.

  3. 2024

  4. Paper

    Hugging Face released FineWeb, a 15-trillion-token web corpus filtered from Common Crawl, one week after the DataComp-LM (DCLM) benchmark of 17 June 2024 offered a standardized testbed for comparing data-curation methods at fixed training recipes. Both showed that model-based quality filtering can change downstream accuracy as much as scale does.

  5. Paper

    The Llama 3 report from Meta described a staged pre-training recipe: a long main run, context-length extension, and a final annealing phase on small amounts of high-quality code and math data while the learning rate decays to zero. This late data-schedule step, now often called mid-training, targets specific skills before post-training begins.

  6. 2025

See also (21)

Open full graph
16 more connections