EleutherAI released The Pile, an 825 GiB English corpus assembled from 22 sources, including academic papers, code, and web text. Open, documented corpora like this let outside groups train and study base language models without a private data pipeline.
Pre-Training
Also known as: Pretraining · Foundation Model Pre-Training
Plain English
The first and most expensive stage of building a foundation model: training a network from scratch on a huge, mostly unlabeled corpus so it learns general patterns before any fine-tuning.
Technical Overview
Pre-training optimizes a self-supervised objective (next-token prediction for LLMs, denoising or flow objectives for image and video models) over trillions of tokens or billions of samples, using distributed training across thousands of accelerators. Results depend on the compute-to-data balance described by scaling laws, on corpus filtering and mixing, on the optimizer, and on late-stage data schedules such as annealing. The resulting base checkpoint is then adapted by post-training.
Timeline (7)
2020
Paper2022
PaperDeepMind trained over 400 models and concluded that parameters and training tokens should grow in roughly equal proportion. Its 70B-parameter Chinchilla, trained on 1.4T tokens, outperformed the larger 280B Gopher at the same compute budget, shifting later pre-training runs toward more data per parameter.
2024
PaperThe Allen Institute for AI published Dolma, a three-trillion-token open corpus, together with its curation toolkit. Publishing the filtering and deduplication steps made data preparation, not only model weights, something others could inspect and reproduce.
- Paper
Hugging Face released FineWeb, a 15-trillion-token web corpus filtered from Common Crawl, one week after the DataComp-LM (DCLM) benchmark of 17 June 2024 offered a standardized testbed for comparing data-curation methods at fixed training recipes. Both showed that model-based quality filtering can change downstream accuracy as much as scale does.
- Paper
The Llama 3 report from Meta described a staged pre-training recipe: a long main run, context-length extension, and a final annealing phase on small amounts of high-quality code and math data while the learning rate decays to zero. This late data-schedule step, now often called mid-training, targets specific skills before post-training begins.
2025
PaperMoonshot AI showed that the Muon optimizer, which orthogonalizes matrix updates, can be scaled to large mixture-of-experts models by adding weight decay and matching update magnitudes to AdamW. Its Moonlight models reached comparable quality with roughly half the training FLOPs of AdamW in the authors' scaling experiments, and Kimi K2 later used a clipped variant for a 15.5T-token run.
- Release
Ai2 released Olmo 3 (7B and 32B) with its complete Dolma 3 (~6T tokens) pre-training, mid-training, and long-context extension checkpoint sequence, letting researchers trace how each data-mix transition reshapes downstream reasoning and code performance.
See also (21)
Open full graphPre-Training
16 more connections
- Foundation ModelOutgoing · Part of
- GPTIncoming · Uses
- Olmo (Open Language Model)Outgoing · MentionsIncoming · Uses
- Amazon & AWS AIIncoming · Mentions
- AttentionIncoming · Mentions
- FP8Incoming · Mentions
- Google DeepMindOutgoing · Mentions
- Hugging FaceOutgoing · Mentions
- Meta AIOutgoing · Mentions
- Mixed-Precision TrainingIncoming · Mentions
- Mixture of Experts (MoE)Outgoing · Mentions
- Moonshot AI & KimiOutgoing · MentionsIncoming · Mentions
- QwenIncoming · Mentions
- Reasoning & Test-Time ComputeIncoming · Mentions
- Supervised Fine-Tuning (SFT)Incoming · Mentions
- TransformerIncoming · Mentions