IndexMapAbout
ConceptLLMCross-Modal

Scaling Laws

Also known as: Neural Scaling Laws · Chinchilla Scaling · Compute-Optimal Scaling

Plain English

Empirical power-law rules showing how much better a foundation model gets as you add more parameters, training data, and compute—turning model training from guesswork into predictable engineering.

Technical Overview

Neural scaling laws model cross-entropy loss as smooth power laws of parameter count (N), dataset tokens (D), and training FLOPs (C). After Kaplan et al. (2020) favored scaling parameters faster than data, Hoffmann et al. (2022, Chinchilla) showed compute-optimal pre-training scales N and D equally (~20 tokens per parameter), while 2024–2025 research extended scaling laws to test-time compute.

Timeline (4)

RSS

2020

2022

  1. Paper

    Hoffmann et al. at DeepMind published Training Compute-Optimal Large Language Models (Chinchilla), proving prior models were undertrained and establishing that compute-optimal pre-training scales model parameters and training tokens in equal proportion (~20 tokens per parameter).

2024

See also (bkz.) (8)

Open full graph
3 more connections