Kaplan et al. at OpenAI published Scaling Laws for Neural Language Models, demonstrating that Transformer cross-entropy loss follows smooth power-law curves over seven orders of magnitude in compute, parameters, and dataset size—motivating massive parameter scaling in GPT-3.
Scaling Laws
Also known as: Neural Scaling Laws · Chinchilla Scaling · Compute-Optimal Scaling
Plain English
Empirical power-law rules showing how much better a foundation model gets as you add more parameters, training data, and compute—turning model training from guesswork into predictable engineering.
Technical Overview
Neural scaling laws model cross-entropy loss as smooth power laws of parameter count (N), dataset tokens (D), and training FLOPs (C). After Kaplan et al. (2020) favored scaling parameters faster than data, Hoffmann et al. (2022, Chinchilla) showed compute-optimal pre-training scales N and D equally (~20 tokens per parameter), while 2024–2025 research extended scaling laws to test-time compute.
Timeline (4)
2020
- Paper
2022
- Paper
Hoffmann et al. at DeepMind published Training Compute-Optimal Large Language Models (Chinchilla), proving prior models were undertrained and establishing that compute-optimal pre-training scales model parameters and training tokens in equal proportion (~20 tokens per parameter).
- Paper
Cherti et al. extended power-law scaling analysis to contrastive vision-language models using OpenCLIP and LAION datasets, showing that zero-shot image classification and retrieval improve predictably with batch size, model scale, and paired image-text tokens.
2024
- Paper
Snell et al. published Scaling LLM Test-Time Compute Optimally, showing that dynamically allocating inference-time search and verification per prompt can outperform a 14× larger pre-trained model at equal FLOPs—opening a second inference-time scaling axis.
See also (bkz.) (8)
Open full graphScaling Laws