Han, Mao, and Dally's Deep Compression combined magnitude pruning, trained Quantization, and Huffman coding to shrink convolutional networks by an order of magnitude without accuracy loss, making pruning a mainstream compression technique.
Pruning and Sparsity
Also known as: model pruning · structured sparsity · 2:4 sparsity · weight pruning
Plain English
Making a trained model smaller or faster by removing weights, neurons, layers, or experts that matter least. The pruned model usually needs a short round of retraining to recover quality.
Technical Overview
Unstructured pruning zeroes individual weights, which saves storage but speeds up inference only with sparse kernels. Semi-structured patterns such as NVIDIA's 2:4 (two nonzeros in every four values) map to Sparse Tensor Cores, while structured pruning removes whole heads, channels, layers, or MoE experts for speedups on any hardware. One-shot LLM methods like SparseGPT and Wanda prune without retraining, whereas Minitron-style pipelines prune then recover accuracy with Distillation & Synthetic Data. Pruning composes with Quantization, and accuracy loss grows with sparsity.
Timeline (8)
2015
Paper2018
PaperFrankle and Carbin's Lottery Ticket Hypothesis showed that dense networks contain small sparse subnetworks that, trained from their original initialization, can match the full network's accuracy, reframing pruning as a search for trainable subnetworks.
2021
PaperNVIDIA researchers published "Accelerating Sparse Deep Neural Networks", describing the 2:4 fine-grained structured sparsity pattern supported by Sparse Tensor Cores in Ampere GPUs and a prune-then-fine-tune recipe that preserves accuracy, so a regular sparsity pattern could be turned into hardware speedup.
2023
PaperSparseGPT introduced one-shot pruning for very large language models, pruning OPT-175B and BLOOM-176B to at least 50 percent sparsity in a single pass without retraining, using an approximate layer-wise solver.
- Paper
Wanda scored each weight by its magnitude times the norm of its input activation, pruning LLMs per output without retraining or weight updates. It matched or beat magnitude pruning and was far simpler than second-order methods.
2024
PaperNVIDIA published Minitron, which compresses a pretrained LLM with structured pruning of depth, width, heads, and MLP dimensions followed by knowledge distillation retraining. The paper reports pruned 8B and 4B models from a 15B parent using up to 40 times fewer training tokens than training from scratch.
2025
Paper"REAP the Experts" studied one-shot compression of mixture-of-experts models and argued that pruning whole experts can outperform merging them for generative tasks, extending structured pruning to sparse MoE LLMs.
2026
PaperResearchers introduced GEAR and GEAR² for MoE expert pruning, reconstructing the outputs of pruned experts from remaining active experts plus a lightweight missing-mass residual to reduce the accuracy loss from removing experts without recovery training.