Huang et al. at Google introduced GPipe, partitioning sequential neural network layers across accelerators and splitting each mini-batch into smaller micro-batches to shrink idle pipeline bubbles during synchronous Pipeline Parallelism (PP), enabling 6B-parameter Transformer training across multiple devices.
Parallelism Strategies (TP, PP, EP & CP)
Also known as: Tensor Parallelism · Pipeline Parallelism · Expert Parallelism · Context Parallelism · Wide-EP · 3D Parallelism · 4D Parallelism
Plain English
Strategies for splitting a large AI model and its context across multiple GPUs so models that exceed a single chip's memory or bandwidth can train and serve requests at low latency.
Technical Overview
Multi-accelerator execution decomposes work along four primary axes beyond data parallelism: Tensor Parallelism (TP) slices individual layer matrices within high-bandwidth NVLink domains; Pipeline Parallelism (PP) assigns consecutive layer stages to different devices and overlaps micro-batches; Context Parallelism (CP) shards long sequence activations and KV caches across ranks; and Expert Parallelism (EP) distributes sparse MoE feed-forward experts across cluster nodes using all-to-all dispatch and combine collectives.
Timeline (6)
2018
Paper2019
PaperShoeybi et al. at NVIDIA introduced Megatron-LM, formalizing intra-layer Tensor Parallelism (TP) by splitting MLP weight matrices column-wise and row-wise and partitioning attention heads across GPUs so each layer requires only two
all-reducesynchronizations per forward-backward pass.2023
PaperLiu et al. formulated Ring Attention for Context Parallelism (CP), sharding long token sequences across a ring of accelerators and circulating key-value blocks concurrently with blockwise attention compute so sequence length scales linearly with device count.
2025
ReleaseDeepSeek open-sourced DualPipe and DeepEP, pairing bidirectional pipeline scheduling with RDMA- and NVLink-aware all-to-all Expert Parallelism (EP) dispatch and combine kernels that hide cross-node communication behind computation in large MoE training and disaggregated inference.
- Milestone
The vLLM team detailed Wide Expert Parallelism (Wide-EP) with data-parallel attention (
TP=1), Dual-Batch Overlap (DBO), and an Expert Parallel Load Balancer (EPLB) to serve DeepSeek MoE models at 2,200 tokens/s per H200 GPU without duplicating the KV cache across tensor-parallel ranks. 2026
MilestoneFollowing its May 2026 release of Elastic Expert Parallelism for zero-restart MoE worker scaling, vLLM introduced Decode Context Parallelism (DCP), partitioning the KV cache along the sequence dimension across GPUs during autoregressive decoding and merging partial attention outputs via log-sum-exp reduction to expand long-context serving capacity.
See also (12)
Open full graphParallelism Strategies (TP, PP, EP & CP)