IndexMapAnti-DoomscrollAbout
See also
ConceptLLMImageVideoCross-Modal

Parallelism Strategies (TP, PP, EP & CP)

Also known as: Tensor Parallelism · Pipeline Parallelism · Expert Parallelism · Context Parallelism · Wide-EP · 3D Parallelism · 4D Parallelism

Plain English

Strategies for splitting a large AI model and its context across multiple GPUs so models that exceed a single chip's memory or bandwidth can train and serve requests at low latency.

Technical Overview

Multi-accelerator execution decomposes work along four primary axes beyond data parallelism: Tensor Parallelism (TP) slices individual layer matrices within high-bandwidth NVLink domains; Pipeline Parallelism (PP) assigns consecutive layer stages to different devices and overlaps micro-batches; Context Parallelism (CP) shards long sequence activations and KV caches across ranks; and Expert Parallelism (EP) distributes sparse MoE feed-forward experts across cluster nodes using all-to-all dispatch and combine collectives.

Timeline (6)

RSS
  1. 2018

  2. 2019

  3. 2023

  4. 2025

  5. 2026

See also (12)

Open full graph
7 more connections