Narayanan and colleagues combined tensor, pipeline, and data parallelism to benchmark training iterations for a one-trillion-parameter Transformer on 3,072 GPUs. Their pipeline divided each batch into smaller pieces so different groups of layers could work on different pieces concurrently.
Distributed Training
Also known as: Distributed model training · Training parallelism
Plain English
Several devices work together to train one model. They can process different examples at the same time or divide the model itself when it is too large to fit on one device.
Technical Overview
In data parallelism, workers process different batches and combine their gradients. Tensor parallelism splits operations within a layer, while pipeline parallelism places successive groups of layers on different devices. The choice affects how much data workers exchange and how long each device waits for the next piece of work.
Chronological Entries (2)
The Megatron-LM authors split matrix multiplications inside Transformer layers across GPUs. Combined with data parallelism, this let the team train an 8.3-billion-parameter language model on 512 GPUs while each device held only part of the model.
Direct Relations (0)
No explicit outgoing relations yet.