FIFoundationIndex
IndexMapAbout
ConceptLLMImageVideoCross-Modal

Distributed Training

Also known as: Distributed model training · Training parallelism

Plain English

Several devices work together to train one model. They can process different examples at the same time or divide the model itself when it is too large to fit on one device.

Technical Overview

In data parallelism, workers process different batches and combine their gradients. Tensor parallelism splits operations within a layer, while pipeline parallelism places successive groups of layers on different devices. The choice affects how much data workers exchange and how long each device waits for the next piece of work.

Chronological Entries (2)

Direct Relations (0)

No explicit outgoing relations yet.