DeepSeek released the DeepSeek-V3 technical report detailing fine-grained FP8 mixed-precision training on a 671B-parameter Mixture of Experts (MoE) model. Scaling activations in 1×128 tiles and weights in 128×128 blocks kept training loss close to BF16 while cutting compute and memory costs across Distributed Training clusters.
FP8
Also known as: E4M3 · E5M2 · OFP8 · 8-bit Floating Point
Plain English
An 8-bit floating-point number format that cuts model memory in half compared to 16-bit formats while keeping enough range and precision to train and run modern LLMs and image generators directly on datacenter GPUs.
Technical Overview
Standardized under the OCP 8-bit Floating Point (OFP8) specification into two variants—E4M3 (4 exponent bits, 3 mantissa bits, higher precision for weights and activations) and E5M2 (5 exponent bits, 2 mantissa bits, wider dynamic range for gradients). Paired with per-tensor or fine-grained block scaling, FP8 executes natively on NVIDIA Hopper/Blackwell and AMD CDNA3+ matrix cores for both mixed-precision training and high-throughput serving.
Chronological Entries (3)
The Open Compute Project published OFP8 Revision 1.0, standardizing binary interchange layouts for E4M3 and E5M2 across hardware vendors including AMD, Arm, Intel, Meta, and NVIDIA. A shared interchange spec prevented incompatible vendor-specific 8-bit encodings across compilers and Inference Engines.
Researchers from NVIDIA, Arm, and Intel published FP8 Formats for Deep Learning, proposing the dual E4M3 and E5M2 8-bit floating-point encodings. Splitting precision between E4M3 for forward weights and activations and E5M2 for wider-range gradients let Transformer models train in 8-bit without architectural changes.
Direct Relations (5)
- Part ofQuantization
- UsesNVIDIA
- EnablesInference Engines
- EnablesDistributed Training
- Alternative toFP4