Researchers from NVIDIA and Baidu described training with FP16 weights, activations, and gradients plus an FP32 master copy of weights. Loss scaling prevented small gradients from underflowing in FP16, which roughly halved memory use for those tensors while matching FP32 accuracy on the tested models.
Mixed-Precision Training
Also known as: AMP · Low-Precision Training
Plain English
A way to train neural networks using smaller, faster number formats for most of the math while keeping a few critical values in higher precision, so training runs faster and uses less memory without losing accuracy.
Technical Overview
Mixed-precision training runs matrix multiplications in a narrow format (FP16, BF16, FP8, or FP4) while keeping master weights, optimizer state, and sensitive reductions in FP32. Techniques include loss scaling for FP16 gradients, per-tensor or block scaling factors for 8-bit and 4-bit formats, and stochastic rounding. Unlike post-hoc quantization for inference, it changes the arithmetic of the training loop itself and depends on hardware support from CUDA kernels and tensor cores.
Timeline (7)
2017
Paper2019
PaperIntel and Facebook researchers published a study of BFLOAT16 training across vision, speech, language, generative, and recommendation models. BF16 keeps the FP32 exponent range with a shorter mantissa, so it trained these models without loss scaling, which is why it became the default pre-training format on many accelerators.
2022
PaperNVIDIA, Arm, and Intel proposed two 8-bit floating-point encodings: E4M3 for weights and activations, and E5M2 for gradients that need wider range. Training language models up to 175B parameters matched 16-bit quality in their tests, opening the path to FP8 tensor-core training.
2024
PaperThe DeepSeek-V3 report validated FP8 mixed precision on a very large mixture-of-experts model, using fine-grained scaling: 1x128 tiles for activations and 128x128 blocks for weights, with higher-precision accumulation. The authors reported relative loss error under 0.25% against BF16 on their runs.
2025
PaperAn NVIDIA-authored paper gave recipes for pre-training with the OCP MXFP8 format, which uses one shared power-of-two scale per 32 values. An 8B-parameter model trained on 15T tokens with the recipe matched its BF16 baseline.
- Paper
NVIDIA reported pre-training a 12B-parameter model on 10T tokens in NVFP4, a 4-bit format with 16-value blocks and FP8 scales. The recipe combined random Hadamard transforms, two-dimensional weight scaling, and stochastic rounding of gradients, and the run's validation loss and downstream accuracy tracked an FP8 baseline.
2026
PaperResearchers published Quartet II, introducing an unbiased quantization scheme (EDEN) for NVFP4 microscaling Tensor Cores that applies across both forward and backward passes to reduce 4-bit mixed-precision pre-training gradient estimation error without falling back to higher precision.
See also (12)
Open full graphMixed-Precision Training