NVIDIA published an NVFP4 mixed-precision training recipe for JAX and MaxText on Blackwell systems. Applying 4-bit arithmetic to selected matrix multiplications while keeping sensitive operations in higher precision extended NVFP4 beyond inference engines into open-source Distributed Training frameworks.
NVFP4
Also known as: NVIDIA FP4 · NVFP4 Micro-Block Scaling
Plain English
NVIDIA's 4-bit floating-point format for Blackwell GPUs that uses smaller blocks and higher-precision scale factors than standard MXFP4 to keep 4-bit training and inference close to 16-bit quality.
Technical Overview
NVFP4 extends FP4 (E2M1) with a two-level scaling hierarchy: each micro-block of 16 elements shares a fractional E4M3 (FP8) scale factor, while the entire tensor carries a second FP32 global scale. Compared to OCP MXFP4's 32-element blocks and power-of-two E8M0 scales, NVFP4's finer 16-element grouping and mantissa-bearing local scales reduce clipping and rounding error in both post-training Quantization and native 4-bit pretraining.
Chronological Entries (3)
NVIDIA researchers published Pretraining Large Language Models with NVFP4, demonstrating stable 4-bit pretraining of a 12B-parameter model across 10 trillion tokens. Combining 16×16 2D weight block scaling, Random Hadamard Transforms on gradient GEMMs, stochastic rounding, and higher-precision sensitive layers matched FP8 convergence.
NVIDIA detailed the NVFP4 tensor layout for Blackwell GPUs: 16-element E2M1 micro-blocks each paired with an E4M3 scale factor, anchored by a per-tensor FP32 scale. Halving the block size relative to MXFP4 and using fractional FP8 scales lets tensors retain outlier fidelity with roughly 4.5 bits per parameter on average.
Direct Relations (4)
- ExtendsFP4
- Part ofQuantization
- Created byNVIDIA
- EnablesDistributed Training