IndexMapAbout
Format / ProtocolLLMImageCross-Modal

NVFP4

Also known as: NVIDIA FP4 · NVFP4 Micro-Block Scaling

Plain English

NVIDIA's 4-bit floating-point format for Blackwell GPUs that uses smaller blocks and higher-precision scale factors than standard MXFP4 to keep 4-bit training and inference close to 16-bit quality.

Technical Overview

NVFP4 extends FP4 (E2M1) with a two-level scaling hierarchy: each micro-block of 16 elements shares a fractional E4M3 (FP8) scale factor, while the entire tensor carries a second FP32 global scale. Compared to OCP MXFP4's 32-element blocks and power-of-two E8M0 scales, NVFP4's finer 16-element grouping and mantissa-bearing local scales reduce clipping and rounding error in both post-training Quantization and native 4-bit pretraining.

Chronological Entries (3)

#2Paper
AI-Assisted

NVIDIA researchers published Pretraining Large Language Models with NVFP4, demonstrating stable 4-bit pretraining of a 12B-parameter model across 10 trillion tokens. Combining 16×16 2D weight block scaling, Random Hadamard Transforms on gradient GEMMs, stochastic rounding, and higher-precision sensitive layers matched FP8 convergence.

Direct Relations (4)