IndexMapAbout
Format / ProtocolCross-ModalLLMImage

FP8

Also known as: E4M3 · E5M2 · OFP8 · 8-bit Floating Point

Plain English

An 8-bit floating-point number format that cuts model memory in half compared to 16-bit formats while keeping enough range and precision to train and run modern LLMs and image generators directly on datacenter GPUs.

Technical Overview

Standardized under the OCP 8-bit Floating Point (OFP8) specification into two variants—E4M3 (4 exponent bits, 3 mantissa bits, higher precision for weights and activations) and E5M2 (5 exponent bits, 2 mantissa bits, wider dynamic range for gradients). Paired with per-tensor or fine-grained block scaling, FP8 executes natively on NVIDIA Hopper/Blackwell and AMD CDNA3+ matrix cores for both mixed-precision training and high-throughput serving.

Chronological Entries (3)

#1Paper
AI-Assisted

Researchers from NVIDIA, Arm, and Intel published FP8 Formats for Deep Learning, proposing the dual E4M3 and E5M2 8-bit floating-point encodings. Splitting precision between E4M3 for forward weights and activations and E5M2 for wider-range gradients let Transformer models train in 8-bit without architectural changes.

Direct Relations (5)