FIFoundationIndex
IndexMapAbout
ConceptCross-ModalLLMImage

Quantization

Also known as: FP8 · FP4 · INT4 · AWQ · GPTQ

Plain English

Shrinking a model's weights from high-precision decimal numbers (like 16-bit floats) down to smaller numbers (8-bit or 4-bit) so the model uses far less GPU memory and runs faster, with minimal loss in quality.

Technical Overview

Quantization maps FP16/BF16 weights (and optionally activations and KV Cache) to lower bit-widths such as FP8, INT8, INT4 (AWQ/GPTQ/GGUF k-quants), or NVFP4. Because autoregressive LLM decoding is memory-bandwidth-bound, halving weight bit-width often nearly doubles token generation speed while slashing VRAM requirements by 2–4×.

Chronological Entries (2)

Direct Relations (4)