Native FP8 training and serving became mainstream across both LLMs (e.g., DeepSeek's fine-grained FP8 mixed-precision training) and image models (FP8 FLUX Checkpoints), taking direct advantage of Hopper/Blackwell Tensor Cores with negligible quality drop.
Quantization
Also known as: FP8 · FP4 · INT4 · AWQ · GPTQ
Plain English
Shrinking a model's weights from high-precision decimal numbers (like 16-bit floats) down to smaller numbers (8-bit or 4-bit) so the model uses far less GPU memory and runs faster, with minimal loss in quality.
Technical Overview
Quantization maps FP16/BF16 weights (and optionally activations and KV Cache) to lower bit-widths such as FP8, INT8, INT4 (AWQ/GPTQ/GGUF k-quants), or NVFP4. Because autoregressive LLM decoding is memory-bandwidth-bound, halving weight bit-width often nearly doubles token generation speed while slashing VRAM requirements by 2–4×.
Chronological Entries (2)
Post-training quantization methods like GPTQ and AWQ showed that protecting a tiny fraction (~1%) of salient weight channels lets 4-bit LLMs preserve nearly full 16-bit accuracy, making 70B models runnable on multi-consumer-GPU rigs or single H100s via Inference Engines (vLLM, SGLang, llama.cpp).
Direct Relations (4)
- EnablesGGUF
- EnablesInference Engines (vLLM, SGLang, llama.cpp)
- UsesKV Cache
- EnablesLoRA & QLoRA