Frantar et al. published Marlin, a fused FP16×INT4 matrix-multiplication kernel tailored for W4A16 LLM decoding. By overlapping asynchronous global-to-shared memory copies with register-level dequantization, Marlin achieved near-ideal ~4× speedups over FP16 at small-to-medium batch sizes on NVIDIA Ampere and Hopper GPUs.
INT4
Also known as: 4-bit Integer · W4A16 · AWQ · GPTQ
Plain English
Compressing model weights into 4-bit whole numbers (only 16 possible values per weight) so large language models use roughly a quarter of the memory of 16-bit checkpoints and fit on consumer GPUs or laptops.
Technical Overview
INT4 Quantization stores weights as 4-bit signed or unsigned integers grouped into blocks (commonly 32–128 elements) with FP16/BF16 scale and zero-point factors. In W4A16 serving—used by GPTQ, AWQ, and GGUF k-quants—weights are dequantized on the fly inside fused GPU kernels such as Marlin, cutting memory-bandwidth bottlenecks during autoregressive decoding without requiring native 4-bit floating-point hardware.
Chronological Entries (3)
Lin et al. introduced AWQ (Activation-aware Weight Quantization), demonstrating that scaling up the ~1% of weight channels tied to large activation magnitudes before INT4 rounding preserves accuracy without mixed-precision hardware overhead. Because AWQ avoids calibration overfitting, it became a standard 4-bit weight format in vLLM and SGLang.
Frantar et al. published GPTQ, a one-shot post-training method using approximate second-order Hessian information to quantize 175B-parameter Transformers to 4-bit and 3-bit integers. It showed that frontier-scale LLMs could run on a single 80 GB GPU with minimal perplexity degradation.
Direct Relations (4)
- Part ofQuantization
- EnablesGGUF
- EnablesInference Engines
- Alternative toFP4