IndexMapAbout
Format / ProtocolLLMCross-Modal

INT4

Also known as: 4-bit Integer · W4A16 · AWQ · GPTQ

Plain English

Compressing model weights into 4-bit whole numbers (only 16 possible values per weight) so large language models use roughly a quarter of the memory of 16-bit checkpoints and fit on consumer GPUs or laptops.

Technical Overview

INT4 Quantization stores weights as 4-bit signed or unsigned integers grouped into blocks (commonly 32–128 elements) with FP16/BF16 scale and zero-point factors. In W4A16 serving—used by GPTQ, AWQ, and GGUF k-quants—weights are dequantized on the fly inside fused GPU kernels such as Marlin, cutting memory-bandwidth bottlenecks during autoregressive decoding without requiring native 4-bit floating-point hardware.

Chronological Entries (3)

Direct Relations (4)