van den Oord, Vinyals, and Kavukcuoglu published Neural Discrete Representation Learning, introducing VQ-VAE to map encoder features onto a learned discrete codebook via nearest-neighbor vector quantization and straight-through gradients—preventing posterior collapse and enabling discrete image, video, and audio tokenization.
Neural Tokenizers
Also known as: Visual Tokenizers · Neural Audio Codecs · VQ-VAE · VQGAN · RVQ · FSQ
Plain English
Learned neural compressors that turn images, video clips, and audio waveforms into compact sequences of discrete or continuous tokens so Transformers and diffusion models can read and generate media efficiently.
Technical Overview
Neural tokenizers compress high-dimensional pixel grids, video volumes, and audio waveforms into low-rate token streams using vector quantization (VQ-VAE, VQGAN), Residual Vector Quantization (RVQ in SoundStream and EnCodec), or codebook-free Lookup-Free and Finite Scalar Quantization (LFQ/FSQ in MAGVIT-v2 and Cosmos). Unlike continuous diffusion VAEs alone, discrete and 1D unified tokenizers (such as TiTok, FlexTok, UniTok, and AVTok) bridge perceptual reconstruction with semantic alignment for autoregressive Transformers, Audio & Speech Models, and multimodal LLMs.
Timeline (6)
2017
Paper2020
PaperEsser, Rombach, and Ommer introduced VQGAN, augmenting discrete autoencoders with perceptual and patch-based adversarial losses so high-resolution images compress into crisp token grids without pixel-loss blur, allowing autoregressive Transformers to synthesize megapixel images.
2022
PaperDéfossez et al. at Meta AI published EnCodec, pairing streaming convolutional encoder-decoders with Residual Vector Quantization (RVQ), multiscale spectrogram discriminators, and Transformer entropy coding—establishing the discrete acoustic codec pattern behind neural Audio & Speech Models and Text-to-Speech (TTS) & Voice Cloning.
2023
PaperYu et al. introduced MAGVIT-v2, replacing learned vector codebooks with Lookup-Free Quantization (LFQ) for joint image and video tokenization—scaling discrete visual vocabularies past 262,000 tokens without codebook collapse and outperforming Diffusion Models on ImageNet generation.
2025
ReleaseNVIDIA released the Cosmos Tokenizer family alongside its Cosmos world foundation model platform, offering causal 3D continuous and Finite Scalar Quantization (FSQ) discrete image and video tokenizers with up to 8× temporal and 16× spatial compression.
2026
PaperPham et al. proposed AVTok, extending 1D visual tokenization into a unified dual-stream Transformer tokenizer that compresses paired audio and video streams into a single compact 1D discrete codebook—enabling joint audio-video, audio-to-video, and video-to-audio generation without separate modality tokenizers.
See also (13)
Open full graphNeural Tokenizers
8 more connections
- Multimodal & Vision-Language ModelsOutgoing · Enables
- TransformerOutgoing · Uses
- Text-to-Speech (TTS) & Voice CloningOutgoing · MentionsIncoming · Uses
- TokenizationIncoming · Alternative to
- Diffusion ModelsOutgoing · Mentions
- Meta AIOutgoing · Mentions
- NVIDIAOutgoing · Mentions
- World ModelsOutgoing · Mentions