IndexMapAnti-DoomscrollAbout
ArchitectureImageVideoCross-Modal

Neural Tokenizers

Also known as: Visual Tokenizers · Neural Audio Codecs · VQ-VAE · VQGAN · RVQ · FSQ

Plain English

Learned neural compressors that turn images, video clips, and audio waveforms into compact sequences of discrete or continuous tokens so Transformers and diffusion models can read and generate media efficiently.

Technical Overview

Neural tokenizers compress high-dimensional pixel grids, video volumes, and audio waveforms into low-rate token streams using vector quantization (VQ-VAE, VQGAN), Residual Vector Quantization (RVQ in SoundStream and EnCodec), or codebook-free Lookup-Free and Finite Scalar Quantization (LFQ/FSQ in MAGVIT-v2 and Cosmos). Unlike continuous diffusion VAEs alone, discrete and 1D unified tokenizers (such as TiTok, FlexTok, UniTok, and AVTok) bridge perceptual reconstruction with semantic alignment for autoregressive Transformers, Audio & Speech Models, and multimodal LLMs.

Timeline (6)

RSS
  1. 2017

    Paper

    van den Oord, Vinyals, and Kavukcuoglu published Neural Discrete Representation Learning, introducing VQ-VAE to map encoder features onto a learned discrete codebook via nearest-neighbor vector quantization and straight-through gradients—preventing posterior collapse and enabling discrete image, video, and audio tokenization.

  2. 2020

  3. 2022

  4. 2023

  5. 2025

  6. 2026

See also (13)

Open full graph
8 more connections