FIFoundationIndex
IndexMapAbout
ConceptLLMCross-Modal

Tokenization

Also known as: BPE · Byte-Pair Encoding · Tokens

Plain English

How AI models chop text (and increasingly audio or image patches) into numerical chunks called tokens before processing them. A token is roughly three-quarters of an English word, which is also how LLM APIs measure pricing and speed.

Technical Overview

Most LLMs use subword tokenization algorithms like Byte-Pair Encoding (BPE) or SentencePiece with vocabularies between 32k and 256k entries. Vocabulary design directly affects multilingual compression efficiency, coding throughput, arithmetic quirks, and effective context window capacity.

Chronological Entries (2)

#1Milestone
Curated

Frontier and open-weight models expanded vocabulary sizes sharply—from 32k in Llama 2 to 128k in Llama 3 and 200k+ in OpenAI's o200k_base—cutting token counts by 15–30% for non-English languages and source code.

Direct Relations (2)