Unified Multimodal & Vision-Language Models systems extend tokenization beyond text by converting image patches, audio frames, and video clips into discrete or continuous token streams fed into a single Transformer backbone.
ConceptLLMCross-Modal
Tokenization
Also known as: BPE · Byte-Pair Encoding · Tokens
Plain English
How AI models chop text (and increasingly audio or image patches) into numerical chunks called tokens before processing them. A token is roughly three-quarters of an English word, which is also how LLM APIs measure pricing and speed.
Technical Overview
Most LLMs use subword tokenization algorithms like Byte-Pair Encoding (BPE) or SentencePiece with vocabularies between 32k and 256k entries. Vocabulary design directly affects multilingual compression efficiency, coding throughput, arithmetic quirks, and effective context window capacity.
References:tiktoken (OpenAI BPE Tokenizer)
Chronological Entries (2)
Direct Relations (2)
- EnablesTransformer
- Part ofContext Window & RoPE