FIFoundationIndex
IndexMapAbout
ConceptLLMCross-Modal

Context Window & RoPE

Also known as: Context Window · RoPE · Rotary Position Embedding · Long Context

Plain English

The working memory limit of a model—how many tokens of text, code, or video frames it can 'see' at once—together with RoPE, the math trick that lets models know which token came before which even across hundreds of thousands of tokens.

Technical Overview

Rotary Position Embeddings (RoPE) encode relative token positions by rotating Query and Key vectors in 2D subspaces inside self-attention. Extensions like YaRN, NTK-aware scaling, Ring Attention, and sliding-window/hybrid attention enabled Transformer context windows to grow from 4K to 128K–2M+ tokens, though practical retrieval quality still depends on long-context evals and RAG & Hybrid Retrieval.

Chronological Entries (2)

Direct Relations (3)