Context windows expanded from 8K–32K to 128K–2M tokens across Gemini, Claude, OpenAI, and Llama, enabling whole-codebase and multi-hour video ingestion—while prompting new benchmarks testing whether models suffer 'context rot' vs. targeted RAG & Hybrid Retrieval.
Context Window & RoPE
Also known as: Context Window · RoPE · Rotary Position Embedding · Long Context
Plain English
The working memory limit of a model—how many tokens of text, code, or video frames it can 'see' at once—together with RoPE, the math trick that lets models know which token came before which even across hundreds of thousands of tokens.
Technical Overview
Rotary Position Embeddings (RoPE) encode relative token positions by rotating Query and Key vectors in 2D subspaces inside self-attention. Extensions like YaRN, NTK-aware scaling, Ring Attention, and sliding-window/hybrid attention enabled Transformer context windows to grow from 4K to 128K–2M+ tokens, though practical retrieval quality still depends on long-context evals and RAG & Hybrid Retrieval.
Chronological Entries (2)
Su et al. introduced RoPE (Rotary Position Embedding), which encodes relative distance cleanly inside the attention dot product and became universal across open and frontier LLMs as well as 2D/3D spatial-temporal DiT models.
Direct Relations (3)
- Part ofTransformer
- UsesKV Cache
- Alternative toRAG & Hybrid Retrieval