Brown et al. at OpenAI published Language Models are Few-Shot Learners, demonstrating that scaling a 175B-parameter autoregressive Transformer (GPT-3) unlocks zero-, one-, and few-shot task adaptation directly from prompt demonstrations without updating model weights.
In-Context Learning
Also known as: ICL · Few-Shot Prompting · Many-Shot Prompting · Chain-of-Thought Prompting
Plain English
The ability of a foundation model to pick up a new task, format, or reasoning style solely from examples and instructions inside the prompt—without changing any of its saved weights.
Technical Overview
In-context learning (ICL) emerges in scaled Transformers when attention and MLP blocks condition next-token predictions on prompt demonstrations rather than gradient updates. Mechanistic work attributes ICL to two-layer Attention induction heads and implicit low-rank or gradient-descent-like weight updates across context tokens, scaling from few-shot and chain-of-thought prompting to thousands of examples across long context windows, while 2025–2026 methods like Nested Learning, Doc-to-LoRA, and test-time training blur the boundary between prompt context and explicit fast-weight or LoRA adaptation.
Timeline (6)
2020
Paper2022
PaperWei et al. published Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, showing that including intermediate step-by-step derivations inside few-shot prompts unlocks multi-step arithmetic, commonsense, and symbolic reasoning at scale—laying the prompting foundation for later test-time reasoning models.
- Paper
Olsson et al. at Anthropic published In-context Learning and Induction Heads, linking the sharp emergence of in-context learning during pre-training to two-layer Attention circuits that match earlier token pairs (
[A][B] ... [A] -> [B]) and generalize to abstract pattern completion. 2024
PaperAgarwal et al. at Google DeepMind published Many-Shot In-Context Learning, showing that expanding prompts from dozens to hundreds or thousands of demonstrations inside long context windows (such as Gemini 1.5 Pro) yields steep log-linear gains that rival or surpass supervised fine-tuning on non-natural and reasoning tasks.
2025
PaperBehrouz et al. at Google Research published Nested Learning: The Illusion of Deep Learning Architectures, unifying neural architectures and gradient optimizers as nested multi-frequency context-compression problems in which in-context learning emerges naturally and motivates self-modifying continual memory modules (
Hope).2026
PaperCharakorn et al. at Sakana AI introduced Doc-to-LoRA, training a hypernetwork to convert unseen long prompts into LoRA adapters in a single forward pass—internalizing context without KV-cache memory overhead and maintaining zero-shot recall at over 4× a model's native context window.
See also (15)
Open full graphIn-Context Learning
10 more connections
- Foundation ModelOutgoing · Enables
- GPTOutgoing · MentionsIncoming · Enables
- RAG & Hybrid RetrievalIncoming · Uses
- AnthropicOutgoing · Mentions
- Deep LearningIncoming · Mentions
- GeminiOutgoing · Mentions
- Google DeepMindOutgoing · Mentions
- KV CacheOutgoing · Mentions
- OpenAIOutgoing · Mentions
- Recurrent Neural NetworksIncoming · Mentions