IndexMapAnti-DoomscrollAbout
See also
ConceptLLMCross-Modal

In-Context Learning

Also known as: ICL · Few-Shot Prompting · Many-Shot Prompting · Chain-of-Thought Prompting

Plain English

The ability of a foundation model to pick up a new task, format, or reasoning style solely from examples and instructions inside the prompt—without changing any of its saved weights.

Technical Overview

In-context learning (ICL) emerges in scaled Transformers when attention and MLP blocks condition next-token predictions on prompt demonstrations rather than gradient updates. Mechanistic work attributes ICL to two-layer Attention induction heads and implicit low-rank or gradient-descent-like weight updates across context tokens, scaling from few-shot and chain-of-thought prompting to thousands of examples across long context windows, while 2025–2026 methods like Nested Learning, Doc-to-LoRA, and test-time training blur the boundary between prompt context and explicit fast-weight or LoRA adaptation.

Timeline (6)

RSS
  1. 2020

  2. 2022

  3. Paper

    Olsson et al. at Anthropic published In-context Learning and Induction Heads, linking the sharp emergence of in-context learning during pre-training to two-layer Attention circuits that match earlier token pairs ([A][B] ... [A] -> [B]) and generalize to abstract pattern completion.

  4. 2024

  5. 2025

    Paper

    Behrouz et al. at Google Research published Nested Learning: The Illusion of Deep Learning Architectures, unifying neural architectures and gradient optimizers as nested multi-frequency context-compression problems in which in-context learning emerges naturally and motivates self-modifying continual memory modules (Hope).

  6. 2026

See also (15)

Open full graph
10 more connections