IndexMapAnti-DoomscrollAbout
ConceptLLMCross-Modal

Prompt Caching & KV Prefix Reuse

Also known as: Prompt Caching · Context Caching · Automatic Prefix Caching · Distributed KV Cache

Plain English

Saving the precomputed internal state of repeated prompt openings—such as system instructions, tool schemas, or reference documents—so an LLM server or API can skip re-reading them on subsequent requests.

Technical Overview

Prompt caching persists and indexes prefill KV cache tensors across requests via content-addressed block hashes in PagedAttention or prefix trees in RadixAttention. At the API and cluster tiers, explicit and implicit prefix reuse can reduce time to first token and the cost of cached input tokens, while distributed KV stores (such as Mooncake, LMCache, and NIXL connectors) share cached prefixes across host DRAM, NVMe, and disaggregated serving nodes.

Timeline (6)

RSS
  1. 2024

  2. Release

    Anthropic launched Prompt Caching on the Claude API, letting developers mark reusable prefix breakpoints on system instructions, tool definitions, and large documents to cut input token costs by up to 90% and time-to-first-token latency by up to 85%.

  3. Release

    OpenAI rolled out automatic Prompt Caching across GPT-4o and o1 API endpoints, routing requests with matching prefixes longer than 1,024 tokens to cache-warm workers and applying an automatic 50% input-token discount without manual cache-control annotations.

  4. 2025

  5. 2026

See also (14)

Open full graph
9 more connections