Qin et al. at Moonshot AI and Tsinghua University published Mooncake, introducing a KV-cache-centric serving architecture that pools prefix caches across CPU DRAM and SSD storage and uses cache-aware scheduling to boost long-context throughput in production Kimi deployments.
Prompt Caching & KV Prefix Reuse
Also known as: Prompt Caching · Context Caching · Automatic Prefix Caching · Distributed KV Cache
Plain English
Saving the precomputed internal state of repeated prompt openings—such as system instructions, tool schemas, or reference documents—so an LLM server or API can skip re-reading them on subsequent requests.
Technical Overview
Prompt caching persists and indexes prefill KV cache tensors across requests via content-addressed block hashes in PagedAttention or prefix trees in RadixAttention. At the API and cluster tiers, explicit and implicit prefix reuse can reduce time to first token and the cost of cached input tokens, while distributed KV stores (such as Mooncake, LMCache, and NIXL connectors) share cached prefixes across host DRAM, NVMe, and disaggregated serving nodes.
Timeline (6)
2024
Paper- Release
Anthropic launched Prompt Caching on the Claude API, letting developers mark reusable prefix breakpoints on system instructions, tool definitions, and large documents to cut input token costs by up to 90% and time-to-first-token latency by up to 85%.
- Release
OpenAI rolled out automatic Prompt Caching across GPT-4o and
o1API endpoints, routing requests with matching prefixes longer than 1,024 tokens to cache-warm workers and applying an automatic 50% input-token discount without manual cache-control annotations. 2025
ReleaseBuilding on the explicit Context Caching API launched for Gemini 1.5 in May 2024, Google enabled implicit context caching by default on Gemini 2.5 models, automatically passing a 75% token discount whenever incoming requests share a prefix with cached contexts.
- Paper
Liu et al. published LMCache, an open-source distributed KV cache layer that decouples prefix extraction, multi-tier CPU/disk offloading, and peer-to-peer RDMA transfers from single-instance memory, integrating with vLLM to reuse non-prefix and cross-node context chunks.
2026
EcosystemThe vLLM and Mooncake teams integrated Mooncake Store with vLLM's KV connector interface using zero-copy GPUDirect RDMA transfers, achieving 3.8× higher throughput and up to 46× lower median time-to-first-token on multi-turn agentic workloads by pooling evicted prefix blocks across cluster nodes.
See also (14)
Open full graphPrompt Caching & KV Prefix Reuse