IndexMapAbout
ArchitectureLLMCross-Modal

RadixAttention

Also known as: Radix Tree Prefix Caching · HiCache

Plain English

A caching algorithm that stores previously processed prompt prefixes in a tree structure so an LLM server can instantly skip re-reading shared system instructions, documents, or chat histories across requests.

Technical Overview

Introduced in SGLang, RadixAttention retains KV cache tensors after a request finishes and indexes their token sequences inside an LRU-evicted radix tree (compressed trie). Incoming requests match the longest cached prefix in O(length) time—dramatically cutting time-to-first-token for multi-turn chat, few-shot prompts, RAG, and branching agent workflows—and extends across GPU HBM, host RAM, and distributed storage via HiCache.

Chronological Entries (2)

Direct Relations (3)