The SGLang team released HiCache, extending RadixAttention with a hierarchical HiRadixTree that offloads and promotes prefix KV blocks across GPU HBM (L1), host CPU RAM (L2), and distributed cluster storage (L3) for long-context and multi-turn agent workloads.
RadixAttention
Also known as: Radix Tree Prefix Caching · HiCache
Plain English
A caching algorithm that stores previously processed prompt prefixes in a tree structure so an LLM server can instantly skip re-reading shared system instructions, documents, or chat histories across requests.
Technical Overview
Introduced in SGLang, RadixAttention retains KV cache tensors after a request finishes and indexes their token sequences inside an LRU-evicted radix tree (compressed trie). Incoming requests match the longest cached prefix in O(length) time—dramatically cutting time-to-first-token for multi-turn chat, few-shot prompts, RAG, and branching agent workflows—and extends across GPU HBM, host RAM, and distributed storage via HiCache.
Chronological Entries (2)
Zheng et al. introduced RadixAttention in the SGLang paper, organizing finished requests' KV cache blocks in an LRU radix tree rather than discarding them. Cache-aware scheduling grouped requests by shared prefix, speeding up multi-call structured generation and agent trees by up to 5×.
Direct Relations (3)
- ExtendsKV Cache
- EnablesSGLang
- Alternative toPagedAttention