Researchers at the University of Washington, OctoAI, and CMU launched FlashInfer alongside Cascade Inference, decoupling shared-prefix attention from unique per-request suffixes. Keeping shared KV cache tiles in GPU shared memory (SMEM) while merging partial outputs via log-sum-exp states cut redundant HBM reads and accelerated shared-prefix batch decoding by up to 31× on H100 GPUs.
FlashInfer
Also known as: FlashInfer Kernel Library · FlashInfer-Bench
Plain English
An open-source GPU kernel library for LLM serving that computes attention, sampling, and Mixture-of-Experts layers over paged and prefix-shared memory without wasting bandwidth.
Technical Overview
FlashInfer represents heterogeneous KV cache structures—including paged tables, radix trees, and ragged buffers—as compressed block-sparse matrices and compiles specialized CUDA and CuTe-DSL kernels just in time. Integrated into vLLM, SGLang, and TensorRT-LLM, it provides load-balanced prefill, decode, cascade-attention, sorting-free sampling, and fused MoE kernels across NVIDIA Hopper, Blackwell, and Rubin GPUs.
Timeline (6)
2024
Release- Release
The FlashInfer team released FlashInfer v0.2, introducing a PyTorch-based JIT compilation engine for custom attention variants (such as logit soft-capping and RoPE fusion), deterministic
plan/runscheduling for CUDA Graph capture, and native DeepSeek Multi-head Latent Attention (MLA) support in vLLM and SGLang. 2025
PaperYe et al. published the FlashInfer paper (accepted to MLSys 2025), unifying ragged, paged, and radix-tree KV caches under a block-sparse matrix representation with load-balanced thread-block scheduling. Compared to compiler-generated backends, FlashInfer reduced inter-token latency by 29–69% on standard serving benchmarks and 13–17% on long-context generation.
- Milestone
FlashInfer introduced sorting-free GPU sampling kernels for Top-K, Top-P, and Min-P token sampling, replacing full vocabulary radix sorts with dual-pivot rejection sampling and segmented prefix scans so CPU-free sampling could be captured directly inside CUDA Graphs.
2026
PaperXing et al. published FlashInfer-Bench, defining a standardized trace schema (
FlashInfer Trace), real-world serving workloads, and a dynamicapply()substitution mechanism that lets autonomous coding agents generate, benchmark, and hot-swap low-level GPU kernels directly inside production vLLM and SGLang engines.- Release
FlashInfer v0.7 expanded the library beyond attention and GEMMs with MegaMoE fused expert-parallel MoE kernels that overlap NVLink communication with low-precision FP8/NVFP4 computation, alongside Autotuner v2 for serving-aware runtime kernel selection and open CUTLASS/CuTe-DSL kernel paths.
See also (17)
Open full graphFlashInfer
12 more connections
- TensorRT-LLMOutgoing · Enables
- vLLMOutgoing · EnablesIncoming · Mentions
- Inference EnginesIncoming · Uses
- AttentionOutgoing · Mentions
- Context Window & RoPEOutgoing · Mentions
- DeepSeekOutgoing · Mentions
- FP8Outgoing · Mentions
- Mixture of Experts (MoE)Outgoing · Mentions
- NVFP4Outgoing · Mentions
- NVIDIAOutgoing · Mentions
- RadixAttentionOutgoing · Mentions
- Sampling & SchedulersOutgoing · Mentions