IndexMapAnti-DoomscrollAbout
Tool / RuntimeLLMCross-Modal

FlashInfer

Also known as: FlashInfer Kernel Library · FlashInfer-Bench

Plain English

An open-source GPU kernel library for LLM serving that computes attention, sampling, and Mixture-of-Experts layers over paged and prefix-shared memory without wasting bandwidth.

Technical Overview

FlashInfer represents heterogeneous KV cache structures—including paged tables, radix trees, and ragged buffers—as compressed block-sparse matrices and compiles specialized CUDA and CuTe-DSL kernels just in time. Integrated into vLLM, SGLang, and TensorRT-LLM, it provides load-balanced prefill, decode, cascade-attention, sorting-free sampling, and fused MoE kernels across NVIDIA Hopper, Blackwell, and Rubin GPUs.

Timeline (6)

RSS
  1. 2024

    Release

    Researchers at the University of Washington, OctoAI, and CMU launched FlashInfer alongside Cascade Inference, decoupling shared-prefix attention from unique per-request suffixes. Keeping shared KV cache tiles in GPU shared memory (SMEM) while merging partial outputs via log-sum-exp states cut redundant HBM reads and accelerated shared-prefix batch decoding by up to 31× on H100 GPUs.

  2. 2025

  3. 2026

See also (17)

Open full graph
12 more connections