FIFoundationIndex
IndexMapAbout
ConceptCross-ModalLLMImageVideo

FlashAttention & GPU Kernels

Also known as: FlashAttention · FlashAttention-3 · Triton · CUDA Kernels

Plain English

Low-level GPU engineering breakthroughs that compute Transformer attention 2–8× faster and with far less memory—without changing the mathematical result—by keeping data inside ultra-fast on-chip SRAM instead of constantly shuttling it to main VRAM.

Technical Overview

Standard self-attention materializes an O(N²) attention matrix in GPU HBM. Tri Dao's FlashAttention tiles Q, K, and V blocks inside on-chip SRAM using online softmax recomputation, achieving IO-aware O(N) memory complexity. Hardware-specific kernels (FlashAttention-3/4, SageAttention, FlexAttention, OpenAI Triton) power both LLM training/Inference Engines (vLLM, SGLang, llama.cpp) and long-sequence Video Generation Models (Sora, Veo, Kling, Wan).

Chronological Entries (2)