FlashAttention-3 exploited Hopper GPU asynchronous Tensor Memory Accelerator (TMA) instructions and native FP8 Quantization, pushing GPU utilization past 75% and accelerating both LLM serving and spatiotemporal Video Generation Models (Sora, Veo, Kling, Wan) inside ComfyUI (alongside quantized kernels like SageAttention).
FlashAttention & GPU Kernels
Also known as: FlashAttention · FlashAttention-3 · Triton · CUDA Kernels
Plain English
Low-level GPU engineering breakthroughs that compute Transformer attention 2–8× faster and with far less memory—without changing the mathematical result—by keeping data inside ultra-fast on-chip SRAM instead of constantly shuttling it to main VRAM.
Technical Overview
Standard self-attention materializes an O(N²) attention matrix in GPU HBM. Tri Dao's FlashAttention tiles Q, K, and V blocks inside on-chip SRAM using online softmax recomputation, achieving IO-aware O(N) memory complexity. Hardware-specific kernels (FlashAttention-3/4, SageAttention, FlexAttention, OpenAI Triton) power both LLM training/Inference Engines (vLLM, SGLang, llama.cpp) and long-sequence Video Generation Models (Sora, Veo, Kling, Wan).
Chronological Entries (2)
Tri Dao et al. published FlashAttention, proving that exact IO-aware GPU kernel design can eliminate the quadratic memory wall of Transformer self-attention and directly unlock long context windows and high-resolution DiT training.
Direct Relations (3)
- EnablesTransformer
- EnablesInference Engines (vLLM, SGLang, llama.cpp)
- EnablesContext Window & RoPE