IndexMapAnti-DoomscrollAbout
ArchitectureLLMCross-Modal

Linear Attention

Also known as: Linear Transformers · Gated Linear Attention · GLA · DeltaNet · Gated DeltaNet · Lightning Attention · Kimi Delta Attention · RetNet

Plain English

A family of Transformer attention mechanisms whose compute scales linearly rather than quadratically with sequence length. By folding past keys and values into a fixed-size running matrix state, linear attention avoids the growing KV cache that slows down long-context decoding.

Technical Overview

Linear attention replaces softmax(QK^T)V with kernel or linear inner products so matrix-multiplication associativity yields φ(Q)(φ(K)^T V) in O(N d^2) training time and O(1) per-token decoding state S_t. Subsequent variants resolved additive memory interference via data-dependent decay gates (RetNet, GLA, Lightning Attention) and error-correcting delta updates (DeltaNet, Gated DeltaNet, Kimi Delta Attention), converging closely with Mamba-2 state-space duality and modern RNNs. Because fixed-size states compress exact token lookups, 2025–2026 frontier LLMs (MiniMax-01, Qwen3.5, Kimi K3) interleave linear layers with periodic softmax attention in ratios such as 3:1 to shrink KV cache memory while retaining retrieval accuracy.

Timeline (6)

RSS
  1. 2020

  2. 2023

    Paper

    Following July 2023's RetNet (arXiv:2307.08621), Yang et al. introduced Gated Linear Attention (GLA), adding data-dependent outer-product forget gates alongside a chunkwise Tensor Core algorithm (FlashLinearAttention). By replacing sequential prefix-sum scans with parallel intra-chunk matmuls, GLA closed the training-speed and perplexity gap with standard Transformers.

  3. 2024

    Paper

    Building on June 2024's WY-parallel DeltaNet (arXiv:2406.06484), Yang, Kautz, and Hatamizadeh at NVIDIA published Gated DeltaNet, pairing scalar memory decay α_t with the error-correcting delta rule S_t(I - β_t k_t k_t^T) + β_t v_t k_t^T so models can simultaneously erase stale keys and overwrite targeted associations in chunk-parallel Tensor Core kernels.

  4. 2025

  5. 2026

    Paper

    Following February 2026's 397B open-weight Qwen3.5 rollout (3:1 Gated DeltaNet + Gated Attention), Moonshot AI published the 2.8T-parameter Kimi K3 technical report combining 69 Kimi Delta Attention layers with 24 Gated MLA layers—cementing asymmetric linear-softmax hybrids as a frontier standard alongside distillation pipelines like HALO (arXiv:2601.22156).

See also (13)

Open full graph
8 more connections