Katharopoulos et al. (Transformers are RNNs) factored self-Attention using positive kernel feature maps (elu(x) + 1), reordering (QK^T)V into Q(K^T V) to formulate O(N) sequence training and a fixed-size matrix recurrence S_t = S_{t-1} + φ(k_t) v_t^T for constant-memory autoregressive decoding in Transformers.
Linear Attention
Also known as: Linear Transformers · Gated Linear Attention · GLA · DeltaNet · Gated DeltaNet · Lightning Attention · Kimi Delta Attention · RetNet
Plain English
A family of Transformer attention mechanisms whose compute scales linearly rather than quadratically with sequence length. By folding past keys and values into a fixed-size running matrix state, linear attention avoids the growing KV cache that slows down long-context decoding.
Technical Overview
Linear attention replaces softmax(QK^T)V with kernel or linear inner products so matrix-multiplication associativity yields φ(Q)(φ(K)^T V) in O(N d^2) training time and O(1) per-token decoding state S_t. Subsequent variants resolved additive memory interference via data-dependent decay gates (RetNet, GLA, Lightning Attention) and error-correcting delta updates (DeltaNet, Gated DeltaNet, Kimi Delta Attention), converging closely with Mamba-2 state-space duality and modern RNNs. Because fixed-size states compress exact token lookups, 2025–2026 frontier LLMs (MiniMax-01, Qwen3.5, Kimi K3) interleave linear layers with periodic softmax attention in ratios such as 3:1 to shrink KV cache memory while retaining retrieval accuracy.
Timeline (6)
2020
Paper2023
PaperFollowing July 2023's RetNet (
arXiv:2307.08621), Yang et al. introduced Gated Linear Attention (GLA), adding data-dependent outer-product forget gates alongside a chunkwise Tensor Core algorithm (FlashLinearAttention). By replacing sequential prefix-sum scans with parallel intra-chunk matmuls, GLA closed the training-speed and perplexity gap with standard Transformers.2024
PaperBuilding on June 2024's WY-parallel DeltaNet (
arXiv:2406.06484), Yang, Kautz, and Hatamizadeh at NVIDIA published Gated DeltaNet, pairing scalar memory decayα_twith the error-correcting delta ruleS_t(I - β_t k_t k_t^T) + β_t v_t k_t^Tso models can simultaneously erase stale keys and overwrite targeted associations in chunk-parallel Tensor Core kernels.2025
PaperMiniMax published MiniMax-01, scaling linear attention to a 456B-parameter MoE foundation model via a 7:1 hybrid of Lightning Attention and softmax Attention. Tiling intra-block exact matrix multiplies with inter-block linear state updates avoided cumulative-sum bottlenecks and supported 4M-token context windows.
- Paper
Shortly after Alibaba's September 2025 Qwen3-Next debuted a 3:1 Gated DeltaNet hybrid, Moonshot AI introduced Kimi Linear, generalizing Gated DeltaNet with channel-wise decay gating and a Diagonal-Plus-Low-Rank (DPLR) specialization (Kimi Delta Attention) in a 3:1 MLA hybrid that cut KV cache memory by up to 75%.
2026
PaperFollowing February 2026's 397B open-weight Qwen3.5 rollout (3:1 Gated DeltaNet + Gated Attention), Moonshot AI published the 2.8T-parameter Kimi K3 technical report combining 69 Kimi Delta Attention layers with 24 Gated MLA layers—cementing asymmetric linear-softmax hybrids as a frontier standard alongside distillation pipelines like HALO (
arXiv:2601.22156).
See also (13)
Open full graphLinear Attention
8 more connections
- TransformerOutgoing · Extends
- Mixture of Experts (MoE)Outgoing · MentionsIncoming · Uses
- Recurrent Neural NetworksOutgoing · MentionsIncoming · Enables
- Alibaba Cloud & QwenIncoming · Mentions
- Context Window & RoPEOutgoing · Mentions
- MiniMax & Hailuo VideoOutgoing · Mentions
- NVIDIAOutgoing · Mentions
- Olmo (Open Language Model)Incoming · Mentions