<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Linear Attention — FoundationIndex</title>
    <link>https://foundationindex.ustunozgur.com/s/linear-attention</link>
    <description>A family of Transformer attention mechanisms whose compute scales linearly rather than quadratically with sequence length. By folding past keys and values into a fixed-size running matrix state, linear attention avoids the growing KV cache that slows down long-context decoding.</description>
    <language>en</language>
    <atom:link href="https://foundationindex.ustunozgur.com/s/linear-attention/feed.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Linear Attention · Paper · 2026-07-27</title>
      <link>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-kimi-k3-hybrid-2026</link>
      <guid>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-kimi-k3-hybrid-2026</guid>
      <pubDate>Mon, 27 Jul 2026 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Following February 2026&amp;apos;s 397B open-weight Qwen3.5 rollout (3:1 Gated DeltaNet + Gated Attention), Moonshot AI published the 2.8T-parameter Kimi K3 technical report combining 69 Kimi Delta Attention layers with 24 Gated MLA layers—cementing asymmetric linear-softmax hybrids as a frontier standard alongside distillation pipelines like HALO (arXiv:2601.22156).&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2607.24653&quot;&gt;Kimi K3: Open Frontier Intelligence&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Linear Attention · Paper · 2025-10-30</title>
      <link>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-kimi-linear-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-kimi-linear-2025</guid>
      <pubDate>Thu, 30 Oct 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Shortly after Alibaba&amp;apos;s September 2025 Qwen3-Next debuted a 3:1 Gated DeltaNet hybrid, Moonshot AI introduced Kimi Linear, generalizing Gated DeltaNet with channel-wise decay gating and a Diagonal-Plus-Low-Rank (DPLR) specialization (Kimi Delta Attention) in a 3:1 MLA hybrid that cut KV cache memory by up to 75%.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2510.26692&quot;&gt;Kimi Linear: An Expressive, Efficient Attention Architecture&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Linear Attention · Paper · 2025-01-14</title>
      <link>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-minimax01-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-minimax01-2025</guid>
      <pubDate>Tue, 14 Jan 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;MiniMax published MiniMax-01, scaling linear attention to a 456B-parameter MoE foundation model via a 7:1 hybrid of Lightning Attention and softmax Attention. Tiling intra-block exact matrix multiplies with inter-block linear state updates avoided cumulative-sum bottlenecks and supported 4M-token context windows.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2501.08313&quot;&gt;MiniMax-01: Scaling Foundation Models with Lightning Attention&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Linear Attention · Paper · 2024-12-09</title>
      <link>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-gated-deltanet-2024</link>
      <guid>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-gated-deltanet-2024</guid>
      <pubDate>Mon, 09 Dec 2024 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Building on June 2024&amp;apos;s WY-parallel DeltaNet (arXiv:2406.06484), Yang, Kautz, and Hatamizadeh at NVIDIA published Gated DeltaNet, pairing scalar memory decay α_t with the error-correcting delta rule S_t(I - β_t k_t k_t^T) + β_t v_t k_t^T so models can simultaneously erase stale keys and overwrite targeted associations in chunk-parallel Tensor Core kernels.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2412.06464&quot;&gt;Gated Delta Networks: Improving Mamba2 with Delta Rule&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Linear Attention · Paper · 2023-12-11</title>
      <link>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-gla-2023</link>
      <guid>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-gla-2023</guid>
      <pubDate>Mon, 11 Dec 2023 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Following July 2023&amp;apos;s RetNet (arXiv:2307.08621), Yang et al. introduced Gated Linear Attention (GLA), adding data-dependent outer-product forget gates alongside a chunkwise Tensor Core algorithm (FlashLinearAttention). By replacing sequential prefix-sum scans with parallel intra-chunk matmuls, GLA closed the training-speed and perplexity gap with standard Transformers.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2312.06635&quot;&gt;Gated Linear Attention Transformers with Hardware-Efficient Training&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Linear Attention · Paper · 2020-06-29</title>
      <link>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-katharopoulos-2020</link>
      <guid>https://foundationindex.ustunozgur.com/s/linear-attention#linear-attention-katharopoulos-2020</guid>
      <pubDate>Mon, 29 Jun 2020 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Katharopoulos et al. (Transformers are RNNs) factored self-Attention using positive kernel feature maps (elu(x) + 1), reordering (QK^T)V into Q(K^T V) to formulate O(N) sequence training and a fixed-size matrix recurrence S_t = S_{t-1} + φ(k_t) v_t^T for constant-memory autoregressive decoding in Transformers.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2006.16236&quot;&gt;Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention&lt;/a&gt;&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
