<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Attention — FoundationIndex</title>
    <link>https://foundationindex.ustunozgur.com/s/attention</link>
    <description>The mechanism that lets a neural network weigh which parts of an input sequence matter most when processing each token, replacing slow step-by-step recurrence with parallelizable all-to-all context.</description>
    <language>en</language>
    <atom:link href="https://foundationindex.ustunozgur.com/s/attention/feed.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Attention · Paper · 2024-05-07</title>
      <link>https://foundationindex.ustunozgur.com/s/attention#attention-mla-deepseek-v2-2024</link>
      <guid>https://foundationindex.ustunozgur.com/s/attention#attention-mla-deepseek-v2-2024</guid>
      <pubDate>Tue, 07 May 2024 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;DeepSeek introduced Multi-Head Latent Attention (MLA) in DeepSeek-V2, compressing keys and values into a shared low-rank latent vector before caching—slashing KV cache memory by over 90% compared to standard MHA while maintaining frontier accuracy.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2405.04434&quot;&gt;DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Attention · Paper · 2023-05-22</title>
      <link>https://foundationindex.ustunozgur.com/s/attention#attention-gqa-2023</link>
      <guid>https://foundationindex.ustunozgur.com/s/attention#attention-gqa-2023</guid>
      <pubDate>Mon, 22 May 2023 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Ainslie et al. at Google Research proposed Grouped-Query Attention (GQA), sharing key and value heads across groups of query heads to shrink KV cache size by 4–8× while retaining Multi-Head Attention quality, quickly becoming standard in Llama and Mistral.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2305.13245&quot;&gt;GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Attention · Paper · 2022-05-27</title>
      <link>https://foundationindex.ustunozgur.com/s/attention#attention-flash-2022</link>
      <guid>https://foundationindex.ustunozgur.com/s/attention#attention-flash-2022</guid>
      <pubDate>Fri, 27 May 2022 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Dao et al. introduced FlashAttention, tiling attention matrix multiplications inside GPU SRAM so the quadratic N × N attention matrix never materializes in HBM—unlocking longer context windows with exact numerical equivalence.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2205.14135&quot;&gt;FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Attention · Paper · 2017-06-12</title>
      <link>https://foundationindex.ustunozgur.com/s/attention#attention-transformer-2017</link>
      <guid>https://foundationindex.ustunozgur.com/s/attention#attention-transformer-2017</guid>
      <pubDate>Mon, 12 Jun 2017 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Vaswani et al. published Attention Is All You Need, replacing recurrent layers entirely with Multi-Head Self-Attention (softmax(QK^T / sqrt(d_k))V) inside the Transformer, enabling massive GPU parallelism during pre-training.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/1706.03762&quot;&gt;Attention Is All You Need&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Attention · Paper · 2014-09-01</title>
      <link>https://foundationindex.ustunozgur.com/s/attention#attention-bahdanau-2014</link>
      <guid>https://foundationindex.ustunozgur.com/s/attention#attention-bahdanau-2014</guid>
      <pubDate>Mon, 01 Sep 2014 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Bahdanau et al. published Neural Machine Translation by Jointly Learning to Align and Translate, introducing additive attention over encoder hidden states so sequence-to-sequence decoders could dynamically focus on relevant source words instead of compressing an entire sentence into a fixed-length vector.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/1409.0473&quot;&gt;Neural Machine Translation by Jointly Learning to Align and Translate&lt;/a&gt;&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
