<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Pruning and Sparsity — FoundationIndex</title>
    <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity</link>
    <description>Making a trained model smaller or faster by removing weights, neurons, layers, or experts that matter least. The pruned model usually needs a short round of retraining to recover quality.</description>
    <language>en</language>
    <atom:link href="https://foundationindex.ustunozgur.com/s/pruning-sparsity/feed.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Pruning and Sparsity · Paper · 2026-11-16</title>
      <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-gear2-moe-reuse-2026</link>
      <guid>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-gear2-moe-reuse-2026</guid>
      <pubDate>Mon, 16 Nov 2026 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Researchers introduced GEAR and GEAR² for MoE expert pruning, reconstructing the outputs of pruned experts from remaining active experts plus a lightweight missing-mass residual to reduce the accuracy loss from removing experts without recovery training.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2611.20616&quot;&gt;GEAR²: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pruning and Sparsity · Paper · 2025-10-15</title>
      <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-reap-moe-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-reap-moe-2025</guid>
      <pubDate>Wed, 15 Oct 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;&amp;quot;REAP the Experts&amp;quot; studied one-shot compression of mixture-of-experts models and argued that pruning whole experts can outperform merging them for generative tasks, extending structured pruning to sparse MoE LLMs.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2510.13999&quot;&gt;REAP the Experts: Why Pruning Prevails for One-Shot MoE compression&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pruning and Sparsity · Paper · 2024-07-19</title>
      <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-minitron-2024</link>
      <guid>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-minitron-2024</guid>
      <pubDate>Fri, 19 Jul 2024 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;NVIDIA published Minitron, which compresses a pretrained LLM with structured pruning of depth, width, heads, and MLP dimensions followed by knowledge distillation retraining. The paper reports pruned 8B and 4B models from a 15B parent using up to 40 times fewer training tokens than training from scratch.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2407.14679&quot;&gt;Compact Language Models via Pruning and Knowledge Distillation&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pruning and Sparsity · Paper · 2023-06-20</title>
      <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-wanda-2023</link>
      <guid>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-wanda-2023</guid>
      <pubDate>Tue, 20 Jun 2023 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Wanda scored each weight by its magnitude times the norm of its input activation, pruning LLMs per output without retraining or weight updates. It matched or beat magnitude pruning and was far simpler than second-order methods.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2306.11695&quot;&gt;A Simple and Effective Pruning Approach for Large Language Models&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pruning and Sparsity · Paper · 2023-01-02</title>
      <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-sparsegpt-2023</link>
      <guid>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-sparsegpt-2023</guid>
      <pubDate>Mon, 02 Jan 2023 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;SparseGPT introduced one-shot pruning for very large language models, pruning OPT-175B and BLOOM-176B to at least 50 percent sparsity in a single pass without retraining, using an approximate layer-wise solver.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2301.00774&quot;&gt;SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pruning and Sparsity · Paper · 2021-04-16</title>
      <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-nvidia-2-4-sparsity-2021</link>
      <guid>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-nvidia-2-4-sparsity-2021</guid>
      <pubDate>Fri, 16 Apr 2021 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;NVIDIA researchers published &amp;quot;Accelerating Sparse Deep Neural Networks&amp;quot;, describing the 2:4 fine-grained structured sparsity pattern supported by Sparse Tensor Cores in Ampere GPUs and a prune-then-fine-tune recipe that preserves accuracy, so a regular sparsity pattern could be turned into hardware speedup.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2104.08378&quot;&gt;Accelerating Sparse Deep Neural Networks&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pruning and Sparsity · Paper · 2018-03-09</title>
      <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-lottery-ticket-2018</link>
      <guid>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-lottery-ticket-2018</guid>
      <pubDate>Fri, 09 Mar 2018 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Frankle and Carbin&amp;apos;s Lottery Ticket Hypothesis showed that dense networks contain small sparse subnetworks that, trained from their original initialization, can match the full network&amp;apos;s accuracy, reframing pruning as a search for trainable subnetworks.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/1803.03635&quot;&gt;The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pruning and Sparsity · Paper · 2015-10-01</title>
      <link>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-deep-compression-2015</link>
      <guid>https://foundationindex.ustunozgur.com/s/pruning-sparsity#pruning-deep-compression-2015</guid>
      <pubDate>Thu, 01 Oct 2015 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Han, Mao, and Dally&amp;apos;s Deep Compression combined magnitude pruning, trained Quantization, and Huffman coding to shrink convolutional networks by an order of magnitude without accuracy loss, making pruning a mainstream compression technique.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/1510.00149&quot;&gt;Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding&lt;/a&gt;&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
