<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Mixed-Precision Training — FoundationIndex</title>
    <link>https://foundationindex.ustunozgur.com/s/mixed-precision</link>
    <description>A way to train neural networks using smaller, faster number formats for most of the math while keeping a few critical values in higher precision, so training runs faster and uses less memory without losing accuracy.</description>
    <language>en</language>
    <atom:link href="https://foundationindex.ustunozgur.com/s/mixed-precision/feed.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Mixed-Precision Training · Paper · 2026-01-30</title>
      <link>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-quartet-ii-2026</link>
      <guid>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-quartet-ii-2026</guid>
      <pubDate>Fri, 30 Jan 2026 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Researchers published Quartet II, introducing an unbiased quantization scheme (EDEN) for NVFP4 microscaling Tensor Cores that applies across both forward and backward passes to reduce 4-bit mixed-precision pre-training gradient estimation error without falling back to higher precision.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2601.22813&quot;&gt;Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Mixed-Precision Training · Paper · 2025-09-29</title>
      <link>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-nvfp4-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-nvfp4-2025</guid>
      <pubDate>Mon, 29 Sep 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;NVIDIA reported pre-training a 12B-parameter model on 10T tokens in NVFP4, a 4-bit format with 16-value blocks and FP8 scales. The recipe combined random Hadamard transforms, two-dimensional weight scaling, and stochastic rounding of gradients, and the run&amp;apos;s validation loss and downstream accuracy tracked an FP8 baseline.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2509.25149&quot;&gt;Pretraining Large Language Models with NVFP4&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Mixed-Precision Training · Paper · 2025-05-30</title>
      <link>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-mxfp8-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-mxfp8-2025</guid>
      <pubDate>Fri, 30 May 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;An NVIDIA-authored paper gave recipes for pre-training with the OCP MXFP8 format, which uses one shared power-of-two scale per 32 values. An 8B-parameter model trained on 15T tokens with the recipe matched its BF16 baseline.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2506.08027&quot;&gt;Recipes for Pre-training LLMs with MXFP8&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Mixed-Precision Training · Paper · 2024-12-27</title>
      <link>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-deepseek-v3-fp8-2024</link>
      <guid>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-deepseek-v3-fp8-2024</guid>
      <pubDate>Fri, 27 Dec 2024 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;The DeepSeek-V3 report validated FP8 mixed precision on a very large mixture-of-experts model, using fine-grained scaling: 1x128 tiles for activations and 128x128 blocks for weights, with higher-precision accumulation. The authors reported relative loss error under 0.25% against BF16 on their runs.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2412.19437&quot;&gt;DeepSeek-V3 Technical Report&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Mixed-Precision Training · Paper · 2022-09-12</title>
      <link>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-fp8-formats-2022</link>
      <guid>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-fp8-formats-2022</guid>
      <pubDate>Mon, 12 Sep 2022 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;NVIDIA, Arm, and Intel proposed two 8-bit floating-point encodings: E4M3 for weights and activations, and E5M2 for gradients that need wider range. Training language models up to 175B parameters matched 16-bit quality in their tests, opening the path to FP8 tensor-core training.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2209.05433&quot;&gt;FP8 Formats for Deep Learning&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Mixed-Precision Training · Paper · 2019-05-29</title>
      <link>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-bfloat16-2019</link>
      <guid>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-bfloat16-2019</guid>
      <pubDate>Wed, 29 May 2019 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Intel and Facebook researchers published a study of BFLOAT16 training across vision, speech, language, generative, and recommendation models. BF16 keeps the FP32 exponent range with a shorter mantissa, so it trained these models without loss scaling, which is why it became the default pre-training format on many accelerators.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/1905.12322&quot;&gt;A Study of BFLOAT16 for Deep Learning Training&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Mixed-Precision Training · Paper · 2017-10-10</title>
      <link>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-fp16-2017</link>
      <guid>https://foundationindex.ustunozgur.com/s/mixed-precision#mixed-precision-fp16-2017</guid>
      <pubDate>Tue, 10 Oct 2017 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Researchers from NVIDIA and Baidu described training with FP16 weights, activations, and gradients plus an FP32 master copy of weights. Loss scaling prevented small gradients from underflowing in FP16, which roughly halved memory use for those tensors while matching FP32 accuracy on the tested models.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/1710.03740&quot;&gt;Mixed Precision Training&lt;/a&gt;&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
