<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Pre-Training — FoundationIndex</title>
    <link>https://foundationindex.ustunozgur.com/s/pre-training</link>
    <description>The first and most expensive stage of building a foundation model: training a network from scratch on a huge, mostly unlabeled corpus so it learns general patterns before any fine-tuning.</description>
    <language>en</language>
    <atom:link href="https://foundationindex.ustunozgur.com/s/pre-training/feed.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Pre-Training · Release · 2025-11-20</title>
      <link>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-olmo3-staged-flow-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-olmo3-staged-flow-2025</guid>
      <pubDate>Thu, 20 Nov 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Ai2 released Olmo 3 (7B and 32B) with its complete Dolma 3 (~6T tokens) pre-training, mid-training, and long-context extension checkpoint sequence, letting researchers trace how each data-mix transition reshapes downstream reasoning and code performance.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://allenai.org/blog/olmo3&quot;&gt;OLMo 3: Charting a path through the model flow to lead open-source AI&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pre-Training · Paper · 2025-02-24</title>
      <link>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-muon-moonlight-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-muon-moonlight-2025</guid>
      <pubDate>Mon, 24 Feb 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Moonshot AI showed that the Muon optimizer, which orthogonalizes matrix updates, can be scaled to large mixture-of-experts models by adding weight decay and matching update magnitudes to AdamW. Its Moonlight models reached comparable quality with roughly half the training FLOPs of AdamW in the authors&amp;apos; scaling experiments, and Kimi K2 later used a clipped variant for a 15.5T-token run.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2502.16982&quot;&gt;Muon is Scalable for LLM Training&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/abs/2507.20534&quot;&gt;Kimi K2: Open Agentic Intelligence&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pre-Training · Paper · 2024-07-31</title>
      <link>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-llama3-annealing-2024</link>
      <guid>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-llama3-annealing-2024</guid>
      <pubDate>Wed, 31 Jul 2024 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;The Llama 3 report from Meta described a staged pre-training recipe: a long main run, context-length extension, and a final annealing phase on small amounts of high-quality code and math data while the learning rate decays to zero. This late data-schedule step, now often called mid-training, targets specific skills before post-training begins.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2407.21783&quot;&gt;The Llama 3 Herd of Models&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pre-Training · Paper · 2024-06-25</title>
      <link>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-dclm-fineweb-2024</link>
      <guid>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-dclm-fineweb-2024</guid>
      <pubDate>Tue, 25 Jun 2024 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Hugging Face released FineWeb, a 15-trillion-token web corpus filtered from Common Crawl, one week after the DataComp-LM (DCLM) benchmark of 17 June 2024 offered a standardized testbed for comparing data-curation methods at fixed training recipes. Both showed that model-based quality filtering can change downstream accuracy as much as scale does.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2406.17557&quot;&gt;The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/abs/2406.11794&quot;&gt;DataComp-LM: In search of the next generation of training sets for language models&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pre-Training · Paper · 2024-01-31</title>
      <link>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-dolma-2024</link>
      <guid>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-dolma-2024</guid>
      <pubDate>Wed, 31 Jan 2024 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;The Allen Institute for AI published Dolma, a three-trillion-token open corpus, together with its curation toolkit. Publishing the filtering and deduplication steps made data preparation, not only model weights, something others could inspect and reproduce.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2402.00159&quot;&gt;Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pre-Training · Paper · 2022-03-29</title>
      <link>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-chinchilla-2022</link>
      <guid>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-chinchilla-2022</guid>
      <pubDate>Tue, 29 Mar 2022 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;DeepMind trained over 400 models and concluded that parameters and training tokens should grow in roughly equal proportion. Its 70B-parameter Chinchilla, trained on 1.4T tokens, outperformed the larger 280B Gopher at the same compute budget, shifting later pre-training runs toward more data per parameter.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2203.15556&quot;&gt;Training Compute-Optimal Large Language Models&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Pre-Training · Paper · 2020-12-31</title>
      <link>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-pile-2020</link>
      <guid>https://foundationindex.ustunozgur.com/s/pre-training#pre-training-pile-2020</guid>
      <pubDate>Thu, 31 Dec 2020 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;EleutherAI released The Pile, an 825 GiB English corpus assembled from 22 sources, including academic papers, code, and web text. Open, documented corpora like this let outside groups train and study base language models without a private data pipeline.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2101.00027&quot;&gt;The Pile: An 800GB Dataset of Diverse Text for Language Modeling&lt;/a&gt;&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
