<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Scaling Laws — FoundationIndex</title>
    <link>https://foundationindex.ustunozgur.com/s/scaling-laws</link>
    <description>Empirical power-law rules showing how much better a foundation model gets as you add more parameters, training data, and compute—turning model training from guesswork into predictable engineering.</description>
    <language>en</language>
    <atom:link href="https://foundationindex.ustunozgur.com/s/scaling-laws/feed.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Scaling Laws · Paper · 2024-08-06</title>
      <link>https://foundationindex.ustunozgur.com/s/scaling-laws#scaling-laws-test-time-2024</link>
      <guid>https://foundationindex.ustunozgur.com/s/scaling-laws#scaling-laws-test-time-2024</guid>
      <pubDate>Tue, 06 Aug 2024 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Snell et al. published Scaling LLM Test-Time Compute Optimally, showing that dynamically allocating inference-time search and verification per prompt can outperform a 14× larger pre-trained model at equal FLOPs—opening a second inference-time scaling axis.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2408.03314&quot;&gt;Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Scaling Laws · Paper · 2022-12-14</title>
      <link>https://foundationindex.ustunozgur.com/s/scaling-laws#scaling-laws-openclip-2022</link>
      <guid>https://foundationindex.ustunozgur.com/s/scaling-laws#scaling-laws-openclip-2022</guid>
      <pubDate>Wed, 14 Dec 2022 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Cherti et al. extended power-law scaling analysis to contrastive vision-language models using OpenCLIP and LAION datasets, showing that zero-shot image classification and retrieval improve predictably with batch size, model scale, and paired image-text tokens.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2212.07143&quot;&gt;Reproducible scaling laws for contrastive language-image learning&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Scaling Laws · Paper · 2022-03-29</title>
      <link>https://foundationindex.ustunozgur.com/s/scaling-laws#scaling-laws-chinchilla-2022</link>
      <guid>https://foundationindex.ustunozgur.com/s/scaling-laws#scaling-laws-chinchilla-2022</guid>
      <pubDate>Tue, 29 Mar 2022 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Hoffmann et al. at DeepMind published Training Compute-Optimal Large Language Models (Chinchilla), proving prior models were undertrained and establishing that compute-optimal pre-training scales model parameters and training tokens in equal proportion (~20 tokens per parameter).&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2203.15556&quot;&gt;Training Compute-Optimal Large Language Models&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Scaling Laws · Paper · 2020-01-23</title>
      <link>https://foundationindex.ustunozgur.com/s/scaling-laws#scaling-laws-kaplan-2020</link>
      <guid>https://foundationindex.ustunozgur.com/s/scaling-laws#scaling-laws-kaplan-2020</guid>
      <pubDate>Thu, 23 Jan 2020 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Kaplan et al. at OpenAI published Scaling Laws for Neural Language Models, demonstrating that Transformer cross-entropy loss follows smooth power-law curves over seven orders of magnitude in compute, parameters, and dataset size—motivating massive parameter scaling in GPT-3.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2001.08361&quot;&gt;Scaling Laws for Neural Language Models&lt;/a&gt;&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
