<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Parallelism Strategies (TP, PP, EP &amp; CP) — FoundationIndex</title>
    <link>https://foundationindex.ustunozgur.com/s/parallelism-strategies</link>
    <description>Strategies for splitting a large AI model and its context across multiple GPUs so models that exceed a single chip&apos;s memory or bandwidth can train and serve requests at low latency.</description>
    <language>en</language>
    <atom:link href="https://foundationindex.ustunozgur.com/s/parallelism-strategies/feed.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Parallelism Strategies (TP, PP, EP &amp; CP) · Milestone · 2026-08-07</title>
      <link>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-vllm-dcp-elastic-ep-2026</link>
      <guid>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-vllm-dcp-elastic-ep-2026</guid>
      <pubDate>Fri, 07 Aug 2026 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Following its May 2026 release of Elastic Expert Parallelism for zero-restart MoE worker scaling, vLLM introduced Decode Context Parallelism (DCP), partitioning the KV cache along the sequence dimension across GPUs during autoregressive decoding and merging partial attention outputs via log-sum-exp reduction to expand long-context serving capacity.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://vllm.ai/blog/2026-08-07-decode-context-parallelism&quot;&gt;Efficient Decode Context Parallelism with vLLM for Long Context Workloads&lt;/a&gt;, &lt;a href=&quot;https://vllm.ai/blog/2026-05-14-elastic-expert-parallelism&quot;&gt;Elastic Expert Parallelism in vLLM&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Parallelism Strategies (TP, PP, EP &amp; CP) · Milestone · 2025-12-17</title>
      <link>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-vllm-wide-ep-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-vllm-wide-ep-2025</guid>
      <pubDate>Wed, 17 Dec 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;The vLLM team detailed Wide Expert Parallelism (Wide-EP) with data-parallel attention (TP=1), Dual-Batch Overlap (DBO), and an Expert Parallel Load Balancer (EPLB) to serve DeepSeek MoE models at 2,200 tokens/s per H200 GPU without duplicating the KV cache across tensor-parallel ranks.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://vllm.ai/blog/2025-12-17-large-scale-serving&quot;&gt;vLLM Large Scale Serving: DeepSeek @ 2.2k tok/s/H200 with Wide-EP&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Parallelism Strategies (TP, PP, EP &amp; CP) · Release · 2025-02-24</title>
      <link>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-deepseek-dualpipe-deepep-2025</link>
      <guid>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-deepseek-dualpipe-deepep-2025</guid>
      <pubDate>Mon, 24 Feb 2025 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;DeepSeek open-sourced DualPipe and DeepEP, pairing bidirectional pipeline scheduling with RDMA- and NVLink-aware all-to-all Expert Parallelism (EP) dispatch and combine kernels that hide cross-node communication behind computation in large MoE training and disaggregated inference.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://github.com/deepseek-ai/open-infra-index&quot;&gt;DeepSeek Open Infra Index&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Parallelism Strategies (TP, PP, EP &amp; CP) · Paper · 2023-10-03</title>
      <link>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-ring-attention-cp-2023</link>
      <guid>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-ring-attention-cp-2023</guid>
      <pubDate>Tue, 03 Oct 2023 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Liu et al. formulated Ring Attention for Context Parallelism (CP), sharding long token sequences across a ring of accelerators and circulating key-value blocks concurrently with blockwise attention compute so sequence length scales linearly with device count.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/2310.01889&quot;&gt;Ring Attention with Blockwise Transformers for Near-Infinite Context&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Parallelism Strategies (TP, PP, EP &amp; CP) · Paper · 2019-09-17</title>
      <link>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-megatron-tensor-2019</link>
      <guid>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-megatron-tensor-2019</guid>
      <pubDate>Tue, 17 Sep 2019 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Shoeybi et al. at NVIDIA introduced Megatron-LM, formalizing intra-layer Tensor Parallelism (TP) by splitting MLP weight matrices column-wise and row-wise and partitioning attention heads across GPUs so each layer requires only two all-reduce synchronizations per forward-backward pass.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/1909.08053&quot;&gt;Megatron-LM: Training Multi-Billion Parameter Language Models Using GPU Model Parallelism&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    <item>
      <title>Parallelism Strategies (TP, PP, EP &amp; CP) · Paper · 2018-11-16</title>
      <link>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-gpipe-pipeline-2018</link>
      <guid>https://foundationindex.ustunozgur.com/s/parallelism-strategies#parallelism-gpipe-pipeline-2018</guid>
      <pubDate>Fri, 16 Nov 2018 00:00:00 GMT</pubDate>
      <description>&lt;p&gt;Huang et al. at Google introduced GPipe, partitioning sequential neural network layers across accelerators and splitting each mini-batch into smaller micro-batches to shrink idle pipeline bubbles during synchronous Pipeline Parallelism (PP), enabling 6B-parameter Transformer training across multiple devices.&lt;/p&gt;&lt;p&gt;Sources: &lt;a href=&quot;https://arxiv.org/abs/1811.06965&quot;&gt;GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism&lt;/a&gt;&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
