FIFoundationIndex
IndexMapAbout
ArchitectureLLMCross-Modal

Mixture of Experts (MoE)

Also known as: MoE · Sparse MoE

Plain English

A design that packs multiple specialized sub-networks ('experts') inside one large model, but only activates a small fraction of them for any given token—delivering big-model intelligence at small-model speed.

Technical Overview

Sparse MoE replaces dense Transformer feed-forward layers with a router network that dispatches each token to the top-k experts (e.g., 2–8 out of 64–256 experts). Total parameter count governs VRAM capacity, while active parameter count governs FLOPs and inference latency, making Quantization and distributed serving in Inference Engines (vLLM, SGLang, llama.cpp) critical.

Chronological Entries (2)

#2Paper
Curated

DeepSeek's DeepSeek-V3 popularized fine-grained MoE (671B total parameters, 37B active) with shared always-on experts and auxiliary-loss-free load balancing, setting the template for frontier open-weight MoE scaling across 2025–2026.

#1Release
Curated

Mistral AI released Mixtral 8×7B (46.7B total parameters, ~12.9B active per token), proving to the open-weight community that sparse Mixture of Experts (MoE) models could match much larger dense models at a fraction of inference compute.