Llama 4 (Scout and Maverick) shifted to early-fusion multimodal Mixture of Experts (MoE) architectures (17B active parameters per token), optimizing high-throughput serving on FP8 Inference Engines (vLLM, SGLang, llama.cpp).
Llama
Also known as: Llama 3 · Llama 4 · Meta Llama
Plain English
Meta's flagship family of open-weight language and multimodal models. For years, Llama served as the reference architecture around which almost every open-source LLM runtime, fine-tuning tool, and quantization format was built.
Technical Overview
Beginning as dense decoder-only Transformers (Llama 1–3, scaling from 1B to 405B parameters with 128k vocab and GQA KV Cache), the lineage evolved in Llama 4 to native multimodal sparse Mixture of Experts (MoE) models distributed as Safetensors on Hugging Face and widely served via Inference Engines (vLLM, SGLang, llama.cpp).
Chronological Entries (2)
Meta AI released Llama 3.1 (8B, 70B, 405B) with 128K context windows and an updated license explicitly permitting synthetic data Distillation & Synthetic Data from Llama outputs—closing much of the gap with proprietary frontier models.
Direct Relations (4)
- Created byMeta AI
- ImplementsTransformer
- ImplementsMixture of Experts (MoE)
- UsesHugging Face