Microsoft Research published Textbooks Are All You Need, training the 1.3B-parameter Phi-1 on ~7B tokens of filtered code and synthetically generated Python textbooks and exercises to outperform much larger models on HumanEval.
Phi
Also known as: Microsoft Phi · Phi-1 · Phi-2 · Phi-3 · Phi-3.5 · Phi-4
Plain English
Microsoft Research's family of open-weight Small Language Models (SLMs) that proved compact 1.3B–15B parameter models trained on curated 'textbook-quality' synthetic data can rival models an order of magnitude larger.
Technical Overview
Developed by Microsoft Research, the Phi lineage (Phi-1 through Phi-4) combines multi-agent Synthetic Data & Self-Play generation, strict web-corpus quality filtering, pivotal-token DPO, Mixture-of-LoRAs multimodal routing (LoRA adapters for vision and speech over a frozen language backbone), and RL test-time reasoning—making it a staple for on-device quantized inference via GGUF and Ollama.
Timeline (6)
2023
Paper- Release
Microsoft Research released Phi-2 (2.7B parameters), scaling its synthetic textbook and educational web data recipe to 1.4 trillion tokens and matching or surpassing 13B–70B open models of late 2023 on common-sense and math benchmarks.
2024
ReleaseMicrosoft launched the open-weight Phi-3 series—starting with Phi-3-mini (3.8B, trained on 3.3T tokens in 4K and 128K context variants) followed by Small (7B), Medium (14B), and the August 2024 Phi-3.5 MoE and Vision checkpoints—designed for 4-bit Quantization on phones and laptops.
- Release
Microsoft Research released Phi-4 (14B) under an MIT license, introducing 50+ multi-agent Synthetic Data & Self-Play generation pipelines alongside pivotal-token DPO (which weights preference optimization on the specific tokens that flip a reasoning chain from incorrect to correct).
2025
ReleaseMicrosoft released Phi-4-Multimodal (5.6B) and Phi-4-Mini (3.8B), introducing a Mixture-of-LoRAs architecture that attaches modality-specific LoRA adapters for vision and speech/audio onto a frozen text backbone without degrading base language accuracy.
- Release
Microsoft open-weighted Phi-4-reasoning and Phi-4-reasoning-plus (14B) alongside Phi-4-mini-reasoning (3.8B), pairing supervised reasoning traces with reinforcement learning for compact test-time compute scaling.
See also (14)
Open full graphPhi
9 more connections
- Synthetic Data & Self-PlayOutgoing · Uses
- Audio & Speech ModelsOutgoing · Mentions
- Context Window & RoPEOutgoing · Mentions
- GGUFOutgoing · Mentions
- Multimodal & Vision-Language ModelsOutgoing · Mentions
- OllamaOutgoing · Mentions
- Post-Training (SFT, RLHF, DPO & GRPO)Outgoing · Mentions
- QuantizationOutgoing · Mentions
- Reasoning & Test-Time ComputeOutgoing · Mentions