Following OpenAI's o1 paradigm, DeepSeek published DeepSeek-R1 showing that pure Reinforcement Learning with Verifiable Rewards (via GRPO, using compiler/math checks as reward signals) induces emergent chain-of-thought self-verification—shifting lab compute budgets heavily toward post-training RL.
Post-Training (SFT, RLHF, DPO & GRPO)
Also known as: RLHF · DPO · GRPO · SFT · Alignment · Reasoning RL
Plain English
Everything done to a model after initial pre-training to turn a raw text-autocomplete engine into a helpful, safe assistant or a multi-step reasoning model that checks its own work.
Technical Overview
Post-training typically stages Supervised Fine-Tuning (SFT) on instruction/tool traces followed by preference optimization (RLHF via PPO, or reward-model-free DPO) and Reinforcement Learning with Verifiable Rewards (RLVR / GRPO). Verifiable RL on math, code, and agent tasks became the primary driver of test-time reasoning models.
Chronological Entries (2)
Rafailov et al. introduced Direct Preference Optimization (DPO), reframing RLHF as a simple classification loss on human preference pairs—eliminating the unstable reward model and PPO loop and accelerating open-weight Post-Training (SFT, RLHF, DPO & GRPO).
Direct Relations (3)
- UsesCheckpoints
- EnablesDistillation & Synthetic Data
- UsesEvals & Benchmarks