ByteDance Seed and Tsinghua researchers published DAPO, an open-source large-scale RL recipe refining GRPO with asymmetric clip ranges, dynamic sampling, and token-level policy gradients to prevent entropy collapse and length bias during long-horizon reasoning runs.
GRPO
Also known as: Group Relative Policy Optimization · RLVR
Plain English
A reinforcement learning method that grades a group of candidate answers from the same prompt against each other—often using automatic math or code checks—so models can learn step-by-step reasoning without needing a separate critic network.
Technical Overview
Introduced in DeepSeekMath, Group Relative Policy Optimization drops the learned value (critic) network required by PPO in RLHF and instead estimates advantages by normalizing rewards across a group of sampled completions for each prompt. Paired with rule-based verifiable rewards (RLVR) such as unit tests or math verifiers, GRPO halves RL memory overhead and became the primary algorithm behind open-weight reasoning Post-Training (SFT, RLHF, DPO & GRPO).
Chronological Entries (3)
DeepSeek published DeepSeek-R1, demonstrating that running GRPO with rule-based verifiable rewards (math and code checks) caused emergent long chain-of-thought self-verification without supervised reasoning traces—triggering industry-wide adoption of GRPO-style Post-Training (SFT, RLHF, DPO & GRPO).
DeepSeek introduced GRPO in the DeepSeekMath technical report, replacing PPO's separate value network with group-normalized reward baselines computed across multiple completions per prompt. Removing the critic model cut GPU memory requirements substantially during online RL.
Direct Relations (4)
- Part ofPost-Training (SFT, RLHF, DPO & GRPO)
- Created byDeepSeek
- Alternative toRLHF
- Alternative toDPO