OpenAI published InstructGPT, demonstrating that supervised fine-tuning followed by PPO-based RLHF made a 1.3B model preferred by human evaluators over a 175B base GPT-3 checkpoint—establishing the template for modern assistant Post-Training (SFT, RLHF, DPO & GRPO).
RLHF
Also known as: Reinforcement Learning from Human Feedback · PPO Alignment
Plain English
A training stage where human raters compare model answers, a separate reward model learns their preferences, and reinforcement learning nudges the AI toward responses people actually prefer.
Technical Overview
RLHF trains a scalar reward model on pairwise human preference rankings and optimizes the language model policy against that reward using Proximal Policy Optimization (PPO) with a KL-divergence penalty to a reference checkpoint. While foundational for turning base foundation models into helpful assistants, its multi-model online loop is compute-heavy and sensitive to reward hacking—motivating simpler offline methods like DPO and verifiable-reward algorithms like GRPO.
Chronological Entries (3)
Stiennon et al. at OpenAI applied RLHF to neural text summarization, showing that optimizing a learned human-preference reward model with PPO outperformed supervised fine-tuning alone and generalized better to out-of-distribution articles.
Christiano et al. (OpenAI and DeepMind) published Deep reinforcement learning from human preferences, showing that an agent could learn complex behaviors from pairwise human comparisons via a learned reward predictor rather than hand-coded reward functions.
Direct Relations (4)
- Part ofPost-Training (SFT, RLHF, DPO & GRPO)
- Alternative toDPO
- Alternative toGRPO
- UsesOpenAI