IndexMapAbout
ConceptLLMCross-Modal

RLHF

Also known as: Reinforcement Learning from Human Feedback · PPO Alignment

Plain English

A training stage where human raters compare model answers, a separate reward model learns their preferences, and reinforcement learning nudges the AI toward responses people actually prefer.

Technical Overview

RLHF trains a scalar reward model on pairwise human preference rankings and optimizes the language model policy against that reward using Proximal Policy Optimization (PPO) with a KL-divergence penalty to a reference checkpoint. While foundational for turning base foundation models into helpful assistants, its multi-model online loop is compute-heavy and sensitive to reward hacking—motivating simpler offline methods like DPO and verifiable-reward algorithms like GRPO.

Chronological Entries (3)

#2Paper
AI-Assisted

Stiennon et al. at OpenAI applied RLHF to neural text summarization, showing that optimizing a learned human-preference reward model with PPO outperformed supervised fine-tuning alone and generalized better to out-of-distribution articles.

Direct Relations (4)