IndexMapAbout
ConceptLLMCross-Modal

DPO

Also known as: Direct Preference Optimization

Plain English

A simpler way to align an AI model with human preferences by directly training it to prefer winning answers over losing ones, skipping the complex reward-model and reinforcement-learning loop of RLHF.

Technical Overview

Direct Preference Optimization reparameterizes the RLHF reward objective into a closed-form binary cross-entropy loss over chosen and rejected response pairs relative to a frozen reference policy. Eliminating the separate reward model and online PPO rollouts drastically lowers VRAM overhead and hyperparameter instability, making preference Post-Training (SFT, RLHF, DPO & GRPO) accessible to open-weight labs and single-node runs.

Chronological Entries (2)

Direct Relations (3)