DPO: learning directly from preference pairs
Direct Preference Optimization uses chosen and rejected responses to train a policy without a separate reward-model rollout loop.
In this lesson
DPO compares how much a policy favors two responses relative to a reference policy.
A different route through post-training
Direct Preference Optimization, introduced by Rafailov and colleagues in 2023, derives a preference-learning objective from a particular reward-optimization formulation. It can train on offline pairs of preferred and rejected responses without fitting a separate explicit reward model and running the usual policy-sampling RL loop during each update.
This is a claim about the method’s structure, not a guarantee that its data are cheap or its outcomes are always better. Someone still needs to obtain suitable response pairs, define how preferences were judged, choose a reference model, and evaluate the resulting policy.
The comparison inside the objective
Let y+ be the preferred response and y− the rejected response for prompt x. A useful quantity is the difference between their policy log-probabilities, corrected by the same difference under a reference policy. It asks how much the new policy has shifted its relative preference, rather than simply how large one response’s probability is.
In compact notation, define Δ = log[π(y+|x)/πref(y+|x)] − log[π(y−|x)/πref(y−|x)]. The basic DPO loss is −log σ(βΔ), where σ is the sigmoid and β is a tuning coefficient. This equation summarizes the basic objective; implementations may add variations. It is not a formula for the factual correctness of either response.
Read a numerical example carefully
In an original simplified calculation, suppose the new policy has doubled the probability of the preferred response relative to its reference probability and left the rejected response unchanged relative to reference. Then Δ = ln 2. With β = 1, σ(Δ) = 2/3, giving loss about 0.405. At Δ = 0 the loss would be about 0.693.
These are hypothetical sequence-probability ratios, not token frequencies or performance results. Sequence log-probabilities aggregate token log-probabilities across a response. Formatting, response length, and which positions are included therefore matter when implementing or interpreting the objective.
The assumptions travel with the method
The derivation connects a preference model and a reference-regularized optimization problem. Real preference datasets can violate the neat assumptions: annotators disagree, prompts differ from deployment, and chosen responses can still contain errors. Direct optimization does not make the labels infallible.
A useful comparison with RLHF focuses on which components are trained, where samples come from, and what objective is optimized. Avoid reducing it to “simple versus complex” or “safe versus unsafe.” Both routes need data quality checks and evaluations beyond the training objective. A high-quality response should survive inspection of its claims, not merely win a likelihood comparison.
The zero-change baseline
When the policy exactly matches its reference, both log-ratios are zero. The preference probability inside the basic DPO objective is then 0.5, and the loss is ln 2. This is a clean implementation check.
DPO removes the need for preference judgments.
DPO changes how preference data train the model. It still depends on the quality and coverage of those data.
Take it with you
- DPO uses paired chosen and rejected responses.
- The reference policy is part of the objective.
- A preference loss is not a truth metric.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Choose your next learning path
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model ↗Rafael Rafailov et al. · 2023-05-29; revision 2024-07-29 · checked 13 Nov 2026