SFT, RLHF & DPO
Compare supervised demonstration learning with two preference-based routes. Post-training is a broad stage, not one compulsory sequence.
| Question | Supervised fine-tuning (SFT) | Preference learning: RLHF / DPO |
|---|---|---|
| Training signal | A desired target response | Comparisons of responses under a specified rubric |
| Basic objective | Increase the likelihood of demonstrated response tokens | Reward-model RLHF: optimize a learned score. DPO: optimize reference-adjusted preference comparisons. |
| Separate reward model | Not required by the SFT objective | Reward-model RLHF: yes. Original DPO: no separate explicit reward model. |
| Responses used for updates | Demonstrated responses from the training dataset | PPO-style RLHF: sampled policy responses. Original offline DPO: fixed chosen/rejected pairs. |
| Reference policy | Not inherently required for the SFT likelihood objective | Common RLHF recipes constrain drift from a reference. The basic DPO objective uses a reference policy. |
| Shared limitation | Demonstrations can contain errors and gaps | Preferences and reward scores can contain errors and gaps |
A demonstration says “respond like this.” A preference pair says “prefer this response to that one under this rubric.” RLHF and DPO use that preference information through different objectives and training workflows.
Follow the evidence
Read the connected lessons for detailed mechanisms, assumptions, and primary references.
Supervised fine-tuning: demonstrations of behavior
Examples of good responses shift a pretrained model toward the behavior an application needs.
Preference learning: comparing alternatives
Sometimes people can judge which of two answers is better more easily than they can write an ideal answer.
RLHF: optimizing a learned reward
A common historical recipe learns a reward from comparisons, then uses it to improve a response policy.
DPO: learning directly from preference pairs
Direct Preference Optimization uses chosen and rejected responses to train a policy without a separate reward-model rollout loop.