Preference learning: comparing alternatives
Sometimes people can judge which of two answers is better more easily than they can write an ideal answer.
In this lesson
A preference label is a judgment under a rubric, not a universal certificate of truth.
From examples to comparisons
Supervised demonstrations say “produce something like this.” A preference dataset instead presents a prompt and alternative responses, then records which response was preferred. The comparison can express a distinction that a single reference answer misses: two answers may both be relevant, but one may be clearer, better supported, or more appropriately cautious.
Christiano and colleagues’ 2017 work studied learning reward signals from human comparisons in reinforcement-learning settings. Language-model post-training later adopted related ideas. The historical connection concerns the learning signal; a comparison between text answers is not identical to a comparison between short action trajectories.
What the label actually means
Consider an original pair: answer A gives a short, supported conclusion; answer B gives a fluent but unsupported paragraph. A rubric emphasizing evidence should favor A. If evaluators instead reward apparent helpfulness or length, the labels may favor B. The model sees the labels produced by this process, not the intentions that remained unwritten.
Disagreement is information. Different readers may prefer different detail levels, and some comparisons are genuinely ambiguous. Recording uncertainty, ties, or separate quality dimensions can reveal what a forced binary label would hide. A preference dataset is a measurement instrument with a population, instructions, and error modes.
Several ways to use the signal
One route fits a reward model to predict preferences, then optimizes a response-generating model against that score. Another uses the pair directly in a preference objective, as in DPO. A third might generate several candidates and select one with a scoring model. These approaches differ in training cost, sampling, and what changes at inference.
Feedback need not always come directly from a person. The 2022 Constitutional AI paper investigated AI-generated critiques and preferences guided by written principles. That changes the source of supervision; it does not make the judgments independent of the principles, prompting, or models that produced them.
Evaluate the resulting behavior
Improved agreement with a preference set is useful evidence, but the set may omit difficult cases. An evaluator might prefer an incorrect answer because it is persuasive. A reward predictor can also make errors on responses unlike those in its training data. Optimization can amplify those weaknesses.
Separate the criteria you care about. Factual accuracy, tone, completeness, refusal behavior, and latency are not interchangeable. A sensible evaluation includes direct tests for the important properties and looks for cases where a pleasing style conceals failure. The point of a preference signal is to improve behavior, not to replace independent inspection of it.
Two rubrics, different winners
For “Explain this to a ten-year-old,” a concise analogy may beat a dense derivation. For “Show the derivation,” the ranking may reverse. The prompt and rubric are part of the label’s meaning.
Human-preferred means objectively correct.
Preference judgments can reflect accuracy, style, bias, incomplete information, or a mixture. Their meaning depends on the evaluation setup.
Take it with you
- Comparisons complement demonstrations.
- Record who judged what, using which criteria.
- Preference optimization still needs independent factual evaluation.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
RLHF: optimizing a learned reward
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Deep reinforcement learning from human preferences ↗Paul Christiano et al. · 2017-06-12 · checked 13 Nov 2026
- Constitutional AI: Harmlessness from AI Feedback ↗Yuntao Bai et al. · 2022-12-15 · checked 13 Nov 2026