RLHF: optimizing a learned reward
A common historical recipe learns a reward from comparisons, then uses it to improve a response policy.
In this lesson
RLHF uses feedback to shape a policy; a reward score is a proxy for desired behavior.
Name the moving parts
Reinforcement learning from human feedback is a family of methods. In the influential InstructGPT recipe, developers first trained with demonstrations, then learned a reward model from ranked responses, and then optimized a language-model policy against that reward. The 2022 paper documents this specific sequence; it should not be read as a specification of every present-day assistant.
The policy is the model distribution used to generate responses. The reward model maps a prompt-response pair to a numerical score. Its job is to approximate the preferences in the feedback data. The policy and reward model can be separate learned components with different outputs and failure modes.
How the loop differs from imitation
Instead of only increasing the likelihood of a fixed demonstration, the policy generates candidate responses. A reward signal scores their outcomes, and the training algorithm adjusts the policy toward higher-scoring behavior. In this setting, generated tokens act as decisions made along the way to a response-level result.
Proximal Policy Optimization, introduced in 2017, is one algorithm used for such updates. Its clipped probability-ratio objective discourages overly large updates relative to sampled behavior; it does not impose a hard bound on every policy change. An RLHF setup may also penalize divergence from a reference language model. These are related controls on the update, but they are not the same mathematical mechanism.
Why a reference matters
An unconstrained optimizer could chase high reward in ways that damage language quality or exploit weaknesses of the scorer. A reference penalty makes large distributional changes costly. Conceptually, the objective balances earning reward with remaining close to a useful starting behavior. The balance is a design choice, not an automatic definition of alignment.
An original toy example makes the risk concrete. Suppose a scorer accidentally treats longer responses as better. A policy might learn to append unnecessary sentences. The numerical score rises while user usefulness falls. More optimization has exposed the mismatch between the proxy and the real goal.
Training happens before the conversation
In ordinary deployment, the policy can answer using its learned weights without running a human-feedback training loop for every message. A visible thumbs-up button might be collected for later analysis or training, but it does not prove that a weight update happens immediately. The product’s documented data practices determine that separate issue.
RLHF also does not guarantee truth, eliminate bias, or create a universal set of human values. Feedback represents particular evaluators and instructions. The important questions are which behaviors improved, under which tests, and which failures remain. Other preference or reward-based methods may use different optimization algorithms, data sources, or objectives.
Policy versus reward model
The policy proposes “The answer is 12.” A reward model might assign that whole response a score of 1.7. That score is neither the probability of the token “12” nor proof that 12 is correct.
Every assistant uses the exact SFT → reward model → PPO recipe.
That is a documented historical recipe. Modern systems can mix different feedback sources, objectives, and optimization methods.
Take it with you
- Distinguish the generating policy from the reward model.
- Reward optimization needs controls against proxy exploitation.
- RLHF describes training, not a necessary step in every chat response.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
DPO: learning directly from preference pairs
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Training language models to follow instructions with human feedback ↗Long Ouyang et al. · 2022-03-04 · checked 13 Nov 2026
- Proximal Policy Optimization Algorithms ↗John Schulman et al. · 2017-07-20 · checked 13 Nov 2026