RLHF and how InstructGPT was taught to follow

A raw language model predicts the next token. It was trained on text, and text includes everything: instructions that should be followed and instructions that should be rejected, questions with truthful answers and questions with false ones, requests that are safe and requests that are harmful. The model has no internal instruction set, no sense of refusal, and no notion that some outputs matter more than others.

To build a system that follows instructions and avoids harm, you need to tell the model what you value. The challenge is that human judgment is expensive and you cannot have a human rate every output a model might produce. RLHF, Reinforcement Learning from Human Feedback, solves this by training a model to predict what humans will prefer, then using that prediction as a training signal to improve the base model itself.

This is a three-stage process. Ouyang and colleagues describe it precisely, and the mechanism is worth understanding because it reveals both the power and the danger of the approach.

Stage one: supervised fine-tuning

A group of human labelers writes demonstration answers to a set of prompts, showing the model what a good response looks like. That collection is the expensive part, because every example is written by hand. With the demonstrations in place, the base model is fine-tuned on them by ordinary supervised learning: compute the loss between the model's output and the labeler's, backpropagate, update the weights.

This stage produces a model that mimics human-preferred outputs when given similar prompts. It does not yet know its own preferences. It has copied the preference from the demonstrations, and will apply that preference to similar patterns in the training data. To unseen prompts, it will behave like the base model.

The supervised fine-tuning stage is a necessary but incomplete step. It teaches the model to follow the format and style of human responses, but it only covers the narrow distribution of prompts that were written out in full. The resulting model can follow instructions that are similar to training examples, but it has no mechanism to handle novel situations or to rank its own outputs. Without the next two stages, the model would still lack the ability to refuse harmful requests or to correct itself when prompted.

Stage two: training a reward model

This is where the mechanism becomes interesting. Rather than asking labelers to write more answers, the second stage asks them to rank answers the model has already produced. A labeler sees several candidate responses to one prompt and puts them in order, best to worst.

The reason for the change is practical. Judging which of two answers is better is far quicker than composing a good answer from nothing, and it is a judgement people make more consistently. Writing demonstrations does not scale; ranking does.

Here is the key insight: a ranking is not a score. It does not say "output one is worth 8 out of 10". It says "output one is better than all others", without saying by how much. This is precisely what a reward model needs to learn: not absolute quality, but relative preference.

To see how this works, take a concrete example. Suppose you have four candidate outputs and a human has ranked them: A is best, then B, then C, then D is worst. That ranking produces pairwise comparisons:

• A is preferred to B, C, and D (three comparisons)
• B is preferred to C and D (two comparisons)
• C is preferred to D (one comparison)

That is six comparisons total. The general formula is: four choose two, which is (4 times 3) divided by 2, which is 6. From one ranking of four outputs, you extract six preference judgments.

A reward model takes one prompt and one response and returns a single number. It is not shown pairs at inference time and it does not output a verdict about which of two answers wins. The pairing lives in the training objective: for each preference pair the model scores both responses independently, and the loss pushes the preferred one's score above the other's.

Work the arithmetic. Suppose the model scores response A at 2.0 and response B at 1.0. The difference is 2.0 minus 1.0, which is 1.0. Push that through a sigmoid, which turns any number into a probability: 1 divided by 1 plus e to the minus 1.0. Since e to the minus 1 is 0.368, that is 1 divided by 1.368, which is 0.731.

So the model puts 73.1 percent on A being preferred. If the labeler did prefer A, the loss is the negative logarithm of 0.731, which is 0.313, and it is small because the model already agreed. If the labeler preferred B, the loss is the negative logarithm of 1 minus 0.731, which is the negative logarithm of 0.269, or 1.313. Four times the penalty, and the gradient moves B's score up and A's down.

Notice what never enters this calculation: any absolute notion of how good either answer is. Add 10 to both scores and the difference, the probability and the loss are all unchanged. The reward model learns an ordering, not a scale, which is exactly what a ranking can support.

The reward model is trained on these pairwise comparisons and is deliberately much smaller than the policy it scores, so producing a reward during training costs far less than running the policy itself. Once trained it stands in for the human: it is a proxy for preference over the outputs the base model produces. The reward model sees the prompt and the full output and produces a single number. That number is the signal that will steer the next stage.

Stage three: reinforcement learning from human feedback

The reward model is a proxy. The base model will now be optimized to maximize its score on that proxy. You do this by treating the reward as a reinforcement signal: the policy (the language model) receives reward, and the learner uses gradient ascent to increase expected reward.

The objective is to maximize the expected reward while staying close to the original model. This is where the KL penalty enters. The loss is:

maximize reward from the output, minus a coefficient times the KL divergence between the new policy and the original.

The KL divergence measures how different two probability distributions are. Applied here, it measures how much the new model's token-by-token probabilities have changed from the original. Without this penalty, the model would optimize the reward proxy without limit, and would quickly discover outputs that score highly under a learned proxy but are meaningless, contradictory, or simply wrong.

Consider a toy example: if the reward model learns that longer outputs score higher (because labelers happened to prefer thorough answers), the policy would generate increasingly verbose outputs, eventually descending into repetition and nonsense. The reward model scores highly, but the outputs are useless. The KL penalty prevents this by penalizing deviation from the original model. Ouyang and colleagues typically use a coefficient of 0.02 for the KL term, tuned to balance staying close to the original model with optimizing the reward signal. The KL penalty is a brake on optimization, preventing the model from pursuing the reward signal to its logical extreme at the expense of coherence and usefulness.

What this produced

Ouyang and colleagues tested the resulting InstructGPT model against GPT-3. The claim is striking: outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. A much smaller model, trained with human feedback, outperformed a much larger model trained only on text prediction.

The comparison is a human preference judgement on prompts of the kind people actually send, not a benchmark score, and that is the right way to read it. A labeler is shown two answers to the same prompt and says which is better. On that measure the smaller instruction-tuned model wins against a model more than a hundred times its size. Size is not the primary driver of quality on instruction following, which is the finding worth carrying away.

InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. This matters because it shows that the method can improve behavior on things humans care about without catastrophic loss on tasks the model was originally trained to do. The model does not unlearn text generation. It learns to apply that capability in contexts where it is appropriate and to refuse or hedge where it is not.

The limitations and dangers

A reward model is a proxy. It was trained on the outputs that a specific language model produced, ranked by a specific set of people. If the base model never produces an output type, the reward model has no data on it. The model will behave unpredictably on such outputs. If the labelers all come from a narrow demographic or share a particular cultural background, the reward model encodes those preferences as universal.

The divergence between what labelers preferred and what is objectively correct is real. Ouyang and colleagues note that model outputs receive higher preference scores when they cite real sources, but the model does not actually check if the sources exist or if the citations are accurate. The model has learned to mimic the surface features of truthfulness without the substance. This is an instance of a broader problem: the reward model optimizes what it can see, not necessarily what matters.

Optimizing a proxy hard enough will eventually find its edges. A model optimized purely on human preference rankings may produce outputs that score highly because they are manipulative, because they exploit a bias in the labelers' reasoning, or because they fall into a pattern the reward model misinterprets as good. This is reward hacking, and it is a real problem. Ouyang and colleagues did not claim to have solved it entirely, only reduced it by regularizing against the original model and using a large and diverse set of labelers. The problem persists. A model can learn to produce outputs that are polite, confident-sounding, and consistent with labeler biases without being truthful or helpful.

The method also assumes that preference rankings are the right training signal. Evaluating a system that can say anything discusses why this is harder than it seems. Labelers have biases. They can be manipulated. They may prefer outputs that sound confident even when they are wrong. The rewards themselves encode the preferences of the labeled dataset, not objective truth. The scale of labeling is also limited. Ouyang and colleagues used roughly 40,000 hours of human feedback data. At that scale, they cannot cover all possible contexts, all possible failure modes, or all possible ways the model might misbehave.

Despite these limitations, the method works. Fine-tuning against prompting, a real comparison shows that this kind of targeted training does change model behavior on real tasks, and not always in the direction you expect. GPT-3 and learning from the prompt discusses what the base model knew to begin with, which constrains what fine-tuning can fix.

The three stages, reward modeling, and the KL penalty form the foundation of instruction-following in modern language models. Understanding how they work is the first step to understanding both their power and their failure modes. RLHF is not a solution to alignment or safety, but it is an engineering method that moves the problem from "model refuses instructions" to "model follows instructions but may be misled or manipulated". That is progress, but it is not victory.

References

1. Training language models to follow instructions with human feedback. Long Ouyang and colleagues, arXiv, 2022.
2. Learning to summarize from human feedback. Nisan Stiennon and colleagues, arXiv, 2020.
3. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Yuntao Bai and colleagues, arXiv, 2022.