Reinforcement Learning From Human Feedback (RLHF) Explained

A language model trained only to predict the next word is good at continuing text but not necessarily at being helpful, honest or safe. Ask it a question and it might continue with another question, ramble or produce something offensive, because such text exists on the internet. Reinforcement Learning from Human Feedback (RLHF) is one of the main techniques used to turn raw language models into assistants that follow instructions and behave the way people prefer.

The core idea

Instead of writing rules for good behaviour, collect human judgements about which answers are better, learn a model of those preferences, and then adjust the language model to produce answers that score well. Human taste becomes the training signal.

The three stages

Stage 1: supervised fine-tuning (SFT)

Start with a pretrained model. Human writers create example conversations: prompts with high-quality responses. The model is fine-tuned on them with ordinary supervised learning. After this stage it already follows instructions much better, but it can only imitate the examples it saw.

Stage 2: train a reward model

For many prompts, the model generates several different answers. Human labellers compare them and rank which is better, following guidelines about helpfulness, accuracy and harmlessness. A separate neural network, the reward model, is trained to predict these preferences: given a prompt and an answer, it outputs a score that is higher for answers humans would prefer.

Comparisons are used because people find it easier to say "A is better than B" than to give a consistent absolute score.

Stage 3: optimise the model with reinforcement learning

Now the language model is treated as a policy that chooses words. It generates an answer, the reward model scores it, and a reinforcement-learning algorithm (commonly PPO, proximal policy optimisation) adjusts the model to increase the expected reward.

A crucial extra term, a KL penalty, keeps the new model close to the original. Without it, the model could discover odd outputs that fool the reward model, a failure called reward hacking, producing high scores with poor real quality.

A tiny sketch of preference learning

The reward model is usually trained with a pairwise loss: the preferred answer should get a higher score than the rejected one. In simplified form:

import numpy as np

def pairwise_loss(score_chosen: float, score_rejected: float) -> float:
    # Bradley-Terry style loss: small when chosen >> rejected
    return float(-np.log(1 / (1 + np.exp(-(score_chosen - score_rejected)))))

print(round(pairwise_loss(2.0, -1.0), 3))   # model agrees with humans -> low loss
print(round(pairwise_loss(-1.0, 2.0), 3))   # model disagrees -> high loss

Real systems use large neural networks for scoring, but the objective is the same.

Alternatives and improvements

Direct Preference Optimization (DPO)

DPO skips the explicit reward model and the reinforcement-learning loop. It directly adjusts the language model using preference pairs with a loss that encourages preferred answers over rejected ones relative to a reference model. It is simpler and more stable to run, and is widely used.

RLAIF and constitutional approaches

Reinforcement learning from AI feedback uses an AI model, guided by written principles, to produce some of the preference labels, reducing the amount of human labelling needed. Humans still design the principles and verify results.

Reinforcement learning with verifiable rewards

For tasks with checkable answers, such as maths problems or code that must pass tests, the reward can come from an automatic verifier instead of human opinions. This approach has been important for improving reasoning abilities.

Process supervision

Rewarding each reasoning step rather than only the final answer can encourage more reliable reasoning, though labelling steps is costly.

What RLHF changes in practice

• Models follow instructions more closely.
• They refuse clearly harmful requests more often.
• They adopt a helpful, polite conversational style.
• They become more likely to admit limitations, although not reliably.

Limitations and risks

1. Human feedback is imperfect. Labellers disagree, get tired and bring cultural and personal biases. The model learns the preferences of those particular people and guidelines, which may not represent everyone.
2. Sycophancy: models may learn to agree with users or flatter them because people tend to rate agreeable answers higher, even if they are less accurate.
3. Reward hacking: optimising a proxy reward can produce answers that look good to the reward model but are not truly better, such as verbose or confident-sounding text.
4. Style over substance: humans often prefer longer, well-formatted answers, which can encourage padding.
5. Reduced diversity: heavy optimisation can make outputs more uniform.
6. Hallucination is not solved: raters cannot always verify facts, so confident wrong answers may be rewarded.
7. Cost and labour: quality labelling is expensive, and annotators may be exposed to disturbing content, which raises ethical concerns about working conditions.
8. Alignment is shallow if poorly done: clever prompts can still bypass trained behaviour.
9. Value choices: deciding what counts as "better" is a social and ethical question, not just a technical one.

What RLHF is not

It does not give a model new factual knowledge; that comes mostly from pretraining. It also does not guarantee truthfulness or safety. It shapes behaviour and style within what the model can already do.

Why this matters for users and builders

When a chatbot refuses a request, writes in a certain tone or gently disagrees, that behaviour reflects training with human feedback and the guidelines behind it. If you build on such models, remember that their defaults came from someone else's preferences. You can still steer them with system prompts, fine-tuning on your own preference data, and guardrails that match your own users and context.

Frequently Asked Questions

Who provides the human feedback?

Trained annotators and sometimes domain experts, following written guidelines, and sometimes user feedback such as thumbs up or down.

Is RLHF the same as fine-tuning?

It is a form of fine-tuning that uses preference data and reinforcement learning. Ordinary supervised fine-tuning uses fixed example answers.

Why not just use rules?

Language is too varied for hand-written rules to cover. Learning preferences from examples generalises better to new situations.

Is DPO replacing RLHF?

DPO and related methods are popular because they are simpler, and are often used alongside or instead of PPO-based RLHF. The best choice depends on resources and goals.

Conclusion

RLHF aligns language models with human preferences by learning a reward from comparisons and optimising the model against it, while staying close to the original. It makes assistants more useful and safer, but inherits the limits and biases of the feedback that shapes it.