LLM + RoboticsUnit 6 · Deep RL & alignment · Day 30

Deep Reinforcement Learning from Human Preferences

When you can't write the reward function, show a person two short clips, ask which is better, and learn the reward from their answers.

AuthorsPaul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei
Year / venue2017 · NIPS (OpenAI and DeepMind)
Core ideaReward model from pairwise comparisons
Study time3–5 h

Original: arXiv:1706.03741 · Code, notebook and exercises: 30u30 Day 30

Start here (no maths needed)

When it's too hard to write down exactly what you want, you can teach a computer by showing a person two short videos and asking "which one is better?"

Think of it like this

Imagine teaching a dog a new trick, but you can't explain the trick in words. Instead, you watch two of its attempts and give a treat for the better one. Over time the dog figures out what you like. Here the computer does something extra: it builds its own "guess of what the person likes" from all the answers. Then it practises against that guess, so the person only has to answer now and then.

How it works

  1. The robot tries the task and records short video clips, 1–2 seconds each.
  2. A person watches two clips side by side and clicks the better one.
  3. A "reward model" learns to predict which clips the person will prefer.
  4. The robot practises lots, using the reward model's scores as points.
  5. Keep asking the person about new clips, so the reward model doesn't get fooled.

Why robots care

In this paper a simulated one-legged hopping robot learned to do backflips and land on its foot. Nobody wrote a rule for a "good backflip". People just picked the better of two clips, about 900 times, in under an hour. The same idea is now used to train chatbots to give helpful answers.

Try it

Words to know

Preference
Which of two options a person likes more.
Reward model
A program that learns to guess what a person would like.
Reward hacking
When the robot finds a sneaky way to get points without doing what you really wanted.
RLHF
Reinforcement learning from human feedback: learning from people's choices instead of a written score.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

Many tasks have goals that are easy to recognise and hard to specify, like "clean the table" or "do a backflip". This paper trains a reward model from human comparisons of 1–2 second video clips, and trains a policy with RL on that learned reward, all at the same time. The human labels under 1% of the agent's experience. With 700 human labels, agents nearly matched RL with the true reward on MuJoCo robot tasks. With 900 queries and under an hour of an author's time, the Hopper robot learned repeated backflips, a behaviour with no obvious hand-written reward. The same recipe, scaled up and run with PPO, later became RLHF for language models.

2. The core idea

P̂[σ¹ ≻ σ²] = exp(Σ_t r̂(o¹_t, a¹_t)) / ( exp(Σ_t r̂(o¹_t, a¹_t)) + exp(Σ_t r̂(o²_t, a²_t)) )
            = sigmoid( R̂(σ¹) − R̂(σ²) )

loss(r̂) = − Σ_{(σ¹, σ², μ) ∈ D}  μ(1)·log P̂[σ¹ ≻ σ²] + μ(2)·log P̂[σ² ≻ σ¹]

with label noise:  P = 0.9 · sigmoid(ΔR̂) + 0.1 · 0.5

Only reward differences are identified, so the paper normalises r̂ to a fixed mean and standard deviation before handing it to the RL algorithm.

3. Deep dive

The preference model, made concrete

Interactive: Bradley-Terry with a noisy human

x-axis: predicted reward difference ΔR̂ = R̂(σ¹) − R̂(σ²). Dashed: plain sigmoid. Solid: with the random-answer rate you pick (the paper uses 10%). The marker is the ΔR̂ you pick.

P(σ¹ preferred)
–
Loss if human picks σ¹
–
Loss if human picks σ²
–
Worst-case loss
–

Try this: keep ΔR̂ = 5 and set the random-answer rate to 0. If the human picks σ² anyway (a slip), the loss is about 5 nats and keeps growing as ΔR̂ grows. At 10% it's capped near 3 nats. One careless click can't dominate training.

What the experiments showed

The paper also removed hidden hints from the environments, such as episode termination when the robot falls, because that alone encodes part of the task. A neat detail worth copying when you evaluate learned rewards.

From backflips to chatbots

InstructGPT (2022) kept the structure: collect comparisons, fit a Bradley-Terry reward model, optimise with RL. It used PPO from Day 29 plus a KL penalty that keeps the policy close to the starting model, the guard against reward hacking this paper warned about. Two links back into this course: the reward model is a classifier whose probabilities should be calibrated (Module 01), and modelling the human as "usually picks the better clip, sometimes not" is the noisily-rational human model from Module 03. Shane Legg, a co-author, wrote Day 27.

4. Why it matters for robotics

5. How to study it

  1. Read sections 1–3 of the paper (about 1 h), then appendix A for the details on removing termination signals and label scheduling.
  2. Watch the backflip video linked from the paper and the OpenAI blog post.
  3. In the 30u30 Day 30 folder, read implementation.py: the reward network, the preference loss, a synthetic oracle and the RLHF loop with a PPO agent. Run train_minimal.py.
  4. Work through notebook.ipynb, then the exercises: Bradley-Terry, preference loss, reward normalisation, synthetic oracle, full loop.
  5. Stretch: train the reward model on early data only and watch the policy exploit it, reproducing the paper's offline ablation.

6. Self-check

Why compare two clips instead of asking for a score?

People give much more consistent comparisons than absolute scores. Comparisons also avoid the scale problem: rewards vary hugely across tasks, and predicting which is better is an easier learning problem than regressing exact values.

Why can the reward model only be identified up to an offset?

The Bradley-Terry probability depends only on the difference of summed rewards. Adding a constant to every reward changes nothing, so the paper normalises rewards before RL.

What does the 10% random-response assumption buy?

Human errors don't vanish when the reward gap is large. Without the term, a confident wrong label creates an unbounded loss. With it, predicted probabilities stay in [0.05, 0.95] and the loss is capped.

Why keep asking for labels during training?

As the policy improves it visits new states the reward model never saw. A frozen model gets exploited there, as the Pong agent that never scored showed.

How many queries did the Hopper backflip take?

900, in under an hour of human time, with feedback from the authors.

Which RL algorithms did the paper use, and what replaced them in LLM RLHF?

TRPO for MuJoCo and A2C for Atari. InstructGPT-style RLHF used PPO with a KL penalty to the initial model.

7. Further reading