LLM + RoboticsBuild a Robot Brain · Week 12 · Lesson 12

Learning what people prefer: reinforcement learning from human feedback, by hand

Nobody can write down the rule for "the best route". But anyone can look at two routes and say "that one". This lesson turns those quick answers into a reward the robot can learn from. It's the same recipe that turned raw language models into helpful chat assistants.

Time2–3 hours
You'll needPython 3, your finished Lessons 1, 4 and 6 (move, sigmoid, astar_path), paper, and one patient family member
New wordspreference, feature, reward model, Bradley-Terry, best-of-n, policy, RLHF, reward hacking
Project piecePicks the route people like best

Start here

If you can't write down what "good" means, ask people to compare pairs, learn a score that agrees with their choices, then pick whatever scores highest.

The problem

Robo-Helper can now carry the red block to the bin along the shortest route (Lesson 11). Your little brother watches and says "it keeps scraping the wall, and it wiggles too much". Your mum says "I don't care, just be quick". You try to write a rule: "shortest, unless it's near a wall, unless the other route has more turns, unless…". It never ends. Every rule you write has an exception. Yet everyone in the room can instantly point at a route on the screen and say "I like that one better".

The idea in one picture

Think about an eye test at the optician. They never ask "on a scale of 1 to 10, how sharp is this?" because nobody can answer that reliably. They ask "better with lens one… or lens two?" again and again. From a pile of easy "one or two" answers, they work out a precise prescription. We'll do the same: lots of easy "A or B?" answers, and maths turns them into a precise taste the robot can use.

Words to know

Preference
One answer to "which of these two do you like more?". Here, a pair of routes and which one the person chose.
Feature
A number that describes something about an option. Each route gets three: its length (steps), near_wall (how many squares brush a wall) and turns (how often it changes direction).
Reward model
A function that gives every option a score, so that the options people prefer get higher scores. Ours is a weighted sum of the three features.
Taste (weights)
The three numbers in the reward model. They say how much a person minds each feature. Negative means "I dislike this".
Bradley-Terry
The rule that turns a score gap into a chance of choosing: chance A wins = sigmoid(reward(A) − reward(B)). Named after two statisticians from the 1950s.
Best-of-n
Score all n options with the reward model and pick the top one.
Policy
Whatever decides what the robot (or a chatbot) does. Your A* planner is a policy. So is a language model choosing its next word.
RLHF
Reinforcement Learning from Human Feedback: learn a reward from people's comparisons, then use reinforcement learning to make the policy earn high reward.

Going deeper: everything below is the full lesson: the idea built from nothing, a demo where you are the person, your homework, and a challenge away from the screen.

1. The problem

Here is the room from world.py. The robot starts at (1, 1) and the bin B is at (7, 6).

##########
#R.......#
#..##....#
#..#.....#
#....##..#
#........#
#.#....B.#
##########

There isn't one route to the bin. There are lots. candidate_routes (written for you) makes them by forcing A* to pass through each free square on the way, a waypoint. That gives 24 different routes. Most are 11 steps long, some are 13 (they wander into a corner and come back). Here are two of them, both 11 steps:

Route P (A*'s shortest)      Route Q
##########                   ##########
#*.......#                   #*******.#
#*.##....#                   #..##..*.#
#*.#.....#                   #..#...*.#
#*...##..#                   #....##*.#
#***.....#                   #......*.#
#.#****B.#                   #.#....B.#
##########                   ##########

Same length. Which is "better"? Route P hugs the left wall and then scrapes along the bottom wall, and it changes direction 3 times. Route Q runs along the top, turns once, and drives straight down column 7, mostly in open floor. Most people prefer Q. But A* from Lesson 6 only knows about length, so it can't tell them apart and happens to give you P.

Obvious idea 1: write the rule yourself. "Each step costs 1, each wall-scrape costs 0.5, each turn costs 1." Where did 0.5 come from? You made it up. Is a turn worse than a scrape? For whom? Your brother and your mum would give different numbers, and neither could tell you theirs if you asked.

Obvious idea 2: ask people to score each route out of 10. Try it: someone says "7" for a route today and "5" for the same route tomorrow. One person's 7 is another's 9. Scores out of 10 are shaky.

What people are good at: comparing. "A or B?" takes a second, and the answers are much more consistent. So that's what we'll collect.

2. First principles

Step 1: describe each route with numbers (features)

A computer can't "look" at a route. So we describe it with three numbers, (length, near_wall, turns):

Let's count route P by hand. Its moves are S, S, S, S, E, E, S, E, E, E, E.

length    = 11 moves                                          → 11
turns     = S→E (after 4 S), E→S, S→E                         → 3
near_wall = (1,2) (1,3) (1,4) (1,5)   wall on their West      → 4
            (2,5)                     wall below it at (2,6)  → 1
            (3,5)                     open on all 4 sides     → 0
            (3,6) (4,6) (5,6) (6,6) (7,6)   bottom wall        → 5
                                                       total → 10
features(P) = (11, 10, 3)
Predict first: what are route Q's features?

(11, 8, 1). Moves: E × 6, then S × 5, so length 11 and 1 turn. Now near_wall, square by square:

(2,1) (3,1) (4,1) (5,1) (6,1) (7,1)   top wall above each       → 6
(7,2) (7,3)                           open on all 4 sides       → 0
(7,4)                                 wall to the West at (6,4) → 1   ← the sneaky one
(7,5)                                 open on all 4 sides       → 0
(7,6)                                 bottom wall below it      → 1
                                                         total → 8

Most people get 7 the first time, because the ## in row 4 is easy to miss. A square only counts once, even if it touches two walls (like (3,1), which has walls above and below). Counting by hand is fiddly, which is exactly why we write route_features and test it.

Step 2: a reward model is a weighted sum

Give the person a taste: three weights, one per feature. The reward of a route is

reward = w1 × length + w2 × near_wall + w3 × turns

In the code, the simulated person has TRUE_TASTE = (−1.0, −0.6, −0.8): every step costs 1 point, every wall-scrape 0.6, every turn 0.8. All negative, because they dislike all three.

reward(P) = −1.0 × 11 + −0.6 × 10 + −0.8 × 3 = −11 − 6.0 − 2.4 = −19.4
reward(Q) = −1.0 × 11 + −0.6 × 8  + −0.8 × 1 = −11 − 4.8 − 0.8 = −16.6

Q scores higher (less negative), so this person prefers Q. This is exactly Lesson 4's neuron: inputs times weights, added up. The big question is how to find the weights when nobody can tell you them.

Step 3: people are a bit random (Bradley-Terry)

If you show someone the same pair twice, they won't always pick the same one, especially when the two are close. So we don't say "the person picks the higher reward". We say:

chance the person picks A = sigmoid(reward(A) − reward(B))

That's Lesson 4's sigmoid, the S-shaped squash from any number into 0…1. Look at what it does with the reward gap:

gap = 0      → sigmoid(0)   = 0.50    a coin flip: they like both equally
gap = 0.8    → sigmoid(0.8) = 0.69    a slight favourite
gap = 2.8    → sigmoid(2.8) = 0.94    a clear favourite
gap = −2.8   → sigmoid(−2.8) = 0.06   a clear loser

Q against P has a gap of −16.6 − (−19.4) = 2.8, so this person picks Q 94% of the time. Routes (11, 9, 3) and (11, 9, 4) differ by one turn, a gap of 0.8, so they pick the smoother one only 69% of the time. Bigger gap, more certain choice. This is the Bradley-Terry model.

Predict first: two routes with exactly the same features. What's the chance the person picks A?

0.5. The gap is 0, and sigmoid(0) = 0.5. As far as the model is concerned, the person can't tell them apart, so they flip a coin. (That's also one of the checks in check.py: "equal rewards -> 0.5".)

Step 4: learn the taste from comparisons

Now flip it around. We don't know the weights. We only have a list of comparisons: (features of A, features of B, chose_a), where chose_a is 1 if they picked A and 0 if they picked B.

Start with weights [0, 0, 0] (no opinions). For each comparison, make a prediction p (the chance of A), see how wrong it was, and nudge each weight:

p = sigmoid(reward(A) − reward(B))
for each feature i:
    w[i] += lr × (chose_a − p) × (A[i] − B[i])

That's exactly Lesson 4's neuron learning rule, with one twist: the input is the difference between A and B. Let's run it by hand with lr = 0.01, on two real comparisons from the program's data.

Comparison 1: A = (11, 8, 1)  B = (11, 8, 3)  person chose A (chose_a = 1)
  weights now      [0, 0, 0]
  p                = sigmoid(0 − 0) = 0.5
  chose_a − p      = 1 − 0.5 = 0.5
  A − B            = (0, 0, −2)
  change           = 0.01 × 0.5 × (0, 0, −2) = (0, 0, −0.01)
  weights after    [0, 0, −0.01]       ← "turns are bad", learned!

Comparison 2: A = (13, 10, 4)  B = (11, 9, 5)  person chose A (!)
  (pretend weights are back at [0, 0, 0] so the numbers stay simple)
  p                = 0.5,   chose_a − p = 0.5
  A − B            = (2, 1, −1)
  change           = 0.005 × (2, 1, −1) = (0.01, 0.005, −0.005)
  weights after    [0.01, 0.005, −0.005]   ← "longer is good"?!

Look at comparison 2. The person picked the longer route. That happens: people are a bit random (Step 3). One answer pushes the length weight the wrong way. But over 150 comparisons, the random ones are outvoted by the sensible ones, just like averaging noisy sensor readings in Lesson 2. After 300 passes over all 150 comparisons, the program learns:

learned taste: [−1.03, −0.44, −0.73]
true taste:    (−1.0,  −0.6,  −0.8)

Pretty close, and nobody ever told it a single weight. It only saw "A" or "B".

Predict first: in comparison 1, why didn't the length weight or the near_wall weight change at all?

Because A and B had the same length (11 and 11) and the same near_wall (8 and 8). The difference is 0, and anything times 0 is 0. That comparison tells you nothing about length, only about turns. Hold onto that thought for Step 6.

Step 5: use the reward (best-of-n)

Now the robot has a reward model. The simplest way to use it: score all 24 candidate routes and drive the top one. That's best-of-n (here n = 24).

shortest route features (length, near_wall, turns): (11, 10, 3)   ← route P, what A* gave us
robot's chosen route features:                      (11, 8, 1)    ← route Q

The robot picks route Q: exactly as short as A*'s route, but it brushes walls on 8 squares instead of 10 and turns once instead of 3 times. It's also the very route the true taste ranks first. The robot learned what this person likes, from nothing but "A or B?".

How good is it, honestly?

We test on 100 new comparisons it never trained on (Lesson 5: always test on fresh data). The learned taste agrees with the person 84% of the time. Is 84% bad? Here's the twist: the true taste, the exact numbers inside the simulated person's head, also agrees with their own answers only 84% of the time on those same 100. The person is a bit random, so 84% is the ceiling. Nothing could do better. Our model hit it.

Step 6: a discovery (you can only learn about things that vary)

An earlier version of this lesson used only the routes that were exactly 11 steps long. It trained happily, and learned a length weight of 0.0. Not −1.0. Zero. The robot concluded that length doesn't matter at all.

Predict first: why would it learn exactly 0.0?

Every comparison had A[length] − B[length] = 11 − 11 = 0. The learning rule multiplies by that difference, so the length weight is nudged by 0 every single time and never leaves its starting value of 0. The data never showed a case where length differed, so the model had no evidence either way. (This is why some routes in candidate_routes are deliberately silly 13-step detours. They're the only examples that teach the robot "longer is worse".)

This is a big, general lesson: a model can only learn about things that vary in its data. If every car in your photos is red, your model learns nothing about colour. If every route is 11 steps, nothing about length. When you collect data, make sure the things you care about actually change.

Step 7: how this connects to real RLHF (and to Lesson 7)

Big chat assistants were trained with exactly this recipe, scaled up:

  1. Collect comparisons. The language model writes two answers to the same question. A person picks the better one. Repeat hundreds of thousands of times.
  2. Train a reward model. Instead of three features and three weights, the reward model is itself a big neural network that reads a whole answer and outputs one score. But the training rule is the same Bradley-Terry one you just wrote: sigmoid of the score gap, nudge towards what the person chose.
  3. Use the reward. Here's where real RLHF goes further than best-of-n. Best-of-n is honest and works (some real systems do it), but it means writing many answers every time and throwing most away. So instead they use reinforcement learning to change the policy itself (the language model) so it produces high-reward answers on the first try. The popular algorithms are PPO and, more recently, GRPO.

There's a catch. The reward model is only an imitation of people's taste, and it has blind spots. If you push the policy hard enough, it finds weird outputs that the reward model scores highly but that no person would like. That's called reward hacking. Imagine our robot finding a bizarre route the reward model loves because it has 0 turns, even though it drives through a gap no real robot fits through. To stop this, real RLHF adds a KL penalty: a leash that charges the policy points for drifting too far from how it behaved before training. "Get more reward, but stay close to your old self."

Compare with Lesson 7. In Q-learning, we wrote the reward: −1 per step, +10 at the goal. That works when you can write down what you want. Here the reward is learned from people, because nobody can write down "a route that feels nice" or "a helpful, polite answer". RLHF = "learn the reward from humans" + "reinforcement learning on that reward". You've now done both halves.

3. Play with it

Interactive: you are the person

The real ROOM and the same 24 routes as candidate_routes, with the same route_features. Click route A or route B (or use the buttons) to say which you prefer. After every answer, the reward model is retrained from scratch exactly like train_reward_model (lr 0.01, 300 passes), and the robot's pick (right) updates. The simulated-person buttons replay the program's own 150 comparisons from random.Random(0): use all 150 and you get the same taste and 84% as the Python.

Comparisons
0
Learned taste (length, wall, turns)
–
Agrees with the answers given
–
Agrees with simulated person (100 new)
–

Things to try: answer 10 pairs honestly by your own taste. Do your weights come out negative? Which feature do you mind most? Then Reset and try being weird: always choose the route with more turns. Watch the turns weight go positive and the robot pick a wiggly route. Finally Reset and press "+10" once: with only 10 simulated answers, the near_wall weight comes out positive (+0.96), the wrong sign. Keep pressing and watch it settle. Small data fools models.

4. Homework

Open homework/lesson12_preferences.py. It imports move (Lesson 1), sigmoid (Lesson 4) and astar_path (Lesson 6), so those must be finished. Already written for you: TRUE_TASTE, candidate_routes, ask_person (the simulated person) and collect_comparisons. Write these six functions, in order.

4.1 route_features(room, path)

Return the tuple (length, near_wall, turns) as defined in Step 1.

Example: route_features(ROOM, [(1, 1), (2, 1), (3, 1)]) → (2, 2, 0). Two steps, both landing squares touch the top wall, no change of direction.

Hint 1

Do the three numbers separately, one or two lines each. Length is the easiest: how many moves are in a list of squares?

Hint 2

near_wall: for each square in path[1:], use move(p, d) for each d in "NSEW" and is_free to see if any neighbour is a wall. Count each square at most once. turns: first turn the path into a list of directions (dx, dy), one per move, then count neighbours in that list that differ.

Hint 3

length = len(path) - 1. near_wall = number of p in path[1:] where any(not is_free(room, move(p, d)) for d in "NSEW"). dirs = list of (b[0] - a[0], b[1] - a[1]) for each pair a, b in zip(path, path[1:]). turns = number of pairs in zip(dirs, dirs[1:]) that aren't equal.

4.2 reward(weights, features)

The weighted sum: w1*f1 + w2*f2 + w3*f3.

Example: reward([-1, -0.5, -2], (10, 4, 1)) → −10 − 2 − 2 = -14.

Hint 1

This is the neuron from Lesson 4, without a bias.

Hint 2

zip(weights, features) pairs up each weight with its feature.

Hint 3

Return the sum of w * f for each w, f in that zip.

4.3 prob_prefers_a(weights, feats_a, feats_b)

Bradley-Terry: sigmoid(reward(A) − reward(B)).

Example: prob_prefers_a(TRUE_TASTE, (11, 8, 1), (11, 10, 3)) → about 0.943 (the 2.8 gap from Step 3).

Hint 1

One line, using two functions you already have.

Hint 2

Subtract the two rewards first, then squash the result.

Hint 3

return sigmoid(reward(weights, feats_a) - reward(weights, feats_b)) is the shape of it. Make sure A comes first: swapping them gives you the chance of B.

4.4 train_reward_model(comparisons, lr=0.01, epochs=300)

Learn the weights, using the rule from Step 4. Start at [0, 0, 0]. For each epoch, for each (feats_a, feats_b, chose_a): compute p, then nudge every weight. Return the weights.

Example: with the single comparison ((11, 8, 1), (11, 8, 3), 1) and epochs=1, you should get [0, 0, -0.01] (the hand calculation above).

Hint 1

Two loops, one inside the other: epochs on the outside, comparisons on the inside. The weights live outside both loops.

Hint 2

Compute p once per comparison, before changing any weight. Then all three weights use that same p.

Hint 3

w = [0.0, 0.0, 0.0]. For each epoch, for each fa, fb, y: p = prob_prefers_a(w, fa, fb), then build the new w where each weight is wi + lr * (y - p) * (a - b) for wi, a, b in zip(w, fa, fb). Return w.

4.5 agreement(weights, comparisons)

The fraction of comparisons where the model's favourite (whichever of A and B has the higher reward) is the one the person actually chose.

Example: if it gets 84 out of 100 right, return 0.84.

Hint 1

For each comparison, ask two yes/no questions: "does the model prefer A?" and "did the person choose A?". It's a hit when the answers match.

Hint 2

chose_a is 1 or 0, and reward(...) > reward(...) is True or False. bool(1) is True, so bool(y) lets you compare them.

Hint 3

Count comparisons where (reward(weights, fa) > reward(weights, fb)) == bool(y), then divide by len(comparisons).

4.6 best_route(weights, room, routes)

Best-of-n: return the route with the highest reward.

Hint 1

You need the features of each route before you can score it.

Hint 2

Python's max takes a key= function that says what to compare.

Hint 3

max(routes, key=lambda p: reward(weights, route_features(room, p))) is one way.

Run it

cd homework
python3 lesson12_preferences.py

What you should see (this is the real output of the reference solution):

24 different routes to the bin
learned taste: [-1.03, -0.44, -0.73] | true taste: (-1.0, -0.6, -0.8)
agrees with the person on NEW comparisons: 84%
shortest route features (length, near_wall, turns): (11, 10, 3)
robot's chosen route features:                      (11, 8, 1)

Your learned taste matches only if you update the weights in the same order (all comparisons, in order, every epoch). Small differences in the last decimal are fine.

Check it

cd ..
python3 check.py 12

A green ✓ means that part passed. A red ✗ comes with a hint, for example:

All green shows 4/4 passed.

Stretch

5. Real-world challenge: learn a real person's taste

You'll do RLHF on a real human, with paper.

  1. Pick options with features. Choose one: walking routes to school (features: minutes, number of busy roads to cross, how pretty it is from 0 to 3), or snacks in your kitchen (features: price, sweetness 0–5, healthiness 0–5). Write 8–10 options on cards, each with its 3 numbers.
  2. Collect 20 comparisons. Ask a family member "which do you prefer, this one or this one?" for 20 random pairs. Don't let them see the numbers. On paper, record: A's features, B's features, and 1 if they chose A, 0 if B. Sneak in the same pair twice, far apart, and see if they answer the same way.
  3. Work out their taste. By hand: start with weights 0, 0, 0 and apply the update rule from Step 4 with lr = 0.1 for one pass through your 20 answers (round to 2 decimals). Or with code: type the 20 answers into a list of (feats_a, feats_b, chose_a) and call your train_reward_model.
  4. Test it. Ask 5 brand-new pairs. Before each, predict their choice using your weights. Record how many you got right.
  5. Show them. Tell them their "taste weights". Do they agree? ("I care about price twice as much as sweetness? Hmm, true.")

What to notice: the person never told you a weight, but you found them. Did they answer the repeated pair the same way both times? If not, that's the "a bit random" from Bradley-Terry. And if all your snacks cost the same, what did you learn about price? (Step 6!)

6. Your Robo-Helper so far

Robo-Helper (Lesson 11) works, and the bake-off (Lesson 19) showed which brain understands people best, but it only knows one meaning of "best": shortest. Now it has a reward model, a way to learn it from people (train_reward_model), and best_route, which picks the route people actually like from the 24 candidates. The route planner from Lesson 6 is still doing the heavy lifting (every candidate is built by astar_path); the new piece sits on top and chooses between them.

Next, in Lesson 13, you'll teach the robot a different kind of new trick: adapting a factory-trained model to your robot's slightly different hardware with train_lora, by training only a handful of numbers. And in Lesson 14 you'll think about putting all of this on a real robot, where asking people "A or B?" is how real robot companies tune behaviour that no rule can describe.

7. Common mistakes

8. Check yourself

Why do we ask "A or B?" instead of "score this route out of 10"?

People are much more consistent at comparing than at giving absolute scores. One person's 7 is another's 9, and the same person drifts from day to day. "This one's better" is quick and steady.

Predict: prob_prefers_a([-1, 0, 0], (5, 0, 0), (9, 0, 0)). Roughly what comes out?

Rewards are −5 and −9, gap = 4, sigmoid(4) ≈ 0.98. A is much shorter and this person only cares about length, so they almost always pick A. (The checker wants more than 0.95.)

The model agrees with the person only 84% of the time. Why is that fine here?

Because the person's own true taste agrees with their answers only 84% of the time too. They're a bit random on close pairs. 84% is the ceiling, and the model reached it.

All your training routes are 11 steps long. What length weight do you learn, and why?

Exactly 0.0. Every comparison has a length difference of 0, so the update for that weight is always multiplied by 0. You can only learn about things that vary in your data.

What's the difference between the reward in Lesson 7 and the reward here? And what does real RLHF do that we didn't?

In Lesson 7 we wrote the reward by hand (−1 per step, +10 at the goal). Here the reward is learned from people's comparisons. We then used it with best-of-n (score every option, pick the top). Real RLHF also trains the policy itself with reinforcement learning (PPO or GRPO) to produce high-reward outputs, with a KL penalty that keeps it close to its old behaviour so it can't "hack" the imperfect reward model.

9. Go deeper (optional)