LLM + RoboticsBuild a Robot Brain · Week 11 · Lesson 19

The bake-off: every brain, the same 12 tasks

You now have a whole family of brains for Robo-Helper: a word counter, a tiny language model, its attention version, a distilled one and a 4-bit one, plus Claude as a planner or an agent. Which one is best? Today you find out properly: the same room, the same cat, the same 12 tasks, scored on what the robot actually did. Then you weigh up correctness against speed and cost, the way real engineers do.

Time3–4 hours (plus the stretch)

You'll needPython 3, your finished Lessons 9, 15 (with its attention stretch), 16, 17 and 18

New wordsbenchmark, behaviour-level check, scorecard, failure category, trade-off, hybrid system, replay

Project pieceA scorecard: 5 → 8 → 10/12 across the brains you built

Start here

To compare brains fairly, give every one the exact same tasks, then judge the room afterwards, not what the brain says it did.

The problem

Your friend says their brain is "way smarter" than yours. They show you one command where it worked. You show them one where yours worked. Nobody learns anything. And here's a sneaky one: ask the word counter "where's the green one?" and it replies, politely, "The green block is in the bin." That sounds like an answer. But it's true because the robot just drove over and binned it. Nobody asked for that.

The idea in one picture

Think of a baking contest on TV. Every baker gets the same recipe, the same ingredients, the same oven and the same time. The judges don't listen to the bakers explain how lovely their cake is: they cut it open and taste it. And they don't only score taste. A brilliant cake that took nine hours and cost £200 in ingredients might not be the one you'd want to make every Saturday.

Words to know

Benchmark: A fixed set of tasks used to compare different systems fairly. Ours is the 12 tasks in TASKS.
Behaviour-level check: Judging a task by what changed in the world (what's in the bin, where the robot is, whether it asked) instead of by the words it said.
Scorecard: One summary per brain: how many tasks passed, how long it thought (p50 and p95, from Lesson 16), and what it cost.
Failure category: A group of failures with the same cause, such as "can only do one block at a time".
Trade-off: When getting more of one good thing means giving up some of another, like accuracy versus speed.
Hybrid system: A design that uses different brains for different jobs, such as a fast local one for reflexes and a big one for hard thinking.
Replay: A recording of everything the robot did, frame by frame, that you can watch again later.

The full lesson: everything below builds a fair test from scratch, reads the real results, then turns them into a design decision. You write the judge and the scorecard.

1. The problem

In Lesson 11 you tested one brain on 12 commands and checked whether result["ok"] was true and which blocks were binned. Now there are several brains, built in completely different ways. One replies with one action, another writes a whole plan, another calls tools ten times. Some ask questions. They don't even produce the same kind of output.

So what can you compare? Only one thing is the same for all of them: the robot and the room. Every brain drives the same SimRobot, starting at (1, 1), with the cat hiding at (4, 1). So the fair question is: after the brain has finished, is the room in the state the person wanted?

And then a second question that real products always face: if two brains both get it right, but one takes a fraction of a millisecond and runs on your laptop, while the other needs the internet and costs money for every command, which do you ship?

2. First principles

Step 1: judge the room, not the reply

The obvious way to test a brain is to read its reply. Let's try it on the real result for task 2, "where's the green one?", with the word counter:

reply: "The green block is in the bin."

Is that a good answer? You can't tell from the words. It could be a correct "find" answer (if the block was already in the bin) or the result of doing the wrong thing. So look at the world instead. Before: nothing in the bin, robot at (1, 1). After: the green block is in the bin, and the robot is parked at (7, 6). The person asked a question and the robot tidied something up. Fail.

That's a behaviour-level check: compare the room after the run with what a correct robot would leave behind. It works for every kind of brain, because every brain ends up moving the same robot.

Step 2: write down "correct" before you run anything

Each task in TASKS says what a correct robot ends up doing. There are five kinds of check:

Key	Meaning	Why it's needed
`binned`	The bin must hold exactly these blocks.	Every task has it. "Exactly" means binning an extra block fails too.
`still`	The robot must not move at all.	Refusing, stopping and asking all mean "don't drive".
`at`	Where the robot must end up.	"Go to the blue block" bins nothing, so `binned` alone can't tell success from doing nothing.
`says`	Text the reply must contain.	"Where's the green one?" must mention (1, 5).
`asks`	The reply must be a question (end with "?").	"Put the block away", with three blocks, should get "Which one?".

Here are all 12, with their checks:

#	Command	A correct robot…
1	put the red block in the bin	bins only the red block
2	where's the green one?	bins nothing, stays still, says "(1, 5)"
3	make me a sandwich	bins nothing, stays still
4	put the red and blue blocks away	bins red and blue
5	tidy up the room	bins all three
6	bin everything except the green block	bins red and blue
7	put away the block that's closest to the bin	bins only blue (4 squares away, from Lesson 18)
8	put the block away	bins nothing, stays still, asks a question
9	go to the blue block	bins nothing, ends at (6, 3)
10	stop	bins nothing, stays still
11	hmm what about the blue	bins nothing (it's not clear what they want)
12	put the purple block in the bin	bins nothing, stays still (there is no purple block)

Why write these before running? Because once you've seen what a brain did, it's tempting to decide that was fine. Scientists call this fixing the test in advance. The checks are the referee, and the referee can't change the rules at half time.

Step 3: one judge, checks in order

score_task runs the checks in a fixed order and reports the first one that fails. Work through task 5, "tidy up the room", for the tiny LM, by hand. After the run, the robot's bin holds the red and blue blocks, but not the green one:

check 1: binned {'red block', 'blue block'}  ==  wanted {'red block', 'blue block', 'green block'} ?   NO
→ (False, "binned ['blue block', 'red block'], wanted ['blue block', 'green block', 'red block']")

Now task 9, "go to the blue block", word counter. The bin is empty, the robot is at (6, 3):

check 1: binned set()  ==  wanted set() ?    yes
check 2: "still" in task?                    no, skip
check 3: "at" (6, 3)  ==  robot.pos (6, 3) ? yes
check 4: "says"?  check 5: "asks"?           not in this task, skip
→ (True, "ok")

Step 4: predict, then look

Predict first: task 12 is "put the purple block in the bin". There is no purple block. What does the tiny LM do? What does the word counter do?

Tiny LM: it asks "Which block do you mean: red, blue or green?" Pass. Here's why. "purple" isn't a word it has ever seen, so it ignores it. What's left reads like "put the block in the bin", which is one of the training examples where the right answer is ask. It passed by asking, which is the safe thing to do, though it doesn't know purple blocks don't exist.

Word counter: it passes too. It finds no known colour word, so the object is None, and validate_action refuses: "'put' needs an object". The robot stays still. It passed by refusing, not by understanding: it would do the same for "put the thing in the bin".

Step 5: the real results, and what they mean

Here's the real result of python3 lesson19_bakeoff.py, with the five brains you built yourself:

Task	words	tiny LM	attention	distilled	4-bit
1. put the red block in the bin	✓	✓	✓	✓	✓
2. where's the green one?	✗ binned green	✗ wrote just "put"	✗ binned green	✓ "at (1, 5)"	✓ "at (1, 5)"
3. make me a sandwich	✓ (said "Stopping.")	✓ refused	✓ refused	✓ refused	✓ refused
4. put the red and blue blocks away	✗ only red	✓	✓	✓	✓
5. tidy up the room	✗ refused	✗ red and blue only	✗ asked	✗ green only	✗ green only
6. bin everything except the green block	✗ binned green!	✓	✓	✓	✓
7. put away the block that's closest to the bin	✗ refused	✗ said where red is	✗ drove to blue	✗ asked	✗ asked
8. put the block away	✗ refused, didn't ask	✓ asked	✓ asked	✓ asked	✓ asked
9. go to the blue block	✓	✓	✓	✓	✓
10. stop	✓	✓	✓	✓	✓
11. hmm what about the blue	✗ binned blue	✗ binned blue	✗ binned blue	✓ said where blue is	✓ said where blue is
12. put the purple block in the bin	✓ refused	✓ asked	✓ asked	✓ asked	✓ asked
Passed	5/12	8/12	8/12	10/12	10/12

Each brain you built made the robot better, and you can see exactly where. A score alone tells you little. Group the failures by cause (failure categories), and the picture gets sharp:

One action only (tasks 4, 6, and 8's question). The word counter outputs exactly one {"intent", "object"}, so "two blocks", "everything except" and "which one?" can't even be expressed. Lesson 15 fixed this by changing the output format: the tiny LM writes a whole plan, or ask.
Hears the wrong intent (tasks 2, 11). "where's" and "what about" weren't in the templates the tiny LM and the attention model learned from, so they guess "put" (or, for the tiny LM on task 2, write a plan that makes no sense). Lesson 17's distillation fixed this: the teacher's labels included "where's the red one" and friends, and the distilled brain now answers instead of binning.
An unseen word (task 5). "room" is a word none of the small brains has ever seen. They ignore it, and what's left, "tidy up the", is vague. The attention model asks which block. The others guess: the tiny LM bins red and blue, and the distilled and 4-bit brains bin only green.
No idea where things are (task 7). "Closest" needs the room. No language-only model ever sees robot.look(), so no amount of training on words can fix it. That's what week 10's Claude, with the whole room in its prompt, is for.

Notice: more (and better) training examples fixed "hears the wrong intent". They could never have fixed "one action only": that needed a new output format. And task 7 needs a brain that can see the room. Knowing which kind of failure you have tells you what to do next. Squeezing the distilled brain to 4 bits didn't cost a single task.

Step 6: the scorecard (correct, fast, cheap)

scorecard squeezes one brain's 12 rows into five numbers, using Lesson 16's summarize (p50 and p95) and cost_usd. Practise on the small example from check.py: three tasks, thinking times 1, 2 and 3 seconds, 1,000 input tokens each, no output tokens, and the first two passed.

passed = 2, total = 3
p50: sorted [1, 2, 3], position ceil(0.50 × 3) − 1 = 2 − 1 = 1  → 2.0 s
p95: position ceil(0.95 × 3) − 1 = 3 − 1 = 2                    → 3.0 s
cost: 3,000 / 1,000,000 × $4 + 0 × $20                          = $0.012

Now the real scorecard from our run. Times change every run and depend on your computer. They're bigger than in Lesson 16 because each time covers the whole act() call, including the typo fixer, the safety checks and driving the robot, not just the model. The word counter's p95 is the biggest here, which is a reminder that 12 timings make a jumpy p95.

Brain	Passed	Thinking p50	Thinking p95	Cost for all 12	Needs internet?
words (Lesson 9)	5/12	0.22 ms	2.93 ms	$0.000	no
tiny LM (Lesson 15)	8/12	0.51 ms	0.74 ms	$0.000	no
attention (Lesson 15 stretch)	8/12	0.50 ms	0.74 ms	$0.000	no
distilled (Lesson 17)	10/12	0.50 ms	1.05 ms	$0.000	no
tinylm-4bit (Lesson 17)	10/12	0.39 ms	0.67 ms	$0.000	no
Claude planner	from your run of `planner` or `planner-replay`				yes (or a recording)
Claude agent	from your run of `agent` or `agent-replay`				yes (or a recording)

Remember Lesson 16's thinking distance: at 1 metre per second, a robot that keeps moving while its brain thinks travels 1 metre for every second of thinking. Our local brains think for a few thousandths of a second at most. A world-class model has to send your request across the internet and run a huge network on it, so expect its column to be measured in a very different unit. Fill in the real numbers from your own run, or from the recordings.

Predict first: a brain passes 12/12 but its p95 thinking time is several seconds. Would you let it handle "stop"?

No. "Stop" is exactly the command where waiting is dangerous. If the robot keeps rolling while the request travels to a data centre and back, the correct answer arrives too late. Correctness and speed are separate scores, and for some tasks speed is part of being correct.

Step 7: the real-world answer is usually "both"

So which brain wins? Look at what each one is good at. The local brains are instant, free, and work with the Wi-Fi off, but they understand very little. A world-class model understands nearly everything but is slow, costs money per command, and disappears when the internet does. Real robots, cars and phones don't choose one. They use a hybrid:

always, on the robot (reflexes, never wait for the internet):
    stop-neuron + A* replanning around surprises     (Lessons 4, 6, 11)
    "stop" / "halt" / "freeze"  → stop NOW            (word counter, ~0.01 ms in Lesson 16)

a command arrives:
    simple and the local brain is sure?   → do it locally
    otherwise                             → ask the world-class planner
                                            → validate_plan (Lesson 18)  → act
    no internet, or no answer in time?    → "Sorry, I didn't catch that." (don't guess)

Your body works the same way. Touch a hot pan and your hand jerks back before your brain has even noticed: a fast, simple reflex. Deciding what to cook for dinner takes the slow, clever part. And Lesson 17 showed one more trick: distil a teacher's answers into a small local brain, so more commands can be handled fast and free over time. With just 24 careful labels it took the small brain from 8/12 to 10/12.

3. Play with it

Interactive: the replay viewer

Every brain's real run of the bake-off, frame by frame, side by side. The data below is the real traces/bakeoff.json from python3 lesson19_bakeoff.py (the five brains you built: words, tiny LM, attention, distilled and 4-bit), embedded in this page. Pick a task (or click a row of the scoreboard), then drag the slider or press play. The orange circle is the cat at (4, 1); it gets a red outline once a robot has seen it. Load your own traces/bakeoff.json to watch every brain you ran, including Claude ones: the viewer shows as many brains as the file has.

Task

Frame 0

Scoreboard

Load your own traces/bakeoff.json:

Try: task 6 ("except") and watch the word counter carefully put away the one block it was told to leave, while the other four bin red and blue. Task 2: two robots drive off and bin the green block; the distilled and 4-bit robots stay put and say where it is. Task 9: every robot meets the cat and drives round it. Then run the bake-off yourself and load your file.

4. Homework

Open project/homework/lesson19_bakeoff.py. TASKS, run_bakeoff and print_table are written for you. Read run_bakeoff: every brain gets a fresh robot for every task, with the same surprise. Then write the judge and the scorecard.

4.1 `score_task(task, robot, result)`

Did the robot do the right thing? Return (passed, reason). Check in this order, and return at the first failure:

robot.binned == task["binned"], else (False, "binned X, wanted Y").
If task.get("still"): the robot must be at ROBOT_START, else (False, "moved but shouldn't have").
If "at" is in the task: the robot must be at task["at"], else (False, "ended at P, wanted Q").
If "says" is in the task: that text must be inside result["reply"].
If task.get("asks"): the reply, stripped of spaces, must end with "?".

All passed → (True, "ok"). Examples from the real run: word counter on task 5 → (False, "binned nothing, wanted ['blue block', 'green block', 'red block']"); on task 8 → (False, "should ask a question").

Hint 1

Five ifs, each with its own return False, ..., then return True, "ok" at the end. The order matters: it decides which reason you report when several things are wrong.

Hint 2

robot.pos might be a list or a tuple, so compare tuple(robot.pos) with ROBOT_START. To print the sets nicely and the same way every time, use sorted(robot.binned) or 'nothing': an empty list is falsy, so or swaps in the word. Some replies may be missing: result.get("reply", "").

Hint 3

reply = result.get("reply", "")
binned differs                       → False, f"binned {sorted(...) or 'nothing'}, wanted {sorted(...) or 'nothing'}"
task.get("still") and pos != start   → False, "moved but shouldn't have"
"at" in task and pos != task["at"]   → False, f"ended at {robot.pos}, wanted {task['at']}"
"says" in task and not in reply      → False, f"reply missing {task['says']!r}"
task.get("asks") and not reply.strip().endswith("?") → False, "should ask a question"
→ True, "ok"

4.2 `scorecard(rows)`

rows is one brain's list of {"passed", "seconds", "tokens_in", "tokens_out"}. Return {"passed", "total", "p50", "p95", "cost"}. Example: the check.py rows from Step 6 → {"passed": 2, "total": 3, "p50": 2.0, "p95": 3.0, "cost": 0.012}.

Hint 1

You already wrote the hard parts in Lesson 16. They're imported at the top: summarize and cost_usd.

Hint 2

summarize wants a list of seconds: build it with a list comprehension. True counts as 1 in Python, so sum(r["passed"] for r in rows) counts the passes. Add up all the input tokens and all the output tokens first, then call cost_usd once.

Hint 3

s = summarize([seconds of each row])
return {"passed": number of rows that passed, "total": len(rows),
        "p50": s["p50"], "p95": s["p95"],
        "cost": cost_usd(total tokens_in, total tokens_out)}

Run it

cd project/homework
python3 lesson19_bakeoff.py

What you should see (real output from the solution; your times will differ a little):

  skipping planner (Claude, recorded): (no recording for this command)
  skipping agent (Claude, recorded): (no recording for this command)

task                                                     words          tinylm       attention       distilled     tinylm-4bit
1. put the red block in the bin                              ✓               ✓               ✓               ✓               ✓
2. where's the green one?                                    ✗               ✗               ✗               ✓               ✓
3. make me a sandwich                                        ✓               ✓               ✓               ✓               ✓
4. put the red and blue blocks away                          ✗               ✓               ✓               ✓               ✓
5. tidy up the room                                          ✗               ✗               ✗               ✗               ✗
6. bin everything except the green block                     ✗               ✓               ✓               ✓               ✓
7. put away the block that's closest to the bin               ✗               ✗               ✗               ✗               ✗
8. put the block away                                        ✗               ✓               ✓               ✓               ✓
9. go to the blue block                                      ✓               ✓               ✓               ✓               ✓
10. stop                                                     ✓               ✓               ✓               ✓               ✓
11. hmm what about the blue                                  ✗               ✗               ✗               ✓               ✓
12. put the purple block in the bin                          ✓               ✓               ✓               ✓               ✓
PASSED                                                    5/12            8/12            8/12           10/12           10/12
thinking time p50                                      0.22 ms         0.51 ms         0.50 ms         0.50 ms         0.39 ms
thinking time p95                                      2.93 ms         0.74 ms         0.74 ms         1.05 ms         0.67 ms
cost for all 12                                         $0.000          $0.000          $0.000          $0.000          $0.000

Traces saved to /.../project/homework/traces/bakeoff.json      (your full path; we shortened it)
  words    task  2: binned ['green block'], wanted nothing   (reply: 'The green block is in the bin.')
  words    task  4: binned ['red block'], wanted ['blue block', 'red block']   (reply: 'The red block is in the bin.')
  words    task  5: binned nothing, wanted ['blue block', 'green block', 'red block']   (reply: "Sorry, I can't do that: 'put' needs an object")
  words    task  6: binned ['green block'], wanted ['blue block', 'red block']   (reply: 'The green block is in the bin.')
  words    task  7: binned nothing, wanted ['blue block']   (reply: "Sorry, I can't do that: 'put' needs an object")
  words    task  8: should ask a question   (reply: "Sorry, I can't do that: 'put' needs an object")
  words    task 11: binned ['blue block'], wanted nothing   (reply: 'The blue block is in the bin.')
  tinylm   task  2: reply missing '(1, 5)'   (reply: 'Sorry, my answer made no sense (put).')
  tinylm   task  5: binned ['blue block', 'red block'], wanted ['blue block', 'green block', 'red block']   (reply: 'The red block is in the bin. The blue block is in the bin.')
  tinylm   task  7: binned nothing, wanted ['blue block']   (reply: 'The red block is at (2, 1).')
  tinylm   task 11: binned ['blue block'], wanted nothing   (reply: 'The blue block is in the bin.')
  attention task  2: binned ['green block'], wanted nothing   (reply: 'The green block is in the bin.')
  attention task  5: binned nothing, wanted ['blue block', 'green block', 'red block']   (reply: 'Which block do you mean: red, blue or green?')
  attention task  7: binned nothing, wanted ['blue block']   (reply: "I'm next to the blue block.")
  attention task 11: binned ['blue block'], wanted nothing   (reply: 'The blue block is in the bin.')
  distilled task  5: binned ['green block'], wanted ['blue block', 'green block', 'red block']   (reply: 'The green block is in the bin.')
  distilled task  7: binned nothing, wanted ['blue block']   (reply: 'Which block do you mean: red, blue or green?')
  tinylm-4bit task  5: binned ['green block'], wanted ['blue block', 'green block', 'red block']   (reply: 'The green block is in the bin.')
  tinylm-4bit task  7: binned nothing, wanted ['blue block']   (reply: 'Which block do you mean: red, blue or green?')

With no names, python3 lesson19_bakeoff.py tries seven brains: your five, plus planner-replay and agent-replay. If the Claude recordings aren't in recordings/ yet, you'll see skipping planner (Claude, recorded): (no recording for this command) and the same for the agent (at the top and bottom of the output above), and the table shows just your five. To run only some brains, name them: python3 lesson19_bakeoff.py words distilled. With a key, add planner and agent to the list to run Claude live.

To watch one brain do one task in the terminal: python3 lesson19_bakeoff.py --watch tinylm 5. It ends with:

[tinylm (Lesson 15)] 'tidy up the room' -> 'The red block is in the bin. The blue block is in the bin.'   (False, "binned ['blue block', 'red block'], wanted ['blue block', 'green block', 'red block']")

Then load traces/bakeoff.json into the replay viewer above.

Check it

cd project
python3 check.py 19

Four checks: score_task: binned, score_task: still / at, score_task: says / asks and scorecard. A red ✗ gives a hint like "an EXTRA block binned -> fail" or "tokens 3,000 in and 0 out cost $0.012". Yellow … means a function isn't written yet, or Lesson 16 or 18 isn't finished.

Stretch: build the hybrid brain

Write a HybridBrain class in a new file (copy the shape of WordsBrain in brains.py: a name and an act(robot, command) method). If the word counter's top intent is "stop", or it wins by a big margin (use intent_scores from Lesson 9, like the Lesson 11 stretch), let WordsBrain handle it. Otherwise hand the command to get_brain("planner-replay") (or "planner" with a key). Score it with run_bakeoff by adding it to get_brain in your copy. Does it keep "stop" instant and get the hard tasks right? Then add three tasks of your own to TASKS, with checks written before you run them.

Stretch 2: an AI model as the judge

Our bake-off checks are exact: we look in the bin, at where the robot ended up, at whether it asked a question. That only works because our world is tiny. Now imagine grading an essay, a chat reply or a summary. There's no bin to look in. So companies often ask another AI model to grade the answer. That's called LLM-as-a-judge. Judges are useful, and also dangerous. This stretch shows you both.

Rule 1: measure the judge against checks you trust

A judge is just another model that can be wrong. Before you trust it, run it on tasks where you already know the right verdict, and count how often it agrees. Here we're lucky: our exact checks are the trusted answers. The judge's agreement is the fraction of tasks where its pass/fail matches the exact check. The tasks where they disagree are the interesting ones.

A judge that only reads the reply

Start with the cheapest judge possible, rule_judge (provided). It reads only the reply and says "pass" unless the reply sounds like a refusal ("sorry", "can't", …). Real output of python3 lesson19_judge.py (offline, no key needed). It judges three of your brains: words, the tiny LM and the distilled one:

words (Lesson 9): exact checks pass 5/12
  rule judge, reply only       agrees  58%  kappa  0.21  caught 3/7 failures   disagrees on tasks [2, 4, 6, 11, 12]
  (lazy judge: always pass)    agrees  42%  kappa  0.00

tinylm (Lesson 15): exact checks pass 8/12
  rule judge, reply only       agrees  67%  kappa  0.14  caught 1/4 failures   disagrees on tasks [3, 5, 7, 11]
  (lazy judge: always pass)    agrees  67%  kappa  0.00

distilled (Lesson 17): exact checks pass 10/12
  rule judge, reply only       agrees  75%  kappa -0.12  caught 0/2 failures   disagrees on tasks [3, 5, 7]
  (lazy judge: always pass)    agrees  83%  kappa  0.00

Look at the tiny LM's task 11, "hmm what about the blue". Nobody asked for anything to be binned. The tiny LM binned the blue block and said:

reply: "The blue block is in the bin."

Confident, friendly, no "sorry": the judge says pass. The exact check says fail. That's the same trap as Step 1, now inside a judge: a judge that only reads the reply is fooled by a confident robot. It's fooled the same way on task 5 ("The red block is in the bin. The blue block is in the bin.", one block short) and task 7. It only caught one of the tiny LM's four failures, task 2, because that reply happened to start with "Sorry". It also goes wrong the other way: on task 3, "make me a sandwich", every brain refused correctly and stayed still, and the judge failed them for saying "sorry". Same for the word counter's correct refusal on task 12.

Now the distilled brain. Its two failures are task 5, where it binned only the green block and said "The green block is in the bin.", and task 7, where it asked "Which block do you mean: red, blue or green?". A confident report and a polite question both sound perfectly reasonable, so the judge caught neither.

Rule 2: compare with a lazy judge (kappa)

Agreement on its own can fool you. Imagine a robot that passes 9 tasks out of 10. A lazy judge that says "pass" to everything agrees 90% of the time, and it has never caught a single mistake. So always ask two more questions:

Catch rate: of the runs that really failed, how many did the judge fail? The rule judge caught 3 of 7 on the word counter, 1 of 4 on the tiny LM and 0 of 2 on the distilled brain.
Kappa (Cohen's kappa): how much better the judge agrees than a guesser with the same pass rate would by luck. 0 means no better than guessing, 1 means perfect. Below 0 means worse than guessing. The usual bar before you let a judge make decisions is about 0.6. Our rule judge scores 0.21, 0.14 and −0.12.

The distilled brain shows the trap at its worst. The rule judge agrees with the exact checks on 75% of its tasks, which sounds good. But a lazy judge that says "pass" to everything agrees on 83%, because the brain passes 10 of 12. The better your robot gets, the easier it is for a useless judge to look good, and only kappa gives it away.

Module 19's article found exactly this with a real AI judge on 60 support replies: 60% agreement sounded fine, but "always pass" would have scored 57%, and kappa was only 0.15 (Module 19).

Give the judge the evidence

The fix is the one from Step 1: show the judge what actually happened in the room, not just what the robot said. Your evidence_text turns a run's recorded frames into one sentence, and judge_prompt adds it to the prompt. Here's the real example the file prints:

A person told a cleaning robot: "where's the green one?"
The robot replied: 'The green block is in the bin.'
What actually happened: Robot started at (1, 1) and ended at (7, 6). In the bin at the end: green block. It moved: yes.
Did the robot do the right thing? A correct robot does exactly what was asked, refuses impossible or unrelated requests without moving, and asks a question when the request is ambiguous.

Predict first: you're the judge. Pass or fail, with and without the "What actually happened" line?

Without it, "The green block is in the bin." is a perfectly good answer to "where's the green one?". With it, you can see the robot drove off and binned the block when it was only asked a question: fail. Same reply, different evidence, different verdict.

Asking Claude to judge

ask_judge (provided) sends the prompt to Claude with a short system prompt ("a strict, fair judge") and asks for structured output: JSON that must match a schema with exactly two fields, verdict ("pass" or "fail") and reason. Forcing a schema means your code can read the verdict without guessing what a sentence meant. With an API key, python3 lesson19_judge.py record asks Claude to judge every run twice, once reply-only and once with the evidence, and saves the verdicts to recordings/judge.json. After that, the plain python3 lesson19_judge.py adds two more agreement lines per brain.

Whatever the numbers turn out to be, the question to ask is the same: how much does the evidence line change the agreement, and on which tasks does even the evidence-reading judge still disagree with the exact checks?

Rule 3: every score gets an error bar

The word counter scored 5/12, the tiny LM 8/12 and the distilled brain 10/12. Is each one really better than the last? Twelve tasks is very few. If we'd picked a slightly different 12, the scores could easily have come out differently. A bootstrap tells you how much: pretend your 12 results are a bag, draw 12 out at random (putting each one back), score that pretend test, and do it 2,000 times. The middle 95% of those pretend scores is your error bar. Real output:

Honest numbers: each score with a 95% error bar (bootstrap over the 12 tasks)
  words (Lesson 9)              5/12   somewhere between  17% and  67%
  tinylm (Lesson 15)            8/12   somewhere between  42% and  92%
  distilled (Lesson 17)        10/12   somewhere between  58% and 100%
  paired: tinylm minus words = +25% [+0%, +50%]  (they differ on 3 task(s); too close to call)
  paired: distilled minus words = +42% [+17%, +67%]  (they differ on 5 task(s); a real difference)

The error bars are huge, and they overlap a lot. The paired lines compare two brains task by task, which is fairer, because both sat the exact same tasks. The tiny LM beat the word counter on 3 tasks, but its interval touches zero (+0%), so with only 12 tasks we can't be sure it's a real win: "too close to call". The distilled brain differs from the word counter on 5 tasks, all in its favour, and its whole interval is above zero (+17% to +67%): a real difference, even with 12 tasks. To tell close brains apart reliably, you need many more tasks. That's why real evals use hundreds.

Judge pitfalls

Agreeing with confident text. As above. Give the judge evidence wherever you can.
Position bias. When a judge compares two answers, it often prefers whichever comes first (or second). Ask twice with the order swapped.
Length bias. Judges often prefer longer, more detailed answers, even when the short one is right.
A judge is a model too. It can misunderstand the task, hallucinate a reason, or change its mind on a re-run. That's why you measure it against trusted checks first, and keep checking.

Interactive: what the judge sees

The same real runs as the replay viewer above (the data embedded in this page). Pick a brain and a task, and choose whether the judge gets the evidence. You'll see the exact prompt judge_prompt builds, rule_judge's verdict on the reply, and the exact check. evidence_text, judge_prompt, rule_judge and agreement are written in JavaScript exactly like the Python.

Brain Task

Show the judge the evidence

Rule judge (reply only)

–

Exact check

–

Rule judge agrees, this brain

–

Kappa, this brain

–

Notice the rule judge's verdict never changes when you tick the box: it can't use the evidence, because it only reads the reply. A model judge can. Try tiny LM task 11, word counter task 2 and distilled task 7 with the box ticked, and decide for yourself what a fair judge should say.

Your homework for this stretch

Open homework/lesson19_judge.py. It uses your finished lesson19_bakeoff.py. You write six small functions; rule_judge, ask_judge and the recording code are provided.

evidence_text(frames): one sentence from the first and last frames: "Robot started at (1, 1) and ended at (x, y). In the bin at the end: A, B. It moved: yes." Positions as tuples, the bin's blocks sorted and joined with ", " (or nothing), and "It moved" is yes if any frame's position differs from the start.

Hint 1

frames[0] is the start and frames[-1] is the end. Each has "pos" (a list like [1, 1]) and "binned" (a list of names).

Hint 2

tuple([1, 1]) prints as (1, 1). ", ".join(sorted(...)) or "nothing" handles an empty bin, because an empty string counts as false.

Hint 3

Moved: any(tuple(f["pos"]) != start for f in frames). A robot that drives off and comes back to (1, 1) still moved. Then one f-string.

judge_prompt(command, reply, evidence=None): lines joined with "\n": the command, the reply, a "What actually happened: " line only if evidence is given, and the question from the docstring last. Match the example prompt above exactly (it uses {command!r} and {reply!r}, which is why the quotes differ).

Hint 1

Build a list of lines, append the evidence line inside an if evidence:, then append the question.

Hint 2

!r in an f-string prints a string with its quotes, like repr(). Python picks double quotes when the text contains an apostrophe, as in "where's".

agreement(exact, judged): return (fraction that agree, indices where they disagree). Example: agreement([True, False, True, True], [True, True, True, False]) → (0.5, [1, 3]).

Hint 1

enumerate(zip(exact, judged)) gives you i, (a, b). Collect the is where a != b.

Hint 2

The fraction that agree is 1 - len(diff) / len(exact).

kappa(exact, judged): observed is the fraction that agree. chance = p_exact * p_judged + (1 - p_exact) * (1 - p_judged), where each p is the fraction marked pass. Return (observed - chance) / (1 - chance), or 0.0 if chance is 1.

Hint 1

sum([True, False, True]) / 3 is the fraction of True. Python counts True as 1.

Hint 2

Why that formula for chance? Two guessers who say pass at rates p_exact and p_judged agree on pass with probability p_exact * p_judged and on fail with probability (1 - p_exact) * (1 - p_judged).

catch_rate(exact, judged): return (caught, total_bad): the number of runs where the exact check failed, and how many of those the judge also failed.

Hint 1

Keep only the judge's verdicts where exact is False, then count how many of those are False too.

bootstrap_ci(results, rounds=2000, seed=0): make rng = random.Random(seed). In each round, pick len(results) items with rng.choice(results) and take their average. Sort the averages and return the ones at positions int(0.025 * rounds) and int(0.975 * rounds) - 1.

Hint 1

One round: sum(rng.choice(results) for _ in results) / len(results). Do it inside a list comprehension over range(rounds), then sorted(...).

Hint 2

It works for differences too: [1, 0, 0, -1, ...] (brain A minus brain B on each task) gives the paired error bar. That's how the "paired" line is made.

Check with python3 check.py 19a: six checks. Real hints: "no evidence given -> no evidence line", "a judge that always says pass gets kappa 0, even though it agrees 75% of the time" and "with only 12 tasks the error bar should be wide (more than 30 points)".

5. Real-world challenge: the family tidy-up bake-off

Run a real bake-off with people as the brains. You need 3 small objects (say a sock, a pen and a toy), a box as the bin, index cards, a stopwatch (a phone is fine) and two or three "contestants".

Write 8 task cards, like TASKS: a simple one ("put the sock in the box"), two objects, "everything except the pen", "the one closest to the box", an ambiguous one with two similar objects ("put the pen away" when there are two pens), an impossible one ("put the sofa in the box"), a question ("where's the toy?") and "stop".
On the back of each card, write the pass rule first: exactly what's in the box afterwards, whether they should move at all, and whether they should ask a question. Do this before anyone plays.
Contestants. One normal human. One "word-counter human", who must follow a strict rule: act on the first object word they hear, and never ask questions. Optionally a third, who must write down a full plan before moving (a planner).
Run every card on every contestant, resetting the room each time (fresh robot!). Time from the end of the instruction to their first movement: that's thinking time.
Judge the room, not the words. Use only the rule on the back of the card. Record ✓/✗, the reason, and the time, in a table like the one in Step 5.
Work out the scorecard: passes out of 8, and the median thinking time (sort the times, take the middle).

What it teaches: the strict word-counter human is the fastest and fails exactly the tasks our word counter failed. Normal humans are slower but ask "which pen?". Writing the rules first stops you being generous to the contestant you like. And you'll notice how often you want to judge by what someone says instead of what they did.

6. Your Robo-Helper so far

What this adds to your robot. Proof, not opinions. Row 19 of python3 progress.py --solutions measured: "the same 12 tasks for every brain: words 5/12, tinylm 8/12, distilled 10/12". And the paired bootstrap says the jump from the word counter to the distilled brain (+42%, between +17% and +67%) is a real difference, not luck.

Robo-Helper now has a scorecard and replays of every brain. score_task judges any brain by its behaviour, scorecard adds speed and cost, and run_bakeoff saves every frame to traces/bakeoff.json for the replay viewer.

Next week, Lesson 12 asks a harder question that the bake-off can't: when several answers are all "correct", which one do people actually prefer? Then in Lesson 14 your bake-off table and a replay become the centrepiece of your project README and demo.

7. Common mistakes

Checking "the wanted blocks are in the bin" instead of "exactly these blocks". Then a robot that bins everything passes "put the red block in the bin". check.py: "an EXTRA block binned -> fail".
Comparing a list with a tuple. [1, 1] == (1, 1) is False in Python. Use tuple(robot.pos). Otherwise a robot that stayed still "moved". check.py: "stayed still -> pass".
Forgetting to strip. "Which block? " ends with a space, not "?". check.py: "a question (ending in '?') -> pass".
Pricing each row and adding the costs is fine, but pricing only the first row, or mixing up input and output prices, isn't. check.py: "tokens 3,000 in and 0 out cost $0.012".
Judging by the reply. "The green block is in the bin." is a lovely sentence and a failed task. Look at the room.
Trusting a judge you haven't measured (Stretch 2). On the distilled brain, a reply-only judge agreed with the exact checks 75% of the time, less than the 83% a lazy always-pass judge gets (kappa −0.12). Measure any judge with agreement first, and give it the evidence. check.py: "with evidence -> a line 'What actually happened: ...'".

8. Check yourself

Why does every brain get a fresh robot for every task?

So every run starts from the same room. If task 1 binned the red block, task 4 would start with a different room for one brain than another, and the comparison would no longer be fair.

Predict: a brain answers "stop" to every single command. What does it score?

It never moves and bins nothing, so it passes every task whose only rules are "bins nothing" and "stays still": 3 (sandwich), 10 (stop), 11 (hmm…) and 12 (purple). That's 4/12, only one behind the word counter! It fails 2 (no "(1, 5)") and 8 (no question). A score only means something next to the tasks behind it.

Why are "one action only" failures worse news than "heard the wrong intent" failures?

More training data can fix wrong intents. But a brain that can only output one action can never do "the red and blue blocks", however well it's trained. You'd have to change its output format, like the planner's list of actions.

For "where's the green one?", the attention model wrote put green, and the action passed Lesson 10's safety check. Was the safety check broken?

No. It did its job: it checks that an action is allowed and about a real block. "Put the green block in the bin" is both. Safety checks catch impossible and dangerous actions; only an evaluation like the bake-off catches allowed-but-wrong ones.

Give one reason a real robot would keep a fast local brain even if a world-class model passed every task.

Reflexes like "stop" can't wait for the internet; the robot must still work when the Wi-Fi drops; and every request to a big model costs money and time.

9. Go deeper (optional)

Decision-model evals: the cost of mistakes: why some errors matter more than others when you score a system.
Benchmarks: how engineers compare model serving setups fairly.
Latency: p50, p95 and where the waiting time actually goes.
Scale and cost: what it costs to answer millions of requests.
Distilling small models: making a small, fast brain from a big one, the hybrid's secret weapon.

The bake-off: every brain, the same 12 tasks

Start here

The problem

The idea in one picture

Words to know

1. The problem

2. First principles

Step 1: judge the room, not the reply

Step 2: write down "correct" before you run anything

Step 3: one judge, checks in order

Step 4: predict, then look

Step 5: the real results, and what they mean

Step 6: the scorecard (correct, fast, cheap)

Step 7: the real-world answer is usually "both"

3. Play with it

Interactive: the replay viewer

Scoreboard

4. Homework

4.1 score_task(task, robot, result)

4.2 scorecard(rows)

Run it

Check it

Stretch: build the hybrid brain

Stretch 2: an AI model as the judge

Rule 1: measure the judge against checks you trust

A judge that only reads the reply

Rule 2: compare with a lazy judge (kappa)

Give the judge the evidence

Asking Claude to judge

Rule 3: every score gets an error bar

Judge pitfalls

Interactive: what the judge sees

Your homework for this stretch

5. Real-world challenge: the family tidy-up bake-off

6. Your Robo-Helper so far

7. Common mistakes

8. Check yourself

9. Go deeper (optional)

4.1 `score_task(task, robot, result)`

4.2 `scorecard(rows)`