LLMBuild a Robot Brain · Week 5 · Lesson 9

Understanding commands: a word-counting classifier

People say the same thing in a hundred ways. Instead of writing a rule for each one, let the robot learn from examples which words go with which request. Then look hard at where it gets things wrong.

Time2–4 hours
You'll needPython 3, your Lesson 8 tokenize, a notebook, and a few patient people
New wordsintent, slot, classifier, training set, test set, accuracy, smoothing, log
Project pieceUnderstands what you ask

Start here

To understand a command, the robot answers two questions: what do you want (the intent), and which thing (the slot). It learns the first by counting words in examples.

The problem

Robo-Helper can move, plan routes and stop safely. But when you say "get rid of the red block", it has no idea that means the same as "put the red block in the bin". Or that "halt", "freeze" and "wait stop" all mean stop. There are endless ways to say each thing, and you can't write them all down.

The idea in one picture

Imagine sorting your post without opening it. Letters with "invoice" or "payment" go on the bills pile. "Party" and "birthday" go on the invitations pile. You've never seen this exact letter before, but the words give it away. You learned which words go with which pile by seeing lots of letters. Our robot sorts commands into 4 piles the same way: put, go, find, stop.

Words to know

Intent
What the person wants: put, go, find or stop.
Slot
The detail that fills in the intent. Here: which object, like "red block".
Classifier
A program that puts things into one of a fixed set of groups.
Training set
Examples with the right answer, used to learn. Ours has 34.
Test set
Different examples, kept aside, used only to check. Ours has 12. (You met this idea in Lesson 5.)
Accuracy
The fraction of test examples it gets right. 11 out of 12 ≈ 0.92.
Smoothing
Adding 1 to every count so a word you've never seen doesn't wreck the score.
Log
A maths function that turns tiny numbers like 0.000003 into friendly ones like −12.7. Bigger is still better.

Going deeper: everything below is the full lesson: the idea built from nothing, a demo, your homework, and a challenge away from the screen.

1. The problem

Here are some real commands from the training data in lesson09_commands.py, with the intent each one means:

"put the red block in the bin"   put      "go to the red block"          go
"get rid of the red block"       put      "drive over to the blue block" go
"can you bin the red one"        put      "where did the blue one go"    find
"tidy up the green block"        put      "halt"                         stop

Obvious idea: keyword rules. If the command contains "put", it's put. If it contains "go", it's go. If it contains "where", it's find.

Try it on the list. "get rid of the red block" has no "put", so the rule fails. "can you bin the red one" fails too. Worse: "where did the blue one go" contains "go", so if the go rule runs first, the robot drives off when you only asked a question. Every rule you add fixes one sentence and breaks another. And you're always one new phrasing behind.

The better idea: collect examples with the right answers, and let the program learn which words point to which intent.

2. First principles

Step 1: split the job in two

"Put the red block in the bin" has two parts. The intent (put) is the hard bit: many words, many phrasings. The slot (red block) is easy in our room, because there are only three objects and each has a different colour. So for the slot we can use a simple rule: look for "red", "blue" or "green". The rest of the lesson is about the intent.

Step 2: count words per intent

Go through all 34 training examples. Tokenize each one with your Lesson 8 tokenize, and keep a tally of how often each word appears in each intent. Here's part of the real table:

wordputgofindstop
bin4000
to0700
where0050
stop0006
the14880
all words61444815
examples10888

Already you can see the clues: "bin" means put, "where" means find. And "the" is everywhere, so it tells you very little.

Step 3: from counts to chances

Raw counts aren't fair. Put has 61 words in total and stop has only 15, so put would win just by being longer. So we use a fraction: how often does this word appear, out of all the words in that intent? "the" is 14 out of 61 in put (0.23) and 8 out of 44 in go (0.18). That's the same move as Lesson 8: counts divided by a total.

We also include how common each intent is overall (the prior). Put has 10 of the 34 examples, so before reading any words, P(put) = 10 / 34 ≈ 0.29. The others are 8 / 34 ≈ 0.24.

Step 4: multiply the clues

To score a whole command for one intent, multiply the prior by the chance of each word:

score(put) = P(put) × P(word1 in put) × P(word2 in put) × ...

This pretends each word is an independent clue, which isn't really true ("in" and "bin" often come together). That simplification is why it's called Naive Bayes. Naive, but it works surprisingly well.

Step 5: the zero problem, fixed by adding 1

Take the command "stop". In the put examples, "stop" appears 0 times. So P(stop in put) = 0 / 61 = 0, and the whole put score becomes 0. Fine here. But now take "please stop right now". If even one word was never seen with stop, stop's score would become 0 too, however many other clues point to it. One missing word shouldn't destroy everything.

The fix is called add-one smoothing: pretend you saw every word once more than you did. To keep it fair, add the number of different words we know (the vocabulary, 52 words) to the bottom:

P(word in intent) = (count of word in intent + 1) / (all words in intent + 52)

Let's score "stop" by hand, with real numbers:

put:   10/34 × (0 + 1)/(61 + 52)  =  0.294 × 0.00885  =  0.0026
stop:   8/34 × (6 + 1)/(15 + 52)  =  0.235 × 0.1045   =  0.0246    ← about 10× bigger. stop wins.

Step 6: tiny numbers, so use logs

A 7-word command multiplies 8 small numbers. You get something like 0.0000000008. With long sentences, numbers get so small the computer rounds them to exactly 0. The fix is the log (in Python, math.log). You don't need to know how it works inside, just three facts:

The real program prints exactly these: for "stop" the scores are put −5.95, go −6.01, find −6.05, stop −3.71.

Step 7: test it on sentences it never saw

We keep 12 commands in TEST that the model never trains on. It gets 11 right: accuracy ≈ 0.92. The one it misses is "locate the blue block". It says put. Why? "locate" appeared only once in training, while "the", "blue" and "block" all appear a lot in put examples. The scores are put −13.62, find −13.65: it lost by a hair.

Step 8: look at your mistakes (this is the real job)

The program also tries a few extra commands. Here's one:

"where's the green one?"
Predict first: which intent will it pick? "where" is a strong find word…

It picks put. Wrong! And here's the detective work. Our tokenizer from Lesson 8 keeps "where's" as one token. The training data has "where" five times but never "where's". To the model, "where's" is a brand-new word that means nothing. So the only clues left are "the", "green" and "one", and those appear most in put examples:

             prior  where's   the    green   one     total
put   (log)  −1.22   −4.73   −2.02   −3.34  −3.63   −14.94   ← winner
go    (log)  −1.45   −4.56   −2.37   −3.47  −3.47   −15.31
find  (log)  −1.45   −4.61   −2.41   −3.51  −3.51   −15.47
stop  (log)  −1.45   −4.20   −4.20   −4.20  −4.20   −18.27

Type where is the green one instead and it correctly says find (−16.49, clearly ahead). The model wasn't stupid. It was never shown "where's".

This is how real machine-learning engineers spend much of their time: not admiring the score, but reading the failures one by one and asking why. Each failure points to a fix: more training examples, a better tokenizer, or a different model. You'll try one fix in the Stretch.

No "I don't know" button. Our classifier always picks one of its four intents. Type "what is the weather" and it confidently says find. Type "hello robot" and it says stop. It has no way to say "that's not a command I understand". Keep this in mind: in Lesson 11 the full robot runs into exactly this problem.

3. Play with it

Interactive: watch the classifier think

The same TRAIN and TEST lists, tokenizer and scoring formula as lesson09_commands.py. Type a command and see each intent's score (higher, i.e. closer to 0, wins), the word-by-word evidence, and the parsed command. Words in red were never seen in training.

Parsed: –

Test accuracy
–
Vocabulary
–
Training examples
–

4. Homework

Open homework/lesson09_commands.py. TRAIN, TEST and COLOURS are written for you, and it imports tokenize from your Lesson 8, so finish that first. Write six functions, in order.

4.1 train_intent_model(examples)

Count words per intent (Step 2). Return a dictionary with three parts: "word_counts" (intent → {word: count}), "intent_counts" (intent → number of examples), and "vocab" (a set of every word seen).

Example: with TRAIN, model["intent_counts"]["stop"] == 8, model["word_counts"]["put"]["bin"] == 4, and "halt" in model["vocab"].

Hint 1

Make three empty containers first: two dicts and a set. Then loop over for text, intent in examples.

Hint 2

For each example, add 1 to that intent's example count. Then loop over tokenize(text) and, for each word, add 1 to that intent's count for the word and add the word to the vocab set.

Hint 3

bucket = wc.setdefault(intent, {}) gives you the intent's word dict (creating it if needed). Then bucket[w] = bucket.get(w, 0) + 1 and vocab.add(w). Return the three in a dict with the exact key names.

4.2 intent_scores(model, text)

For each intent, compute the log score from Steps 3–6. Return {intent: score}.

score = log(examples of intent / all examples)
      + for each word in text:  log((count of word in intent + 1) / (all words in intent + vocab size))

Example: intent_scores(model, "stop") gives about put −5.95, go −6.01, find −6.05, stop −3.71.

Hint 1

Work out three things before the loop: the total number of examples, the vocab size len(model["vocab"]), and (inside the loop over intents) how many words that intent has in total.

Hint 2

"All words in intent" is sum(that intent's word-count dict.values()). Use .get(w, 0) for words that intent never saw. Don't forget import math is already at the top.

Hint 3

For each intent n: s = math.log(n_examples / total). Then for each w in tokenize(text): s += math.log((count + 1) / (size + V)). Store scores[intent] = s. Check your "stop" numbers against the ones above.

4.3 predict_intent(model, text)

Return the intent with the highest score.

Hint 1

Highest means closest to zero, since they're all negative. max handles that correctly.

Hint 2

max(d, key=d.get) returns the key of a dict with the biggest value.

Hint 3

Two lines: get the scores dict, return the max key.

4.4 extract_object(text)

The slot. Look through the tokens for a word in COLOURS. If you find one, return e.g. "red block". Otherwise return None.

Example: extract_object("grab the Blue one") → "blue block". extract_object("stop") → None.

Hint 1

Use tokenize, so "Blue" becomes "blue" for free.

Hint 2

Return as soon as you find a colour. If the loop finishes with nothing, return None.

Hint 3

for w in tokenize(text): if w in COLOURS: return f"{w} block", then return None after the loop.

4.5 parse_command(model, text)

Put it together: {"intent": ..., "object": ...}. For stop, the object is always None.

Example: parse_command(model, "stop now") → {"intent": "stop", "object": None}.

Hint 1

Call the two functions you already wrote.

Hint 2

"stop the red block" should still have no object: stopping is about the robot, not a block.

Hint 3

Get the intent. Object = None if the intent is "stop", else extract_object(text). Return the dict.

4.6 test_accuracy(model, examples)

The fraction of examples where predict_intent matches the label.

Hint 1

Count the right ones, then divide by how many there are.

Hint 2

In Python, True counts as 1 when you add, so you can sum comparisons.

Hint 3

sum(predict_intent(model, t) == y for t, y in examples) / len(examples), or the same with an ordinary loop and a counter.

Run it

cd homework
python3 lesson09_commands.py

What you should see (real output of the reference solution). Yes, the second line is wrong. That's on purpose: it's the mistake from Step 8.

test accuracy: 0.92
'please put the red block in the bin'      -> {'intent': 'put', 'object': 'red block'}
"where's the green one?"                   -> {'intent': 'put', 'object': 'green block'}
'STOP!'                                    -> {'intent': 'stop', 'object': None}
mistakes: [('locate the blue block', 'find', 'put')]

Check it

cd ..
python3 check.py 9

Five checks. Green ✓ = passed, red ✗ = read the hint, yellow … = not written yet. Two real hints:

Stretch

5. Real-world challenge: collect real commands

Your training set was written by one person. Real people talk differently. Let's find out how differently.

  1. Ask at least 4 people (family, friends) this, one at a time, without showing them the training data: "Imagine a robot in a room with a red, a blue and a green block and a bin. How would you tell it to: (a) put a block in the bin, (b) go to a block, (c) tell you where a block is, (d) stop?" Write down exactly what they say, contractions and all.
  2. That's about 16 new commands, labelled with the right intent. You've built a test set from real users.
  3. Before running anything, go through them on paper. Circle any word that isn't in the model's vocabulary (the demo's red words help). Predict which ones it will get wrong.
  4. Add them as a new list in a copy of your file, say REAL_TEST, and run test_accuracy(model, REAL_TEST). Print the mistakes the same way the file does.
  5. For each mistake, write one line: why did it fail? (Unseen word? Contraction? Too polite? Two intents in one sentence?)

Record: the commands, your predictions, the real accuracy, and your "why" list. What it should teach you: a model scored on test data from the same person who wrote the training data looks better than it really is. Real users find the gaps fast. Is your real accuracy lower than 0.92? Most people's is.

6. Your Robo-Helper so far

Robo-Helper can now understand what you ask. parse_command turns "get rid of the red block" into {"intent": "put", "object": "red block"}, a neat little dictionary the rest of the robot can act on.

It's used twice later on:

7. Common mistakes

8. Check yourself

What's the difference between an intent and a slot?

The intent is what you want (put, go, find, stop). The slot is the detail that fills it in (which block).

Why add 1 to every count?

So a word never seen with an intent doesn't make that intent's score 0 (or log of 0, which crashes). One missing word shouldn't overrule all the other clues.

Predict: what does extract_object("go to the red block, not the blue one") return?

"red block". It returns the first colour it finds. It can't understand "not". Another honest limitation.

Why did "where's the green one?" go wrong?

The tokenizer kept "where's" as one token, which never appeared in training, so the strong "where" clue was lost. The leftover words ("the", "green", "one") lean towards put.

Predict: what does predict_intent(model, "hello robot") return, and why is that a problem?

It returns "stop" (neither word was seen, and stop's small word count makes unseen words least costly there). The problem: the classifier must always pick one of its 4 intents. It has no way to say "I don't know".

9. Go deeper (optional)