Faster and cheaper: caching, quantising and distilling

Last lesson you put a stopwatch and a price tag on every brain. Big brains are smart but slow and pricey; small brains are fast and free but make more mistakes. Every AI company in the world uses three tricks to get closer to "smart and fast". Today you build all three, and find the catch in each one.

Going deeper: everything below is the full lesson: three tricks built from nothing, a demo where you squash the attention model's real numbers with a bits slider, your homework, and a challenge with a tape measure.

1. The problem

Lesson 16 gave us the numbers that matter: latency (p50 and p95), and cost per token. Look at where the time and money go:

• Repeated work. "stop", "STOP", "Stop!": three strings, one meaning, three times the thinking. For an internet brain, that's three waits and three bills.
• Fat numbers. Our best small brain so far, Lesson 15's attention model, has 2,702 numbers, each stored as a 32-bit decimal (4 bytes): 10,808 bytes. That's nothing. But a big open LLM can have 70 billion numbers: 280 billion bytes (280 GB), more than fits on any single graphics card. And to write each token, the computer has to read through all of those numbers. Fewer bytes means it fits on smaller machines and runs faster.
• A giant brain for easy jobs. "go to the red block" doesn't need a model that can write poetry. But "where's the green one?" does trip up our small one: it writes put green and bins the block.

The obvious fixes all have a trap: remember answers (but they go out of date), use rougher numbers (but the brain might get dumber), copy a big brain (but you copy its mistakes too). Let's build each fix carefully.

2. First principles

Part A · Caching

Step 1: a cache is a dictionary

You already know the tool: a Python dictionary. The question goes in as the key, the answer is the value.

ask "stop"  → not in the dictionary   → MISS: think (slow), store {"stop": answer}
ask "stop"  → in the dictionary       → HIT: return the stored answer (instant, free)

A hit costs nothing: no thinking, no tokens, no waiting. The hit rate (hits ÷ questions) tells you how much the cache is saving.

Step 2: choosing the key

If the key is the exact text, then "stop", "STOP" and "Stop!" are three different keys, and the cache misses three times. So we tidy the text first with Lesson 8's tokenize: lower-case words, punctuation dropped (except full stops).

tokenize("Put the RED block!")    → ["put", "the", "red", "block"]
joined with spaces                → "put the red block"

Now "Put the RED block!" and "put the red block" share a key. That's half the key.

Step 3: the stale-answer bug

Robo-Helper is asked "where is the red block". It thinks, and answers "at (2, 1)". The cache stores that. Then someone puts the red block in the bin. Five minutes later: "where is the red block". The text matches. Hit! The cache instantly answers "at (2, 1)", and the robot drives confidently to an empty square.

That's a real bug, and real systems have it. The answer depends on the question and on the state of the world. So the world has to be part of the key:

cache_key("Put the RED block!", "w1")  →  "put the red block|w1"
                                             └ the words ┘ └ the world

Same words in a different world: different key, so a miss, so the brain thinks again with fresh information. The | is just a separator so the two halves can't blur together.

Here's what lesson17_faster.py asks, all in the world "red at (2,1)" except the last one. Work down the list and fill in hit or miss:

#  question                          world           key so far?   result
1  put the red block in the bin      red at (2,1)    new           miss (think)
2  Put the RED block in the bin!     red at (2,1)    same as 1     hit
3  where is the blue block           red at (2,1)    new           miss (think)
4  where is the blue block?          red at (2,1)    same as 3     hit
5  stop                              red at (2,1)    new           miss (think)
6  STOP                              red at (2,1)    same as 5     hit
7  where is the red block            red at (2,1)    new           miss (think)
8  where is the red block            red in the bin  new world!    miss (think)

Step 4: how real companies cache

Remember from Lesson 16 why a short command costs about 1,000 input tokens: the instructions and the room description are re-sent every time. LLM providers offer prompt caching: the server remembers its work on the start of your prompt, and when the next request starts with exactly the same text, it skips re-reading it. That part comes back faster and is charged at a fraction of the normal input price. The catch is the same as ours: it only matches if the start is identical. So you put the parts that never change (instructions, tool list) first, and the parts that change (the world, the command) at the end.

Part B · Quantising

Step 5: how a computer stores a number

Computers store everything as bits: 0s and 1s. With 1 bit you have 2 patterns (0, 1). With 2 bits, 4 patterns (00, 01, 10, 11). Every extra bit doubles it: 4 bits give 16, 8 bits give 256. A normal Python decimal number uses 32 or 64 bits, enough to store something like 0.42139865 very precisely. The attention model's 2,702 numbers at 32 bits each:

2,702 numbers × 32 bits ÷ 8 bits per byte = 10,808 bytes

Does the robot need 8 decimal places? Probably not. What if we stored each number with 8 bits, or 4?

Step 6: a ruler with fewer marks

Imagine a ruler with only a few marks on it, evenly spaced, and you have to round each measurement to the nearest mark. That's quantising. With b bits we allow the whole numbers from −top to +top, where

top = 2^(b−1) − 1         8 bits: 127      4 bits: 7      3 bits: 3      2 bits: 1

(One bit is used for the sign, + or −. We skip the very last pattern so that the range is the same on both sides.) Then we pick the scale, the gap between marks, so that the biggest number in the list lands exactly on the top mark:

scale = (biggest absolute value) / top
q     = round(v / scale)       ← a whole number, stored in b bits
back  = q × scale              ← "dequantise": roughly the original

Worked example, the one from check.py: quantize([1, −0.5, 0.25], 8).

top   = 127
scale = 1 / 127 = 0.007874
1     / 0.007874 = 127      → 127
−0.5  / 0.007874 = −63.5    → −64     (Python's round() sends an exact half to the even neighbour)
0.25  / 0.007874 = 31.75    → 32
back: 127 × scale = 1.0,  −64 × scale = −0.5039,  32 × scale = 0.2520

Each number moved by less than 0.004. Now 4 bits: quantize([0.7, −0.35], 4) → top 7, scale 0.1, so 7 and −3.5 → [7, −4]. Back: 0.7 and −0.4. That one moved by 0.05: fewer marks, bigger jumps.

One more detail: the Python gives every row of the model its own scale (that's what quantized_copy does). If one row has numbers around 1.5 and another has numbers around 0.01, a single shared scale would round the whole small row to zero.

Step 7: what happens to the attention model, and to the robot

Here's the real output. It quantises Lesson 15's attention model (the best small brain so far) and tests each copy twice: on the 50 held-out plans it never trained on, and by putting it in charge of the robot for the 12 bake-off tasks from Lesson 19:

32-bit:   10808 bytes   held-out plans  92%   robot bake-off  8/12
 8-bit:    2702 bytes   held-out plans  92%   robot bake-off  8/12
 4-bit:    1351 bytes   held-out plans  92%   robot bake-off  8/12
 3-bit:    1013 bytes   held-out plans  80%   robot bake-off  7/12
 2-bit:     676 bytes   held-out plans   8%   robot bake-off  4/12

Three things to notice:

1. 8 bits and 4 bits are free wins. 4× and 8× smaller, with no loss on the held-out plans or on the robot. Neural networks are surprisingly robust to rounding: the decision usually depends on lots of numbers together, and the small rounding errors mostly cancel out.
2. At 3 bits it starts to slip. Held-out plans drop to 80%, and the robot loses one task. Colours are the first thing to go: "put the red and blue blocks away" now comes out as put red green, so the robot bins the wrong block. The close calls flip first.
3. 2 bits breaks it. With only −1, 0 and +1 allowed, too many numbers get wiped out. It writes nonsense like put put put put put for "put the red and blue blocks away", and the robot falls to 4/12.

Real companies do exactly this. Big LLMs are often served with 8-bit or 4-bit numbers (written int8 and int4), which is how a model that needs a room-sized computer at 32 bits can run on one graphics card, or even a laptop. And because writing each token means reading every number from memory, smaller numbers make every token faster too. They measure carefully at each step, just as we did.

Part C · Distilling

Step 8: a teacher and a student

Our attention model learned from 343 examples: most were made from templates in Lesson 15, plus Lesson 9's 34 hand-labelled commands. Templates only cover the phrasings someone thought of. "where's …" isn't one of them, so it still gets "where's the green one?" wrong. More real examples would help. But labelling by hand is slow and boring. Idea: a big, clever, slow model labels lots of new commands for us. Then our small, fast model trains on those labels. That's distillation: the big model is the teacher, the small one the student. You get some of the teacher's smarts, at the student's speed and price.

The teacher can also say "I don't know" (refuse, or ask a question back). Those commands have no label, so we leave them out:

teacher("where's the red one")    → "find"   keep  ("where's the red one", "find")
teacher("make me a sandwich")     → None     skip

The teacher only gives the intent. Lesson 15's make_targets adds the colour from the command, so the student trains on find red.

Step 9: what our students learned

lesson17_faster.py has 24 new commands nobody labelled (UNLABELLED). Two teachers label them, and a new attention-model student trains on the usual data plus those labels. Each student sits three tests: the 50 held-out plans, 6 harder commands ("hard-6", like "where's the green one?" and "bin the red one"), and the robot bake-off. Real output:

student (attention model) trained on       plans  hard-6  robot
the usual data only                         92%    83%   8/12
+ 24 labels from word counter (Lesson 9)    94%    67%   8/12
+ 24 labels from a careful human            90%   100%  10/12

The careful human's labels fixed both, and hard-6 went to 100%. On the robot, two more tasks now pass: "where's the green one?" (it finds the block and says where it is, instead of binning it) and "hmm what about the blue" (it looks instead of binning). That's 8/12 → 10/12. Notice the held-out plans: 92%, 94%, 90%. The worst teacher's student scored highest, and the best teacher's student lowest. That test is made from templates, and none of its templates say "where's", so it couldn't tell a good teacher from a bad one. Differences of a few points on 50 examples are mostly noise. Only the harder tests showed the real difference.

best_teacher() picks the teacher the robot actually uses: Claude's recorded labels if you have them, otherwise the careful human's. brains.py uses it to build two new brains you can drive the robot with:

• distilled: the attention model trained with the teacher's labels. Bake-off 10/12.
• tinylm-4bit: the distilled model quantised to 4 bits. Still 10/12, in 1,411 bytes instead of 11,288. (The teacher's commands add a few new words, so this model has 2,822 numbers, a little more than 2,702.)

Next, the big one: Claude as the teacher. When its recorded labels for the 24 commands are added, its student's scores go here, along with what the labelling cost.

Step 10: how this connects to LoRA, and to real companies

Our student is tiny, so we train all its numbers. But what if the student is itself a medium-sized LLM with a billion numbers? That's where Lesson 13, the other half of this week, comes in: LoRA trains a thin add-on instead of the whole model, so you can fine-tune a student on the teacher's labels cheaply. People even combine it with this lesson: QLoRA keeps the frozen big model in 4-bit numbers and trains a LoRA add-on on top.

Put it all together and you have how real AI products are served:

• a big model writes examples, and a small distilled model handles the everyday requests;
• the model's numbers are quantised to 8 or 4 bits so it fits on fewer, cheaper machines and writes faster;
• the long, unchanging start of every prompt is cached;
• and everything is measured with Lesson 16's tools: p50, p95, cost per token, plus quality on a hard test set, before and after.

3. Play with it

4. Homework

Open homework/lesson17_faster.py. It uses Lesson 8's tokenize and Lesson 15's train_tiny_lm, accuracy, test_data and make_targets, so those need to be finished. The bottom of the file also uses Lesson 9's word counter as a teacher. Its student is the Lesson 15 attention model if you did the stretch (best_student() falls back to the plain tiny LM if you didn't), and the robot column uses Lesson 19's score_task. Until you reach Lesson 19 in week 11, that column prints ?: run python3 ../progress.py from homework to see your robot's bake-off score now, because it has its own scorer. Already written for you: Cache.__init__ and hit_rate, weight_lists, quantized_copy, size_bytes, the 24 UNLABELLED commands and the careful human's labels. Write these five pieces, in order.

4.1 cache_key(command, world)

The words of the command, tidied, then "|", then the world (Steps 2 and 3). Example: cache_key("Put the RED block!", "w1") → "put the red block|w1".

4.2 Cache.ask(self, command, world)

Look up the key. If it's in self.store: add 1 to self.hits and return the stored answer. If not: add 1 to self.misses, call self.think(command, world), store the answer, return it. Example: ask "Stop!" then "stop" in world "w", then "stop" in world "w2": the slow brain thinks twice, hits = 1, misses = 2.

4.3 quantize(values, bits)

Step 6, exactly. Return a pair: the list of whole numbers, and the scale. Examples: quantize([1, -0.5, 0.25], 8) → ([127, -64, 32], 1/127). quantize([0.7, -0.35], 4) ints → [7, -4]. quantize([0, 0], 8) → ([0, 0], 1.0).

4.4 dequantize(qs, scale)

Back to decimals: each whole number times the scale. Example: dequantize([127, -64], 1/127) → [1.0, -0.5039…].

4.5 distill_examples(commands, teacher)

Ask the teacher about each command; keep the ones that got an intent (Step 8). train_attn_lm turns the intents into plan words for you. Example: with a teacher that knows only {"a": "put", "c": "stop"}, distill_examples(["a", "b", "c"], teacher) → [("a", "put"), ("c", "stop")].

Run it

cd homework
python3 lesson17_faster.py

What you should see (this is the real output of the reference solution; it trains four attention models and drives the robot through the bake-off with each, so give it a minute or so):

1) CACHING. 6 commands in a world where nothing changes, then the red block moves.
   real thinking calls: 5, hit rate 38%

2) QUANTISING the Lesson 15 attention model (the best small brain so far), and driving the robot with it.
   32-bit:   10808 bytes   held-out plans  92%   robot bake-off  8/12
    8-bit:    2702 bytes   held-out plans  92%   robot bake-off  8/12
    4-bit:    1351 bytes   held-out plans  92%   robot bake-off  8/12
    3-bit:    1013 bytes   held-out plans  80%   robot bake-off  7/12
    2-bit:     676 bytes   held-out plans   8%   robot bake-off  4/12

3) DISTILLING: the student learns from a teacher's labels on 24 new commands.
   (no recorded Claude labels yet: run `python3 brains.py record` with an API key)
   student (attention model) trained on       plans  hard-6  robot
   the usual data only                         92%    83%   8/12
   + 24 labels from word counter (Lesson 9)    94%    67%   8/12
   + 24 labels from a careful human            90%   100%  10/12

Check it

cd ..
python3 check.py 17

A green ✓ means that part passed. A red ✗ comes with a hint, for example:

• ✗ cache_key: a different world must give a different key: you left the world out of the key. That's the stale-answer bug from Step 3.
• ✗ quantize: quantize([1, -0.5, 0.25], 8) ints should be [127, -64, 32], got [127, -63, 31]: you used int(), which chops off the decimals. Use round().

All green shows 6/6 passed.

Stretch

• Wrap the attention model in your Cache (think = lambda c, w: model.generate(c)) and time 1,000 asks of the same 5 commands with Lesson 16's time_it. What's the hit rate, and how much faster is p50?
• Give the whole model one shared scale instead of one per row (quantise all 2,702 numbers as a single list). At which bit width do the held-out plans drop now? Why?
• Count how many of the 2,702 numbers become exactly 0 at each bit width.
• Make a "noisy teacher": the careful human, but with 4 of the 24 labels deliberately wrong. Train the student with train_attn_lm(extra) and compare hard-6 and the robot score. How many wrong labels does it take to do worse than no teacher?

5. Real-world challenge: rounding and remembering, at home

Part A: quantise your house.

1. With a tape measure, measure 8–10 things to the nearest millimetre: a doorway's width, a sofa or bed, a box, a gap between two pieces of furniture, a table, a suitcase.
2. Write down 4 yes/no decisions that use these numbers, like "does the box fit through the doorway?" or "does the suitcase fit in the gap?". Answer them with the exact numbers.
3. Now "quantise": round every measurement to the nearest 10 cm. Answer the decisions again. Then round to the nearest 50 cm, and then the nearest metre.
4. Record, for each decision, the roughest rounding at which it still gives the right answer.

Part B: a cache for "where is it?"

1. For one week, keep a notebook page: "keys: hook by the door", "TV remote: sofa", "charger: kitchen".
2. Whenever someone in your home asks where something is, check the notebook first. If it's there and right: a hit. If it isn't listed, go and look, then write it down: a miss. If the notebook was wrong because the thing moved: a stale hit.
3. At the end of the week, work out the hit rate and how many hits were stale. Then add "last checked: day and time" to each entry. Does that help you decide when not to trust it?

What it should teach you: most decisions survive rough numbers, and the close calls flip first, exactly like the colours at 3 bits. And a cache is only as good as its key: "where are the keys" has a different answer after someone goes out, so the world (or at least the time) belongs in the key.

6. Your Robo-Helper so far

Robo-Helper now has a smaller, smarter brain. Cache can wrap any brain's thinking, so repeated commands in the same world are instant and free. quantize and dequantize (through quantized_copy) shrink the brain to an eighth of its size with no loss on the robot. distill_examples turns a teacher's answers into training data for a student, and best_teacher() picks the teacher.

Next, in the same week, Lesson 13 shows how to fine-tune a bigger student cheaply with LoRA. In Lesson 18, the world-class planner becomes the teacher: python3 brains.py record asks it to label exactly these 24 UNLABELLED commands and saves them to recordings/teacher_labels.json, which recorded_teacher() in your file then loads, and best_teacher() prefers. Its plans also depend on the world, so if you cache them, your world-in-the-key rule is what keeps them safe. And in Lesson 19 the bake-off scores every brain on the same 12 tasks for passes, p50, p95 and cost.

7. Common mistakes

• Leaving the world out of the cache key, so the robot gets stale answers after things move. Checker hint: "a different world must give a different key".
• Calling think on a hit too (or forgetting to store the answer on a miss), so nothing is saved. Checker hint: "'Stop!' then 'stop' in the same world should think once, then once more for a new world".
• Using int() instead of round() in quantize. int chops towards zero, so −63.5 becomes −63 instead of −64.
• Dividing by zero when every value is 0. Checker hint: "all zeros: scale 1.0, no dividing by zero".
• Keeping the teacher's None answers in distill_examples, so the student trains on examples with no label. Checker hint: "keep (command, intent) only where the teacher gave an intent".

8. Check yourself

9. Go deeper (optional)

• Prefix caching and cache-aware routing: prompt caching inside a real serving system.
• FP8 vs INT8 vs INT4: pick one and measure the loss: the grown-up version of our bits slider.
• Cut cost per million tokens by 70% without hurting quality: all three tricks working together.
• Big teacher, small student: distillation for robots, with how to read the results honestly.
• QLoRA: a 4-bit frozen model plus a LoRA add-on: this lesson and Lesson 13 in one trick.
• On-policy distillation, explained with videos: a smarter way for students to learn from teachers.