LLMBuild a Robot Brain · Week 8 · Lesson 16

How fast is a brain? Latency, tokens and cost

You've built three kinds of brain for Robo-Helper, and next week it gets a world-class one over the internet. Before you trust any of them, you need two numbers you can't see by reading the code: how long it takes to answer, and what each answer costs. This lesson is the start of inference engineering: measuring brains like an engineer, not like a fan.

Time2–3 hours
You'll needPython 3, your finished Lessons 9 and 15, a 30 cm ruler, a friend, paper and a pencil
New wordsinference, latency, median (p50), percentile, p95, tail, mean, TTFT, streaming, throughput, token, latency budget
Project pieceA stopwatch and a price tag for every brain

Start here

A brain is only useful if it answers in time: time it many times, judge it by its slow answers, not its average, and count what every answer costs in tokens.

The problem

Robo-Helper is rolling across the kitchen at 1 metre per second. A cat strolls out from under the table, 1 metre ahead. You shout "stop!". The words go to the robot's brain, and the brain thinks. If it thinks for 0.01 seconds, the robot moves 1 cm and stops. If it thinks for 2 seconds, the robot has gone 2 metres, straight through where the cat was. Same robot, same command, same correct answer. The only difference is how long the answer took.

The idea in one picture

Think about a school bus. Most days it's 2 minutes late. One day in twenty it's 15 minutes late. If someone asks "how late is the bus?", the honest answer isn't the average (about 3 minutes, which never actually happens). It's "usually 2 minutes, but on bad days 15". And if you have an exam, you plan for the bad day. Engineers measure brains the same way: the usual answer time (p50) and the bad-day time (p95).

Words to know

Inference
Using a trained model to get an answer. Training happens once, in the workshop. Inference happens every time the robot hears a command.
Latency
How long one answer takes, from asking to getting it. Measured in seconds (s) or milliseconds (ms, thousandths of a second).
Mean
The ordinary average: add everything up, divide by how many.
Median (p50)
The middle value once you sort them: half the answers were faster, half slower. "The usual time."
Percentile, p95
p95 is the time that 95% of answers beat. Only 1 in 20 is slower. "The bad-day time."
Tail
The slow answers at the far end of the list. "Tail latency" means p95 or p99.
Token
The chunk of text an LLM reads and writes: a short word, or a piece of a longer one.
TTFT
Time To First Token: how long until the first piece of the answer arrives.
Streaming
Sending the answer piece by piece as it's written, instead of all at the end.
Throughput
How many tokens per second come out once the answer is flowing.
Latency budget
The longest you can afford to wait. For "stop!", ours is 0.2 s.

Going deeper: everything below is the full lesson: the ideas built from nothing, a demo where you watch the robot drive while its brain thinks, your homework, and a reaction-time challenge with a ruler.

1. The problem

By now Robo-Helper can have several different brains:

Which one should drive the robot? "The smartest" sounds right, until you remember the cat. You need to know how fast each one is. So let's measure.

Obvious idea 1: time it once. Start a stopwatch, ask, stop the stopwatch. But computers are busy doing many things at once. The first call is often slower (the program is still "warming up": loading things into memory). An internet answer might hit a slow moment on the network. One measurement is one roll of the dice.

Obvious idea 2: time it many times and take the average. Better, but still a trap. Here are four timings, in seconds: 1, 2, 3, 100. The average is (1 + 2 + 3 + 100) / 4 = 26.5. Not a single answer took anything like 26.5 seconds. The average hides both the usual case (about 2) and the scary case (100). We need better numbers.

2. First principles

Step 1: training versus inference

So far most of our lessons were about training: showing a model examples and nudging its numbers (Lessons 3, 4, 15). Training happens once, and it can be slow. Nobody minds if it takes all night.

Inference is the other half: the trained model, sitting still, being used. Every time someone says "put the red block in the bin", the brain runs inference once. A robot (or a chatbot with millions of users) runs inference again and again, forever. So most of the money and most of the waiting in AI is spent on inference, not training. Inference engineering is the job of making it fast, cheap and reliable. That's this lesson and the next.

Step 2: a precise stopwatch

Python has a very precise stopwatch: time.perf_counter(). It returns a number of seconds since some starting moment. The number itself means nothing; the difference between two readings is what you want:

start = time.perf_counter()
do_the_thing()
seconds = time.perf_counter() - start

Our word counter thinks for about 0.000012 seconds. That's 0.012 milliseconds, or 12 microseconds. A school stopwatch could never see that. perf_counter can.

Step 3: measure many times, then sort

Here are 10 timings of an imaginary internet brain, in seconds. (These are made-up practice numbers, so you can work with a pen.)

1.2  1.1  1.3  1.2  4.8  1.1  1.2  1.4  1.2  1.3

First, sort them. Then number the positions from 0, like a Python list:

position:  0    1    2    3    4    5    6    7    8    9
sorted:    1.1  1.1  1.2  1.2  1.2  1.2  1.3  1.3  1.4  4.8

The mean is the total divided by 10: 15.8 / 10 = 1.58 s. The percentile rule we'll use (it's called "nearest rank") says: the p-th percentile of n values is the one at position

position = ceil(p / 100 × n) − 1         (and never less than 0)

ceil means "round up to a whole number": ceil(2.1) = 3, ceil(5) = 5. So:

p50: ceil(0.50 × 10) − 1 = 5 − 1 = 4     → sorted[4] = 1.2 s
p95: ceil(0.95 × 10) − 1 = 10 − 1 = 9    → sorted[9] = 4.8 s

The usual answer takes 1.2 s. The bad-day answer takes 4.8 s. And the mean, 1.58 s, describes neither.

Predict first: what's the p90 of those 10 timings? And the p100?

p90: ceil(0.90 × 10) − 1 = 9 − 1 = 8, so sorted[8] = 1.4 s. p100: ceil(10) − 1 = 9, so 4.8 s: the 100th percentile is always the slowest one. Notice how p90 and p95 are wildly different here: with only 10 timings, p95 is just "the worst one". That's why the homework times things 20 or 200 times.

Step 4: why the slow answers matter most

p95 means "1 in 20 answers is slower than this". That sounds rare. But a robot that hears 100 commands a day gets about 5 slow answers every day. If any of those is "stop!", the cat doesn't care that the average was fine.

So the rule for a latency budget is: a brain fits only if its p95 fits. Big companies go further and watch p99 (1 in 100) or even p99.9, because with millions of users, "rare" happens every second.

Why averages lie. One very slow answer drags the mean up, so the mean looks worse than the usual case. Lots of fast answers drag it down, so it looks better than the bad case. It's in the middle of nowhere. Always report p50 and p95.

Step 5: a robot moves while it thinks

For a chatbot, a slow answer is annoying. For a robot, it's distance. If the robot keeps moving while the brain thinks:

distance = seconds × speed
0.000013 s × 1 m/s = 0.000013 m = 0.013 mm      (word counter, p95)
1.5 s      × 1 m/s = 1.5 m                      (an internet brain, made-up)
1.5 s      × 0.5 m/s = 0.75 m                   (same brain, robot driving at half speed)

The robot has two choices while the brain thinks, and both cost something. It can keep moving (and travel blind), or it can freeze (and waste time standing still, so a 10-step tidy-up with a 5-second brain spends 50 seconds frozen).

Step 6: the 0.2-second rule, and why robots have layers

Our budget: the robot must react to "stop!" within 0.2 seconds. That's about how fast a person reacts (you'll measure yours in the challenge).

Predict first: the word counter's p95 is 0.013 ms. The tiny LM's is 0.317 ms. An internet brain's p95 is (say) 3 s. Which fit the 0.2 s budget?

The word counter and tiny LM fit easily: 0.317 ms is 0.000317 s, more than 600 times under budget. The internet brain doesn't come close. Even if it was usually fast, one network hiccup and it's out. That's why safety reflexes must stay local, on the robot itself, never waiting for the internet.

Real robots are built in layers, fast at the bottom and smart at the top:

Self-driving cars and warehouse robots work like this: a fast controller keeps them safe many times a second, and a slower planner decides where to go. The big brain plans; the small brains react.

Step 7: tokens, streaming and TTFT

An LLM doesn't read letters or whole words. It reads and writes tokens: chunks of text from a fixed list it learned. Common short words are usually one token ("the", "red"); long or rare words get split into pieces. A rough rule for English is about 4 letters per token, so 100 words is roughly 130 tokens. (It's only a rule of thumb: every model has its own token list.)

Lesson 15's tiny LM showed you the core loop: predict the next word, add it, repeat. A big LLM does exactly that, one token at a time. So instead of waiting for the whole answer, the server can stream it: send each token the moment it's written. That splits the waiting into two parts:

ask ──────────── first token ── token ── token ── … ── last token
    |←── TTFT ──→|←────────── the rest of the answer ──────────→|
    |←──────────────────────── total time ─────────────────────→|

TTFT (time to first token) includes the trip over the internet, waiting in line at the server, reading your whole question, and (for models like Claude that think before answering) the thinking. After the first token, the speed is the throughput:

tokens per second = (tokens_out − 1) / (total − TTFT)
101 tokens, 3 s total, first token after 1 s:   (101 − 1) / (3 − 1) = 100 / 2 = 50 tokens/s

Why minus 1? The first token arrived at TTFT. The other 100 arrived during the remaining 2 seconds.

Predict first: 201 tokens, 5 s total, TTFT 1 s. Throughput? And what if TTFT equals the total time?

(201 − 1) / (5 − 1) = 200 / 4 = 50 tokens/s. If TTFT equals total, the whole answer came at once: there's no time left to divide by (dividing by zero would crash), so we say 0.0.

For a chatbot, a short TTFT makes it feel fast: you start reading straight away. For a robot, streaming can help too: it can start on step 1 of a plan while steps 2 to 10 are still being written. But it can't stop for the cat any sooner.

Step 8: the price tag

Internet models charge per token, priced per million tokens, and writing costs more than reading. For claude-opus-5-5:

input (what you send):    $4  per 1,000,000 tokens
output (what it writes):  $20 per 1,000,000 tokens

one command, 1,000 tokens in and 200 out:
  1,000 / 1,000,000 × $4  = $0.004
    200 / 1,000,000 × $20 = $0.004
  total                   = $0.008       (less than a penny)

1,000 commands a day      = $8 a day

Two surprises. First: why 1,000 tokens in, for an 8-word command? Because every request also sends the instructions ("you are a robot, here are your tools, here are the rules") and a description of the room. That gets re-sent every single time. Second: output costs 5× more than input, so a chatty answer costs more than a long question. Lesson 17 attacks both.

Predict first: a million tokens in and nothing out. How much?

$4. That's what "per million" means. A million tokens is roughly 750,000 words: about ten long novels.

Step 9: the real numbers, on our computer

Running lesson16_speed.py times each local brain 200 times on "put the red block in the bin":

words (Lesson 9)                p50 0.012 ms   p95 0.013 ms   robot travels 0.01 mm while thinking
tinylm (Lesson 15)              p50 0.269 ms   p95 0.317 ms   robot travels 0.32 mm while thinking
attention (Lesson 15 stretch)   p50 0.323 ms   p95 0.384 ms   robot travels 0.38 mm while thinking
tinylm-4bit (Lesson 17)         p50 0.362 ms   p95 0.372 ms   robot travels 0.37 mm while thinking

The tiny LM is about 25× slower than the word counter, because it does thousands of multiplications instead of a few dozen look-ups. Attention is a little slower again, because it works out a fresh set of weights for every word it writes. But all of them are hundreds of times inside the 0.2 s budget. Notice how close p50 and p95 are: a small program on your own computer is very consistent. Internet brains have a much longer tail, because there are many more places to get stuck (your Wi-Fi, the internet, the queue at the server, how long the model decides to think).

One surprise: the 4-bit brain is no faster than the attention model it came from (in this run it was even a little slower, which is just noise). In our Python, Lesson 17's 4-bit copy still keeps its rounded numbers as ordinary decimals, so it does exactly the same sums. It's 8× smaller to store, but speed only comes when the computer does the maths with small whole numbers, which needs special hardware or libraries. Measure; don't assume.

When real recorded runs of Claude as the planner and the agent are added, their p50, p95, travel distance and cost for 12 commands go here.

Until then, the demo below lets you try values for the internet brain and see what they would mean. They are not measurements.

3. Play with it

Interactive: the slow tail and the cat

Top: 100 (or however many you pick) timings of an internet brain. These are try values drawn from the sliders, not measurements. Each dot is one answer. Lines show the mean, p50 and p95, worked out with exactly the same nearest-rank rule as percentile() in the Python. Bottom: three robots drive at 1 m/s towards a cat 1 m ahead while their brain thinks for its p95 time. The word counter and tiny LM use the real p95s measured above.

p50
–
p95
–
Mean
–
Travels in p95
–
Fits 0.2 s?
–

Cost per command
–
Cost per day
–
Time frozen per day (p50)
–

Things to try: set "very slow" to 0 and see p50, p95 and mean land together. Now push it to 10%: p50 barely moves, the mean creeps up, and p95 jumps. Pick 5 timings and press "New random day" a few times: p95 is just "the worst one" and jumps around wildly, which is why we time things many times. Then find the slowest usual time for which the internet robot still stops before the cat. Does it ever fit the 0.2 s budget?

4. Homework

Open homework/lesson16_speed.py. It needs your finished Lesson 9 (the word counter) and Lesson 15 (the tiny LM), because the bottom of the file times them. It also times the attention stretch and Lesson 17's 4-bit brain. Until you've built those, it skips their rows. Already written for you: measure_stream, which times a real streamed answer from Claude (only used if you have an API key), and load_recorded_seconds, which reads recorded real runs. Write these seven functions, in order.

4.1 time_it(fn, repeats=20)

Call fn() repeats times, timing each call on its own. Return the list of times in seconds. Example: time_it(lambda: sum(range(1000)), 5) → a list of 5 small numbers, like [2.1e-05, 1.4e-05, 1.3e-05, 1.3e-05, 1.3e-05] (yours will differ; notice the first one is often the slowest).

Hint 1

You need one stopwatch reading before each call and one after it, not one around the whole loop.

Hint 2

Make an empty list. Loop repeats times (for _ in range(repeats): the _ means "I don't need the counter").

Hint 3

Inside the loop: start = time.perf_counter(), then fn(), then append time.perf_counter() - start to the list. Return the list after the loop.

4.2 percentile(values, p)

The nearest-rank percentile from Step 3. Examples: percentile([5, 1, 3, 2, 4], 50) → sorted [1, 2, 3, 4, 5], position ceil(2.5) − 1 = 2 → 3. And percentile([7], 50) → 7.

Hint 1

Sort first. Use sorted(values), which makes a new sorted list and leaves the caller's list alone.

Hint 2

math.ceil rounds up. The position is math.ceil(p / 100 * len(s)) - 1. What happens for p = 0? The position would be −1, which in Python means "the last one". Wrong! That's why the docstring says "at least position 0".

Hint 3

s = sorted(values), then return s[max(0, math.ceil(p / 100 * len(s)) - 1)].

4.3 summarize(seconds)

Return a dictionary with three numbers. Example: summarize([1, 2, 3, 100]) → {"p50": 2, "p95": 100, "mean": 26.5}. (Check p50 by hand: ceil(0.5 × 4) − 1 = 1 → sorted[1] = 2.)

Hint 1

You've already written the hard part. Use your percentile twice.

Hint 2

The file imports statistics; statistics.mean(seconds) gives the mean.

Hint 3

Return {"p50": percentile(seconds, 50), "p95": percentile(seconds, 95), "mean": statistics.mean(seconds)}.

4.4 tokens_per_second(tokens_out, seconds_total, ttft)

Throughput after the first token (Step 7). Example: tokens_per_second(101, 3.0, 1.0) → 50.0. And tokens_per_second(1, 1.0, 1.0) → 0.0.

Hint 1

Two cases. Deal with "no time left" first.

Hint 2

If seconds_total - ttft is 0 or less, return 0.0.

Hint 3

Otherwise return (tokens_out - 1) / (seconds_total - ttft).

4.5 cost_usd(tokens_in, tokens_out, price=PRICE_PER_MTOK)

Dollars for one answer (Step 8). Example: cost_usd(1000, 200) → 0.008. cost_usd(1_000_000, 0) → 4.0.

Hint 1

price is a dictionary: price["input"] is 4.00 and price["output"] is 20.00, in dollars per million tokens.

Hint 2

Turn tokens into "millions of tokens" first: divide by 1e6 (that's 1,000,000 written short).

Hint 3

Return tokens_in / 1e6 * price["input"] + tokens_out / 1e6 * price["output"].

4.6 thinking_distance(seconds, speed=ROBOT_SPEED) and 4.7 fits_budget(summary, budget_seconds)

thinking_distance(2.5) → 2.5 metres (at 1 m/s); thinking_distance(2, speed=0.5) → 1.0. fits_budget({"p50": 0.1, "p95": 0.3}, 0.2) → False, even though the usual answer is fast. fits_budget({"p50": 0.1, "p95": 0.2}, 0.2) → True.

Hint 1

Distance = time × speed (Step 5). A budget is judged on the slow answers (Step 4).

Hint 2

summary is the dictionary from summarize. Which key holds the slow answers?

Hint 3

return seconds * speed, and return summary["p95"] <= budget_seconds (equal to the budget still fits).

Run it

cd homework
python3 lesson16_speed.py

What you should see (this is the real output of the reference solution; your milliseconds will be different, because your computer is different):

Timing each brain's THINKING on 'put the red block in the bin' (200 runs)

brain                                 p50        p95  robot travels while thinking
words (Lesson 9)                  0.012ms    0.013ms  0.01 mm
tinylm (Lesson 15)                0.269ms    0.317ms  0.32 mm
attention (Lesson 15 stretch)     0.323ms    0.384ms  0.38 mm
tinylm-4bit (Lesson 17)           0.362ms    0.372ms  0.37 mm
planner (Claude)              no recordings yet (see brains.py record)
agent (Claude)                no recordings yet (see brains.py record)

Budget: the robot must react to a 'stop!' within 0.2 s.
Which brains could you trust with that? Which could still plan a whole tidy-up?

Answer those last two questions in a comment at the bottom of your file. (Hint: Step 6.)

Check it

cd ..
python3 check.py 16

A green ✓ means that part passed. A red ✗ comes with a hint, for example:

All green shows 6/6 passed.

Stretch

5. Real-world challenge: measure your own brain's latency

You are a brain too. Let's find your p50 and p95, and see if you fit the 0.2 s stop budget.

Part A: the ruler drop (reaction time).

  1. A friend holds a 30 cm ruler vertically, with the 0 mark at the bottom, between your open thumb and finger (not touching).
  2. Without warning, they let go. You catch it as fast as you can. Write down the centimetre mark where your fingers caught it.
  3. Do it 20 times. Write every number down, including the bad ones. A miss counts as 30+.
  4. Turn each distance into time. Things fall faster and faster, so the rule is time = √(2 × distance ÷ 9.8) with distance in metres. Or use this table: 
    caught at:  5 cm   10 cm   15 cm   20 cm   25 cm   30 cm
time:       0.10 s 0.14 s  0.18 s  0.20 s  0.23 s  0.25 s
  5. Sort your 20 times. Your p50 is position ceil(0.5 × 20) − 1 = 9 (the 10th smallest). Your p95 is position ceil(0.95 × 20) − 1 = 18 (the 19th smallest). Work out the mean too.
  6. Now do 20 more while distracted: your friend asks you maths questions, or you count backwards from 100 in sevens. Work out p50 and p95 again.

Part B: time to first token. Ask a friend "name ten animals" and use a phone stopwatch with laps. Press lap at the first animal and at the last. TTFT = time to the first animal. Throughput = 9 ÷ (total − TTFT) animals per second. Try a harder question ("name ten countries in Africa"). Which grows more: TTFT or total?

What to record: both sets of 20 times sorted, p50, p95 and mean for each, and the animal timings.

What it should teach you: distraction barely moves your p50 but stretches your tail, just like a busy computer or a busy server. Your p95 is probably close to the 0.2 s budget, and that's with a local brain (yours) and no internet. Harder questions mostly add to the "thinking before the first word" time, just like an LLM that thinks before it writes.

6. Your Robo-Helper so far

What this adds to your robot. Before this lesson, nobody knew how long the robot's brain took, so nobody knew whether it could stop in time. Now it does. Row 16 of python3 progress.py --solutions measured: "tiny LM p95 0.3 ms = 0.3 mm rolled while thinking" on "put the red and blue blocks away". That's more than 600 times inside the 0.2 s stop budget. (Your number will differ: it depends on your computer and how busy it is.)

Robo-Helper now has a stopwatch and a price tag. time_it and summarize give any brain a p50 and p95; thinking_distance turns that into metres; fits_budget says yes or no for a safety budget; cost_usd says what each answer costs.

These get used for real later. In Lesson 19, the bake-off runs every brain on the same 12 tasks, and its scorecard() function calls your summarize() and cost_usd() to report each brain's p50, p95 and total cost side by side. Before that, Lesson 17 uses this lesson's way of thinking to make the brains cheaper and faster, and Lesson 18 puts a world-class brain in charge of planning while the fast local layers keep watch.

7. Common mistakes

8. Check yourself

Predict: what does summarize([0.2, 0.1, 0.1, 0.3, 0.1]) return?

Sorted: [0.1, 0.1, 0.1, 0.2, 0.3]. p50: ceil(2.5) − 1 = 2 → 0.1. p95: ceil(4.75) − 1 = 4 → 0.3. Mean: 0.8 / 5 = 0.16. So {"p50": 0.1, "p95": 0.3, "mean": 0.16}. The usual answer is 0.1 s, but the slow one fails a 0.2 s budget.

What's the difference between training and inference, and which one does a robot do every time you speak to it?

Training adjusts the model's numbers using examples, once, ahead of time. Inference uses the finished model to answer. The robot runs inference on every command.

A brain has p50 = 0.05 s and p95 = 0.4 s. Can it handle "stop!" with a 0.2 s budget? Why might its p95 be so much bigger than its p50?

No: 1 in 20 answers takes longer than 0.2 s, and the budget is judged by p95. A long tail usually means the brain sometimes waits on something else: the network, a busy server, other programs on the computer, or a model that occasionally thinks much longer.

An answer has 51 tokens, TTFT 2 s and total 4.5 s. What's the throughput, and which part of the wait would streaming help with?

(51 − 1) / (4.5 − 2) = 50 / 2.5 = 20 tokens/s. Streaming lets you see the first token after 2 s instead of 4.5 s. It doesn't make the total shorter.

500 tokens in and 500 out. What does it cost, and which half is more expensive?

In: 500 / 1,000,000 × $4 = $0.002. Out: 500 / 1,000,000 × $20 = $0.01. Total $0.012. The output is 5× more expensive.

9. Go deeper (optional)