LLM inference engineering: 25 interview problems, worked through with real numbers

Start here (no maths needed)

Training makes a model smart. Inference is using it, for lots of people, fast and cheaply. That's a big engineering job of its own.

Think of it like this

Think of a busy restaurant kitchen. The recipe (the model) is already written. Now the kitchen must cook it for hundreds of diners at once. The chef keeps ingredients ready on the counter (the memory cache), cooks several orders on one stove (batching), starts serving the first bite fast so nobody waits too long, and keeps the gas bill down (cost). Inference engineers run that kitchen.

How it works

1. A question arrives and joins a queue.
2. The model reads the whole question in one go (this is called "prefill").
3. Then it writes the answer one word-piece at a time, sending each as it goes (this is called "decode").
4. To avoid re-reading everything for every new word, it keeps a memory of what it has read (the KV cache).
5. The server juggles many people's requests on the same chips so none are left idle.

Why robots care

A robot's brain has the same steps, but on a small computer inside the robot with a hard time limit. If it takes too long to think, the robot keeps moving in the meantime. Module 04's "how far does the robot travel after you say stop" calculator shows why.

Try it

Next time you're in a queue at a café, notice: does the barista make one drink at a time, or start several at once? Which way gets everyone served sooner? That's the batching question in Problems 6 and 7.

Words to know

Inference

Running a trained model to get answers.

Token

A word-piece the model reads or writes.

Latency

How long you wait.

Throughput

How much work gets done per second, across everyone.

GPU

The chip that does the model's maths.