100 days of inference engineering: a learning journal, the book behind it, and a plan you can follow
Someone has started reading Philip Kiely's Inference Engineering one chapter at a time, writing up what they learn each day. This module maps that book onto the course and turns the idea into a 100-day plan with a tracker.
Start here (no maths needed)
Learn one thing every day for 100 days, write down what you learned, and share it. Small steps add up to expertise.
Think of it like this
It's like a "100 days of drawing" challenge. You draw something every day and post it. Day 1 looks rough, and by day 100 you're far better. You can also look back and see exactly how you got there. Here, the "drawing" is understanding how AI models are served to millions of people.
How it works
- Pick a guide (here, a book about inference engineering).
- Each day, read or build one small piece.
- Write a few lines about what you learned, in your own words.
- Share it publicly, so others can learn and correct you.
- Tick the day off. Keep the streak going, but don't panic if you miss one.
Why robots care
Robots run AI models on a small computer with strict time limits. "Inference engineering" is the skill of making models answer fast and cheaply, and that's exactly what a robot needs.
Try it
Use the 100-day grid below. Tick today's box after you've learned one thing and written one sentence about it.
Words to know
- Inference
- Running a trained AI model to get answers.
- Learning in public
- Sharing your notes as you learn, mistakes and all.
- Latency
- How long an answer takes.
- Streak
- How many days in a row you've kept going.
Going deeper: the book map, the journal so far, and the 100-day plan.
Track: LLM. Inference engineering is about serving models. It carries over to robotics wherever a model must answer within a deadline on limited hardware (Module 04).
1. Summary
TL;DR
A new GitHub journal, "100 days of Inference Engineering", works through Philip Kiely's book chapter by chapter. Kiely is the Baseten author already quoted in Module 05.
The first two entries cover the inference stack (runtime plus infrastructure), shared vs dedicated deployment, model choice, application-specific evals, fine-tuning vs distillation, and the core latency metrics.
The book's eight chapters line up almost one-to-one with Modules 09 and 12, so the journal works as a reading track alongside the course.
2. The book: Inference Engineering
From Baseten's page: a book for engineers who want to understand the technologies behind AI applications. Kiely says it distils four years of work at Baseten, his docs and talks, and interviews with Baseten engineers. The page frames inference engineering as work across the whole stack, "from CUDA to Kubernetes", in a field that's still young. It can be read online, and digital and paper copies are available.
The table maps each chapter title to course pages. The mapping goes by chapter titles and the journal's notes; check the book for each chapter's actual scope.
| Chapter | Read alongside in this course |
|---|---|
| 0 · Inference | 09 hub, 09 · request lifecycle |
| 1 · Prerequisites | 01 evals, 04 distillation, 09 · metrics & observability |
| 2 · Architecture | 05 Kimi K3, 08 Stage 08, 09 · KV cache & attention |
| 3 · Hardware | 09 · GPUs, 09 · roofline & profiling |
| 4 · Software | 09 · batching, 09 · engines, 12 · serving video |
| 5 · Techniques | 09 · speculative decoding, 09 · quantisation, 09 · caching, 12 · video 04 |
| 6 · Modalities | 08 Stage 07 multimodal, 04 speech |
| 7 · Production | 09 · rollouts & autoscaling, 09 · cost & capstone |
3. The journal so far
Summarised in my own words from the README (Days 1–2 at the time of fetching):
- Day 1 (chapter 0):
- Inference engineering is young; it was a niche specialty only a few years ago.
- Since around 2024, open models have been catching up with closed ones. The DeepSeek V3/R1 moment is the example. As a result, many companies fine-tune open models for domain tasks.
- A production stack needs runtime optimisations, infrastructure optimisations and developer tooling.
- Day 2 (chapter 1):
- There's no one-size-fits-all setup. Know your model requirements, interface, latency budget, unit economics and usage pattern.
- Choose between shared (pay-per-token) inference and dedicated deployments. If the workload is unknown, start shared.
- Model choice is the biggest lever: try a smaller model before reaching for tricks like speculative decoding.
- Distrust gamed benchmarks; build application-specific evals.
- Fine-tuning changes weights only, while distillation moves behaviour into a different, smaller architecture.
- The metrics: TTFT, perceived and total tokens/s, inter-token latency, and percentiles.
Each idea already has a deeper page here: model choice and cost (09 · Problem 24), application evals (01, 08 Stage 10), distillation (04, 12 · video 05), and metrics and percentiles (09 · Problem 17).
4. A 100-day plan built on this course
One book chapter per block, each block paired with course pages and one hands-on build. Day counts are suggestions.
| Days | Read | Build (from the course) |
|---|---|---|
| 1–10 | Ch 0–1 | Measure TTFT and tokens/s against a pay-per-token API, and plot percentiles (09 · page 01). |
| 11–25 | Ch 2 | Size the KV cache for three models by hand, then check with the calculator (09 · page 02). |
| 26–40 | Ch 3 | Roofline a decode step and a prefill step for your GPU (09 · page 07). |
| 41–55 | Ch 4 | Run vLLM on a small model: compare static vs continuous batching, and turn on chunked prefill (09 · page 03). |
| 56–75 | Ch 5 | Add prefix caching, then speculative decoding, then FP8. Measure each change separately (04, 05, 06). |
| 76–85 | Ch 6 | Serve a small VLM or a speech model, and measure the vision-token or audio latency budget (08 Stage 07). |
| 86–100 | Ch 7 | Dashboards, a canary rollout and autoscaling. Finish with the 1M-user capstone out loud (09 · page 09, Problem 25). |
For runnable "build" labs, use the companion code of a second book (Module 16 lab track). Use Module 12's videos as your weekend watching: video 09 (serving) during Ch 4, and video 04 (speculative decoding) during Ch 5.
Interactive: your 100-day tracker
Click a day to mark it done. Colours show the book chapter block. Progress is saved in this browser only.
5. How to learn in public well
- Write in your own words. Paraphrasing is where understanding happens (Module 03, desirable difficulty). It also avoids copying the book.
- Add one number a day: a measurement, a calculation, a benchmark. Numbers make notes useful later.
- Say what you didn't understand. Readers will help, and it shows honest progress.
- Publish runs, not just reading notes: configs, logs and what didn't work (Module 08 Stage 12).
- Respect sources: credit the book, and don't repost its figures without permission.
6. Why it matters for robotics
- Latency budgets are hard deadlines on robots. TTFT and inter-token latency become reaction time.
- "Try a smaller model first" is doubly true on a robot's limited compute. Distillation is the main route (Module 04).
- Edge vs cloud is the robot's "shared vs dedicated" choice. Run fast reflexes on board, and send slow, heavy reasoning to a server.
- Fleet-scale serving chapters (autoscaling, multi-tenancy) matter mostly for cloud-side robot services, not the robot itself.
7. Self-check
When should you start with shared (pay-per-token) inference instead of a dedicated deployment?
When you don't yet know your workload's volume and shape. Shared inference avoids paying for idle GPUs while you learn.
What's the difference between fine-tuning and distillation, as the journal notes from the book?
Fine-tuning retargets a model by changing its weights only. Distillation transfers a large model's behaviour into a smaller, different architecture.
Why look at latency percentiles, not just averages?
Averages hide the slow tail. Users and SLOs feel the p95/p99. See Module 09, Problem 17.
What's the cheapest optimisation to try before speculative decoding?
A smaller model that still passes your application evals.