Inference Engineering, interactive edition: how to use inferenceengineering.tech alongside this course
A free, interactive web companion to Philip Kiely's Inference Engineering: eight chapters, three guided paths, calculators, quizzes, cheat sheets and its own vLLM-vs-SGLang benchmark. Here's what's on it, which course pages pair with each part, and how to read its numbers.
Start here (no maths needed)
A book about making AI answer fast and cheaply, turned into a website you can play with.
Think of it like this
Imagine a science textbook where every diagram moves, every formula has a calculator next to it, and every chapter ends with a quiz that marks itself. That's this site, for the book this course already follows in Module 13.
How it works
- Pick a path: "Getting Started" (about 3 hours), "Infrastructure Architect" or "Performance Optimizer" (about 4 hours each).
- Read a chapter, then play with its calculator. For example, see how much memory a model needs.
- Take the quiz at the end of the section.
- Use this course's pages when you want the "why" in more depth.
Why robots care
The site's memory and speed calculators answer the robot's question too: will this model fit on my robot's computer, and will it answer fast enough?
Try it
Open the site's VRAM calculator and pick a small model. Then open Module 09's KV-cache sizer and check you get similar answers.
Words to know
- VRAM
- The memory on a GPU.
- Benchmark
- A fair test that measures speed or quality.
- Cheat sheet
- A one-page summary of the key facts and formulas.
- Learning path
- A suggested order of chapters and exercises.
Going deeper: what's on the site, how it pairs with the course, and how to read its benchmark.
Track: LLM. This is the reading surface for Module 13's 100-day plan. Robotics links appear where models have to fit edge hardware and meet deadlines.
1. What's on the site
| Part | What it offers |
|---|---|
| 8 chapters | 0 Inference (three layers: runtime, infrastructure, tooling) · 1 Prerequisites · 2 Models · 3 Hardware (incl. Hopper, Blackwell, Rubin) · 4 Software (CUDA → vLLM/SGLang/TensorRT-LLM, NVIDIA Dynamo, benchmarking) · 5 Techniques · 6 Modalities · 7 Production |
| 3 learning paths | Getting Started (3 h, 8 steps) · Infrastructure Architect (4 h, 6 steps) · Performance Optimizer (4 h, 8 steps) |
| 7 calculators | VRAM · arithmetic intensity (roofline) · KV-cache sizing · quantisation quality estimator · speculative-decoding simulator · GPU selection advisor · model-to-hardware recommender |
| Quizzes | The site advertises 100+ questions with instant feedback |
| Cheat sheets | One per chapter (formulas, GPU specs, framework comparisons, checklists), downloadable as PDF |
| Glossary, guides, benchmarks | 50+ cross-linked terms, plus a vLLM vs SGLang benchmark (see section 3) |
| Progress tracking | Reading progress, quiz scores and exercises, tracked on the site |
2. Pairing it with this course
| Site | Course page with the deeper "why", or a second calculator |
|---|---|
| Ch 0–1 + the Prerequisites cheat sheet (Little's Law, MFU, cost per token) | 09 · request lifecycle, 09 · 1M-user capstone, 13 · journal notes |
| Ch 2 + VRAM and arithmetic-intensity calculators | 09 · KV-cache sizer, 09 · roofline demo, 05 · Kimi K3 |
| Ch 3 + GPU selection advisor | 09 · GPU specs from NVIDIA pages (with the sparsity caveat) |
| Ch 4 (engines, benchmarking) | 09 · fair engine benchmarking, 12 · serving video |
| Ch 5 + quantisation and speculative-decoding tools | 09 · speculative decoding, 09 · caching & disaggregation, 12 · video 04 |
| Ch 6 (VLMs, ASR, TTS, image/video) | 08 · multimodal & VLAs, 04 · on-device speech |
| Ch 7 (deploy, autoscale, observe) | 09 · rollouts & autoscaling |
Suggested use: follow Module 13's 100-day plan and read each chapter on this site. Run the site's calculator for that chapter, then cross-check the result with the matching Module 09 demo. If two calculators disagree, work out which assumptions differ (precision, overhead, usable-memory fraction). That's the most useful exercise of all.
3. Reading its benchmark critically
The site publishes a vLLM vs SGLang comparison. According to its page, it uses DeepSeek-R1-Distill-Llama-8B on 2× H100 SXM with tensor parallelism 2, CUDA 12.9, and 5,980 requests across six workloads (code generation, long RAG, medium chat, prefix-cache cold/warm, 8K RAG) at several concurrency levels. It reports TPS, TTFT, p50 and p99.
- The dates don't match: the page says "data as of 2025-12-05" and also "results collected June 2026". Check which version of each engine was tested before relying on it.
- It's one small model on one GPU type with one parallelism setting. Rankings can flip for larger or MoE models, other GPUs, or other configs.
- Winners vary by workload and load. On throughput (TPS), SGLang led medium chat at concurrency 100 (4,587 vs 4,432), while vLLM led medium chat at concurrency 50 and the long-RAG, code-generation and 8K-RAG rows shown. That's exactly why Module 09, Problem 18 says to benchmark your own traffic and name no universal winner.
- It's run by the site's team, not independently replicated. Treat it as a useful starting point.
4. Why it matters for robotics
- The VRAM and model-to-hardware tools answer "does this fit on the robot's board?". Remember that edge devices (e.g. Jetson-class) have far less memory bandwidth than data-centre GPUs. Module 09 page 06 notes the Jetson AGX Orin 64GB's 204.8 GB/s.
- Chapter 6 (ASR, TTS, VLMs) covers the modalities a robot actually talks and sees with.
- Latency percentiles (p99) matter more on robots than on websites: a rare slow answer is a rare late reaction.
5. Self-check
Which learning path fits an engineer tuning an existing deployment?
Performance Optimizer: quantisation, speculative decoding, KV caching and parallelism. About 4 hours, 8 steps.
Why cross-check the site's VRAM calculator with Module 09's KV-cache sizer?
Different tools make different assumptions (activation overhead, usable memory, KV precision). Explaining any gap is how you learn what drives memory.
Name two reasons not to generalise from the site's vLLM vs SGLang benchmark.
It uses one 8B model on 2× H100 with TP=2, and the winner changes by workload. The page's dates are also inconsistent.
6. Links
- inferenceengineering.tech: chapters · paths · calculators · cheat sheets · glossary · benchmarks · about
- Official book page: baseten.co/inference-engineering (digital and paper copies; also readable online)
- A second, code-first inference book with runnable labs: Module 16
- In this course: 13 · 100-day plan, 09 · 25 problems, 12 · videos, 05 · Kiely's Kimi K3 deep-dive