LLMModule 16 · Hands-on labs

Hands-On LLM Serving and Optimization: the book's code as a lab track

Chi Wang and Peiheng Hu's O'Reilly book comes with a repository of notebooks and services, from a serving system built from scratch to a full Qwen3-14B optimisation case study. This module turns that repo into a lab sequence and pairs each lab with the course pages that explain it.

Source: orca3/llm-model-inference on GitHub (companion code for Hands-On LLM Serving and Optimization: Hosting LLMs at Scale, O'Reilly), read 4 Nov 2026 · book website · O'Reilly page. The repo has no licence file and asks readers to refer to the book, so it's summarised and linked here, not copied. READMEs saved in sources/16-*.

Start here (no maths needed)

A book that shows you how to build the "kitchen" that serves AI answers, with real code you can run yourself.

Think of it like this

Think of a pizza shop. Orders come in (requests). Someone at the counter takes them (the web server). A manager groups orders so the oven is always full (batching). The oven bakes them (the AI model on a GPU), and slices go out as soon as each one is ready (streaming). This book builds a small version of that shop in code, then shows how the big shops make it fast and cheap.

How it works

Start with tiny examples that run on a normal computer, to see how a model produces words one by one.
Build your own small "shop": a server that takes requests, groups them and streams answers.
Swap in professional engines (vLLM, SGLang, TensorRT-LLM, llama.cpp).
Try speed tricks: smaller numbers (quantisation), guessing ahead (speculative decoding), reusing work (caching).
Finish with a real case study that tunes a large model step by step.

Why robots care

The book's first chapter includes "edge" serving, running a model on a device itself, and its last chapter covers edge AI. A robot is an edge device. It can't always wait for a server far away.

Try it

In the demo below, raise Max wait. Watch the batches get bigger and the oven work more efficiently, but each customer waits longer. That's the trade-off every server must choose.

Words to know

Serving: Running a model so people (or programs) can send it requests.
Batching: Grouping several requests to run together on the GPU.
Streaming: Sending the answer word by word as it's made.
Notebook: A document mixing code you can run with explanations.
Edge: Running AI on the device itself, not in a data centre.

Going deeper: the book map, the lab track, the from-scratch server's design, and practical gotchas.

Track: LLM. It connects to robotics through the book's edge-serving material (chapters 1 and 10) and llama.cpp (chapter 8), which is how models run on small, local hardware.

In this module

Summary
The book, chapter by chapter
The lab track
Inside the from-scratch server
Dynamic batching (interactive)
The Chapter 9 case study as a method
Two inference books: which to read when
Practical gotchas
Why it matters for robotics
Self-check

1. Summary

TL;DR

The repo holds code for chapters 2–4 and 6–9 of a 10-chapter O'Reilly book (chapters 1 and 5 are conceptual). Highlights:

notebooks on transformer internals and the KV cache
a single-model LLM server written from scratch (FastAPI, batching, streaming, a vLLM backend) and a multi-model server, including an NVIDIA Triton example
a RAG-plus-planning "Knowledge Agent" and AWS deployment examples
quantisation, speculative-decoding and LMCache notebooks
SGLang, TensorRT-LLM and llama.cpp notebooks
an end-to-end Qwen3-14B optimisation plan with vLLM

Most labs need an NVIDIA GPU, and the authors suggest Google Colab if you don't have one.

2. The book, chapter by chapter

Chapter topics are from the repo README. The right-hand column is where this course already explains each topic.

Chapter	Topics (per the README)	In this course
1 Introduction to model serving	Model anatomy and lifecycle, why optimise, serving paradigms (edge, single-model, multi-model), platforms	09 hub, 13 · book map
2 LLM serving	Transformer internals, autoregressive generation, attention, KV cache, prefill/decode, running vLLM	09 · lifecycle, 09 · KV cache, 06 · Day 13
3 Serving system design	Single- and multi-model services from scratch, batching, streaming, Triton, cost vs latency designs	Section 4 below, 09 · batching
4 Best practices	Serving for agents (RAG, CAG), enterprise architecture, open-source vs cloud vendor, build or buy, performance measurement	10 · agent harnesses, 09 · observability
5 Challenges	Accelerators and GPU specs, model-loading bottlenecks, KV sizing, arithmetic intensity of prefill vs decode	09 · GPUs, 09 · roofline
6 Essential optimisations	Dynamic/continuous batching, chunked prefill, scalable attention and fusion, quantisation, distillation, pruning, prefix caching and RadixAttention	09 · 03, 09 · 04, 04 · distillation, 12 · serving video
7 Advanced optimisations	Speculative decoding, multi-GPU and multi-node (data/tensor/pipeline/expert), prefill–decode disaggregation, advanced KV caching	09 · 05, 12 · 5D parallelism, 12 · video 04
8 Serving frameworks	vLLM internals, TensorRT-LLM, SGLang, llama.cpp, choosing a framework	09 · Problem 18, 15 · benchmark caveats
9 Optimisation in practice	Qwen3-14B with vLLM: hardware inspection, benchmarking, quantisation, distributed serving, trade-offs	Section 6 below, 09 · Problem 24
10 Advancements	Semantic caching, profiling, multimodal, edge AI, multi-LoRA serving, serving inside RL	12 · LoRA, 08 · multimodal, 07 (RL needs fast rollouts)

3. The lab track

A suggested order, from the repo folders. Hardware notes are mine. Check each folder's README and requirements.txt before running.

#	Lab (repo folder)	What you do	Likely hardware	Pair with
1	`ch02/` notebooks	Step through transformer execution, the KV cache, streaming and batching basics, then run Qwen on vLLM	CPU for the internals; GPU for vLLM	09 · KV sizer
2	`ch03/single_model_llm_serving`	Run and read a FastAPI server with a workload manager, batching, streaming and a vLLM endpoint (uses `facebook/opt-125m`)	Small model, so likely CPU-friendly for the non-vLLM paths	Section 4, 09 · Problem 2 (SSE)
3	`ch03/multi_model_serving`	Serve several models behind one service, including a Triton DenseNet example	GPU helpful	09 · Problem 21 (multi-tenant)
4	`ch04/KnowledgeAgent`, AWS folders	A RAG + planning agent; Bedrock, JumpStart and Deep Learning Container deployments	API keys / AWS account (costs money)	10 · harnesses
5	`ch06/` quantisation notebook	Compare precisions against throughput	GPU	09 · Problem 11
6	`ch07/` SpecDecode, LMCache	Hands-on speculative decoding and advanced KV caching	GPU	09 · Problem 10, 12 · video 04
7	`ch08/` SGLang, TensorRT-LLM, llama.cpp	Run the same model on three engines	GPU (llama.cpp can run on CPU)	09 · Problem 18
8	`ch09/` optimisation plan	Benchmark Qwen3-14B with vLLM, then quantise, distribute and analyse the trade-offs	NVIDIA GPU with at least 24 GB (46 GB+ recommended), 32 GB+ RAM, 100 GB+ disk (per the README)	Section 6

4. Inside the from-scratch server

The chapter 3 single-model service is a good first serving codebase to read, because its parts mirror the ones in production engines. From its README:

HTTP request
  └─ main.py           API layer: FastAPI endpoints, validation, streaming responses
      └─ LLMEngine      orchestration: runs the processing loop, picks the backend
          ├─ WorkloadManager   queues requests (stream vs batch), forms batches, tracks sequences
          └─ ModelExecutor     manages a separate worker process and its queues
              └─ ModelWorker   runs inference (transformers) in that process: CPU/GPU, token generation
                  └─ ModelManager  loads and caches models and tokenizers

Why a separate worker process? The model holds the GPU and runs heavy compute. Isolating it keeps the API responsive, and a crash in inference doesn't take down the web server. Production engines split work the same way (API server, scheduler, model workers).
Where batching lives: in the workload manager, the "scheduler" of Module 09 page 03. Reading it before reading vLLM's scheduler makes vLLM much easier to follow.
Two backends behind one API (plain transformers, and vLLM) let you measure the gap a real engine buys, on your own machine.

5. Dynamic batching

Before continuous batching, the classic server policy is dynamic batching: hold requests until either the batch is full or the oldest has waited T ms, then run them together. Bigger batches use the GPU better, but every request pays the wait.

Interactive: max-wait vs latency and throughput

A simulated stream of requests with random arrivals. A batch takes base + per-request × size ms. The batch launches when it's full or when the oldest request has waited Max wait. All timings are illustrative assumptions.

Arrival rate (requests/s): Max wait T (ms): Max batch size:

Mean batch size

–

Mean latency

–

p95 latency

–

GPU busy

–

The chart sweeps Max wait from 0 to 200 ms at the current rate and batch size: mean latency (line) and mean batch size (bars). The marker shows your setting. With base 30 ms + 4 ms per request, one request at a time (batch size 1) tops out near 29 requests/s. Raise the rate past that with batch size 1, then let batching rescue it.

Continuous batching (Module 09 page 03) removes most of this trade-off for LLMs. It admits new requests at every decode step instead of waiting to form a batch. That's why modern engines use it.

6. The Chapter 9 case study as a method

The ch09 guide describes an end-to-end workflow for Qwen3-14B on vLLM. The parts worth copying into your own work:

Inspect the hardware (GPU, VRAM) before choosing settings.
Inspect the workload. The repo's inspect_dataset.py prints prompt and output length distributions for ShareGPT, random and "prefix repetition" datasets. Its example output shows a heavily skewed distribution, with many very short prompts and a long tail. That shape drives batching and caching choices.
Benchmark a baseline on realistic data (ShareGPT conversations, plus a prefix-repetition set to test caching).
Change one thing at a time (quantisation, distributed serving, caching) and re-measure.
Write up the trade-offs, not just the best number. That's the same discipline as Module 04's honest results and Module 08 Stage 11.

7. Two inference books: which to read when

	Inference Engineering (Kiely, Baseten)	Hands-On LLM Serving and Optimization (Wang & Hu, O'Reilly)
Shape	8 chapters, stack-wide: prerequisites → models → hardware → software → techniques → modalities → production	10 chapters, code-backed: from-scratch serving systems → optimisations → frameworks → a case study
Companion	Interactive site with calculators and quizzes (Module 15)	This repo: notebooks and services
Best for	The mental model and production decisions	Building and running things
In this course	Module 13 100-day plan	This module's lab track

Suggestion: follow Module 13's plan with Kiely for reading, and slot in labs from section 3 as the "build" for each block. For example, run lab 2 during Kiely's chapter 4 (software), and labs 5–6 during chapter 5 (techniques).

8. Practical gotchas

The vLLM version pin is old. The chapter 9 guide suggests git checkout v0.6.0 ("or latest stable version"). vLLM moves fast, and flags and APIs change between versions. Prefer a current release and adjust the notebook.
Dataset size: the guide calls the ShareGPT download ~1.5 GB in one place and ~650 MB in another. Check the actual file.
Git LFS: model weights and large artefacts are tracked with Git LFS and may need a separate git lfs pull.
Cloud examples cost money: the AWS Bedrock, JumpStart and DLC notebooks run real billable resources. Tear them down afterwards.
The chapter 9 notebook is meant to run inside a vLLM checkout (copy it, and its helper scripts, into the vLLM repo root).

9. Why it matters for robotics

Edge serving is one of the book's three paradigms (with single-model and multi-model), and edge AI returns in chapter 10. A robot is an edge deployment: limited memory and power, no network guarantee.
llama.cpp (chapter 8) is a common way to run quantised models on CPUs and small devices. It's a practical route to on-robot language models.
Multi-LoRA serving (chapter 10) means one base model with many small adapters. That maps to one robot foundation model with per-task or per-robot adapters.
"Model serving in RL" (chapter 10): RL training needs fast rollouts. Serving speed sets how quickly robot policies (and LLM reasoning models) can be trained (Module 07).
The dynamic-batching trade-off is sharper on robots: a controller can't wait 50 ms for a batch to fill. Batch size 1 with a tight deadline is common.

10. Self-check

Why does the chapter 3 server run the model in a separate process?

To isolate heavy GPU work from the API layer. The server stays responsive, and an inference crash doesn't kill the web front end.

In dynamic batching, what does raising the max wait do?

Batches get bigger, so GPU efficiency and throughput go up. But each request waits longer, so latency rises, especially at low traffic.

Why does continuous batching beat dynamic batching for LLMs?

LLM requests generate many tokens of varying length. Continuous batching admits and retires requests at every decode step, so nobody waits for a batch to form and slots don't sit idle while long requests finish.

What should you check before trusting the chapter 9 benchmark setup today?

The vLLM version (the guide suggests v0.6.0), the dataset file actually downloaded, and that Git LFS artefacts are present.

Which two book chapters matter most for on-robot inference?

Chapter 1 (the edge serving paradigm) and chapter 10 (edge AI, multi-LoRA). Chapter 8's llama.cpp section is the practical tool.

11. Links

Companion repo · book website · O'Reilly
Engines: vLLM docs · llama.cpp · LMCache
In this course: 09 · 12 · serving video · 13 · 15