The LLM self-study stack: CS336, Build a Reasoning Model, the RLHF Book (and a gym plan)
A four-line answer to "what should young people do in the AI era?": build a language model from scratch, build a reasoning model, learn post-training properly, and train your body too. This module turns it into a plan and maps it onto the rest of the course.
Start here (no maths needed)
Want to really understand AI? Build it yourself, step by step, and keep your body strong while you do it.
Think of it like this
Learning to cook properly isn't watching cooking videos. It's making the dish yourself, many times, from basic ingredients. These three "books" are like a cooking course for AI:
- Course 1 (CS336): build the whole kitchen, from the oven up. Make a language model from scratch.
- Book 2 (Build a Reasoning Model): teach the dish to "think before it answers", like showing your working in maths.
- Book 3 (the RLHF Book): how to train the model to be helpful, using feedback, like a coach giving notes after practice.
- Number 4 is a gym plan. Long study needs energy, sleep and a healthy body.
How it works
- Pick one resource and do a little every day.
- Type the code yourself. Don't just read it.
- When something breaks, figure out why. That's where the learning happens.
- Move your body a few times a week, so your brain has energy for the hard parts.
Why robots care
The robot "brains" in this course are built on the same kind of model. Book 2 teaches a model to try several answers and check them, which is how a robot can try plans in its head before moving. Book 3 teaches learning from feedback, which is how robots learn what people prefer.
Try it
- Plan one week: three study sessions and three exercise sessions. Tick them off on paper.
- In the demo below, set Hours per week to what you could really do, and see how long each resource might take.
Words to know
- From scratch
- Building something yourself without using a ready-made version.
- Reasoning model
- A model that works through steps before giving an answer.
- RLHF
- Training a model using people's feedback about which answers are better.
- Training split
- A gym plan that works different muscles on different days.
Going deeper: the full study plan follows.
Track: LLM. All three resources are about language models. They feed robotics through VLAs, robot foundation models built on LLM backbones and post-trained in similar ways (Module 07).
1. Summary
TL;DR
Three resources that together cover a modern LLM end to end, plus advice to look after yourself:
- CS336 covers pre-training: tokenizer, Transformer, optimiser, GPU systems, scaling, data.
- Raschka covers reasoning: inference-time scaling, RL with GRPO, distillation.
- Lambert covers post-training: instruction tuning, reward models, RL, direct alignment.
They're ordered from foundation to frontier, and all three are code-first or have companion code.
| CS336 | Build a Reasoning Model | RLHF Book | |
|---|---|---|---|
| Author(s) | Tatsunori Hashimoto & Percy Liang (Stanford) | Sebastian Raschka | Nathan Lambert |
| Format | University course: lectures + 5 assignments | Book (Manning) + notebooks + a video course | Book (free web version; print edition July 2026) + code + course |
| Focus | Pre-training and systems | Reasoning on top of a pre-trained LLM | Post-training |
| Hardware | GPU rental for big runs (the site suggests debugging on CPU first) | Pure PyTorch, aimed at a laptop | Mostly reading; the code needs a GPU for real runs |
| Cost | Materials free; GPU time extra | Book purchase; code and video course free | Free online |
2. Stanford CS336: Language Modeling from Scratch
An implementation-heavy course that walks through building a language model end to end. The site warns it involves far more code than most AI classes, with minimal scaffolding. The five assignments, from the Spring 2025 site:
- Basics: implement the tokenizer, Transformer and optimiser, then train a minimal LM.
- Systems: profile and benchmark, write FlashAttention2 in Triton, and build memory-efficient distributed training.
- Scaling: fit a scaling law using a training API.
- Data: turn raw Common Crawl into pre-training data with filtering and deduplication.
- Alignment & reasoning RL: SFT and RL for maths reasoning, with optional DPO and safety alignment.
This course is already the spine of Module 08's course map, covering stages 1–6 and 11.
3. Build a Reasoning Model (From Scratch), by Sebastian Raschka
The sequel to his Build a Large Language Model (From Scratch). It starts from a pre-trained model and adds reasoning, in PyTorch, step by step. Chapters, from the book's repository:
- Understanding reasoning models
- Generating text with a pre-trained LLM
- Evaluating reasoning models
- Improving reasoning with inference-time scaling
- Inference-time scaling via self-refinement
- Training reasoning models with reinforcement learning
- Improving GRPO for reinforcement learning
- Distilling reasoning models for efficient reasoning
Appendices include Qwen3 source code, using larger LLMs, batching and throughput, LLM evaluation approaches, and a chat interface. Raschka also published a free video course based on the material.
4. The RLHF Book, by Nathan Lambert
A comprehensive introduction to post-training, organised around the canonical RLHF recipe. From the book's own description, it covers:
- why RLHF exists and its history
- an RL primer
- instruction tuning
- reward models
- rejection sampling
- RL
- on-policy distillation
- direct alignment algorithms (DPO and friends)
- advanced topics: synthetic data, tool use, character training, evaluation
It also traces RLHF's roots in economics, philosophy and optimal control, which matters for robotics. The site notes a print edition from July 2026, with a corrected reprint in October 2026.
This course has already used it as a reference in Module 07 and Module 08 Stage 05.
5. Number 4: the three-way training split (谭成义三分化)
The last "keyword" isn't about AI. It's a strength-training series by Chinese fitness creators 凯圣王 (Kai Sheng Wang) and 谭成义 (Tan Chengyi), published as a video collection on Bilibili. A "three-way split" works different muscle groups on different days. Their episodes cover a training plan, a chest/shoulders/triceps day, a back/rear-delts/biceps day and a legs day. Secondary write-ups describe it as push / pull / legs. The series says it suits beginners and keen trainees.
The point of putting it in a list of AI resources is clear even without the details. The three items above are months of hard, focused work, and sustaining that takes energy, sleep and health. Treat it as "make exercise part of the plan", not as fitness advice from this course. If you're new to strength training, learn the movements with a qualified coach.
6. A combined plan
A sensible order is CS336 assignments 1–2 → Raschka → CS336 assignments 3–5 alongside the RLHF Book. CS336 A1 gives you the model internals Raschka assumes. Raschka gives you hands-on reasoning RL before CS336 A5. Lambert gives the theory behind A5 and Module 07.
Interactive: how long will this take?
Every hour estimate is an assumption you can change. Neither the source post nor the resources give official hour counts. Run the plan in order, at your weekly study hours.
7. How it maps onto this course
| Resource part | Read alongside |
|---|---|
| CS336 A1 (basics) | 06 Day 13 Attention, Day 14, 08 Stage 01 |
| CS336 A2 (systems) | 08 Stage 06, 09 FlashAttention, 09 profiling |
| CS336 A3–A4 (scaling, data) | 06 Day 25, 08 Stage 02 |
| Raschka ch. 4–5 (inference-time scaling) | 10 RLM harnesses (test-time compute as a program), 09 latency |
| Raschka ch. 6–7 (RL, GRPO) | 07 RLVR, 06 Day 29 PPO, 02 RL ladder |
| Raschka ch. 8 (distillation) | 04 Teacher → student |
| RLHF Book | 07, 08 Stage 05, 06 Day 30, 01 (reward models are classifiers) |
8. Why it matters for robotics
- VLAs are LLMs plus actions. Everything in CS336 A1–A2 (Transformer internals, efficient training) is the base of robot foundation models.
- Inference-time scaling = thinking before acting. Sampling and checking several candidate plans before moving is the robot version of Raschka's chapters 4–5. It trades latency for reliability (Module 04).
- GRPO-style RL with verifiers maps onto robot RL with success detectors (Module 07).
- RLHF has roots in optimal control, the classical robotics discipline. Lambert's book traces that history.
- Distillation (Raschka ch. 8) is how big reasoning models get onto small robot computers.
9. Self-check
Which resource covers pre-training, which covers reasoning, and which covers post-training?
CS336 covers pre-training and systems, Raschka covers reasoning, and Lambert covers post-training.
Why do CS336 A1 before Raschka?
Raschka starts from a pre-trained model. A1 makes you build the tokenizer, Transformer and optimiser, so the internals aren't a black box when you add reasoning on top.
What is GRPO, and where else in the course does that family of ideas appear?
An RL method that scores each sampled answer against its group's average, so it needs no separate value model. See Module 07 (RLVR) and Module 08 Stage 03/05.
What's the point of item 4?
It's a strength-training plan. Months of deep study take energy and health, so make exercise part of the plan too.
10. Links
- Stanford CS336, Spring 2025 (lectures, assignments, schedule)
- Build a Reasoning Model (From Scratch): code repository · Raschka's free video course
- RLHF Book (free web version, code and course links)
- 凯圣王 × 谭成义 three-way split series (Bilibili)
- Earlier Dongxi module: 07, post-training. His series promises deeper posts on each technique.