LLMModule 08 · Career roadmap
The LLM research-engineer roadmap: 12 stages, from autograd to open-source teardowns
A popular 12-stage plan for becoming an AI research engineer. Each stage gets its own page: the concepts explained, a project you can actually run, a reality check, links to where this course already covers it, and the real courses that teach it (course map).
Start here (no maths needed)
A research engineer turns new AI ideas from papers into working, fast, well-tested models. This roadmap lists the 12 skills they use.
Think of it like this
Think of a Formula 1 team. Some people invent new ideas for the car (researchers). Others build the engine, test it on the track, make it faster and fix what breaks (engineers). A research engineer does a bit of both: reads the new idea, builds it, races it, and reports honestly how it did.
How it works
- Know the engine: the maths and code inside a neural network (stage 1).
- Feed it well: choose how big to make it and what data to train it on (stages 2 and 4).
- Try new designs: new kinds of model, new ways to see pictures and hear sound (stages 3 and 7).
- Teach it manners, and make it fast: make it behave well, train it on many computers at once, and make it quick to use (stages 5, 6 and 8).
- Look inside, test hard and share: find out how it thinks, attack it to find weak spots, and publish your work (stages 9–12).
Why robots care
Robot "brains" are now built from the same parts. Stage 7, joining a vision model to a language model, is exactly how robots that follow spoken instructions are made. Stage 8, making models fast, decides whether the robot reacts in time.
Try it
Pick one stage below that sounds interesting and read only its Start-here box. Then come back and tick it off in the tracker.
Words to know
- Research engineer
- Someone who builds and tests new AI ideas in code.
- Training
- Teaching a model by showing it lots of data.
- Inference
- Using a trained model to answer questions.
- GPU
- A chip that does the huge amounts of maths AI needs, very fast.
Going deeper: the rest of this page is the full roadmap. It's mostly LLM track, and a few stages are marked LLM + Robotics where they carry straight over to robot learning.
Progress
Done
0 / 12
Next up
–
The 12 stages
01
PyTorch internals, hand-written backward passes, Triton kernels, memory layout and strides.
02
Compute-optimal model size, deduplication, domain weights, learning-rate schedules.
03
Mixture of experts, state-space models (Mamba/S4), linear attention, ring attention.
04
Self-instruct, rejection sampling, verifier filtering, training a small model on the survivors.
05
DPO, KTO, ORPO, reward models, and how to stop reward hacking.
06
Data, tensor and pipeline parallelism, ZeRO/FSDP, NCCL, gradient checkpointing.
07
Vision encoders, projection layers, cross-attention, early vs late fusion. The core of a VLA.
08
GQA/MQA, KV-cache size and quantisation, speculative decoding.
09
Logit lens, activation patching, finding the circuit behind a fact.
10
Contamination checks, dynamic evals, calibrated LLM judges, attacking your own models.
11
Reproduce a new paper's core idea in under 500 lines and benchmark it honestly.
12
Model cards, public training logs, and honest write-ups of what didn't reproduce.
Courses & content for each stage
Real courses, books, docs and repos you can follow, with each one mapped to the stages it teaches. All links were checked on 22 Oct 2026. Each stage page repeats its own list.
Spine courses (cover several stages)
| Course | Stages | Why |
|---|---|---|
| Stanford CS336 · Language Modeling from Scratch | 01 02 03 04 05 06 11 | The closest thing to the whole roadmap: five low-scaffolding assignments (basics, systems, scaling, data, alignment). Needs GPU rental for the big runs. |
| Karpathy · Zero to Hero + nanoGPT + nanochat | 01 02 04 11 | Free, gentle on-ramp: backprop by hand → GPT-2 reproduction → a small end-to-end chat model. |
| Hugging Face LLM Course + smol-course | 04 05 07 10 12 | Practical fine-tuning, alignment, VLMs, evaluation and sharing on the Hub, on small models. |
| Ultra-Scale Playbook + How To Scale Your Model | 06 08 | The two best free references on parallelism, memory and inference cost. |
| MIT 6.5940 · TinyML & Efficient Deep Learning | 08 | Quantisation, pruning and edge deployment. The bridge to on-robot models. |
| ARENA | 09 | The standard hands-on mech-interp curriculum. |
| AI Evals for Engineers & PMs (Husain & Shankar) | 10 | Paid. Error analysis and judges, from the authors behind Module 01. |
| Hugging Face Robotics Course + LeRobot | 07 (robotics) | The robotics-track companion: from classical robotics to learned policies on hardware. |
Stage by stage
Stage 01 · Tensor maths & custom autograd
| Resource | Type | Use it for |
|---|---|---|
| Stanford CS336 · Assignment 1 (Basics) + Assignment 2 (Systems) | University course (free materials) | Build tokenizer, Transformer and optimizer from scratch; then write FlashAttention2 in Triton and profile it |
| Karpathy · Neural Networks: Zero to Hero | Video course (free) | Backprop by hand with micrograd, then up to a GPT |
| PyTorch docs · Extending PyTorch (custom autograd.Function) | Official docs | Writing your own forward/backward |
| Triton tutorials | Official tutorials | Vector add → fused softmax → matmul → fused attention |
| GPU MODE lectures | Lecture series (free) | Profiling, CUDA and Triton kernels, memory layout |
Stage 02 · Scaling laws & data mixing
| Resource | Type | Use it for |
|---|---|---|
| Stanford CS336 · Assignment 3 (Scaling) + Assignment 4 (Data) | University course (free materials) | Fit a scaling law; turn Common Crawl into training data with filtering and dedup |
| FineWeb: decanting the web for the finest text data | Long-form guide | Dedup (MinHash), quality filtering and the ablations behind each choice |
| datatrove | Library | The data-processing pipeline used for FineWeb |
| Karpathy · build-nanogpt (reproduce GPT-2 124M) | Video + code (free) | A full small pre-training run with LR warmup and cosine decay |
| Karpathy · nanochat | Code (free) | End-to-end small ChatGPT: pre-train, mid-train, SFT, RL, on a small budget |
Stage 03 · Next-gen architectures: MoE, state-space models, linear & ring attention
| Resource | Type | Use it for |
|---|---|---|
| Stanford CS336 lectures | University course (free materials) | Architecture variants and mixture of experts |
| Hugging Face · Mixture of Experts Explained | Blog guide | Routing, load-balancing loss, capacity factor |
| A Visual Guide to Mixture of Experts | Illustrated guide | MoE intuition with diagrams |
| A Visual Guide to Mamba and State Space Models | Illustrated guide | From RNNs to S4 to Mamba's selective scan |
| The Annotated S4 | Annotated code | S4 implemented step by step |
| Stanford CS25 · Transformers United | Seminar talks (free) | Talks from the people building new architectures |
Stage 04 · Synthetic data & curation pipelines
| Resource | Type | Use it for |
|---|---|---|
| Hugging Face smol-course · Synthetic Data unit | Course (free) | Generating synthetic datasets for a domain |
| Cosmopedia: how to create large-scale synthetic data | Case study | Prompt design, diversity and dedup for a large synthetic pre-training set |
| distilabel | Library + docs | Pipelines for generating, judging and filtering synthetic data |
| Karpathy · nanochat | Code (free) | How synthetic and curated data feed the later training stages |
Stage 05 · Advanced alignment & preference optimisation
| Resource | Type | Use it for |
|---|---|---|
| Nathan Lambert · RLHF Book | Free online book | Reward models, PPO, DPO-family methods, over-optimisation |
| Hugging Face TRL · DPO / KTO / ORPO trainers | Official docs | Run each method on a small model |
| DeepLearning.AI · Post-training of LLMs | Short course | SFT, DPO and online RL, hands-on |
| DeepLearning.AI · Reinforcement Learning from Human Feedback | Short course | The classic reward-model + RL pipeline |
| Hugging Face smol-course · Preference Alignment unit | Course (free) | DPO in practice on a small model |
| Stanford CS336 · Assignment 5 (Alignment & reasoning RL) | University course (free materials) | SFT and RL for maths reasoning; optional DPO |
Stage 06 · Distributed training & systems ML
| Resource | Type | Use it for |
|---|---|---|
| Hugging Face · The Ultra-Scale Playbook | Free long-form guide | Data, tensor, pipeline, context and expert parallelism, ZeRO, with measured benchmarks |
| Google DeepMind · How To Scale Your Model | Free online book | Rooflines, sharding and communication cost, worked examples |
| PyTorch · Getting started with FSDP | Official tutorial | Sharded data-parallel training |
| DeepSpeed · ZeRO tutorial | Official tutorial | ZeRO stages 1–3 and offload |
| Stanford CS336 · Assignment 2 (Systems) | University course (free materials) | Build memory-efficient distributed training yourself |
Stage 07 · Multimodal fusion & projection
| Resource | Type | Use it for |
|---|---|---|
| Hugging Face · nanoVLM | Code (free) | A small VLM: vision encoder + projection + language model, trainable on one GPU |
| Hugging Face · Vision Language Models Explained | Blog guide | Architectures and how to fine-tune them |
| Hugging Face smol-course · Vision Language Models unit | Course (free) | Adapting and using multimodal models |
| LLaVA project | Paper + code | The classic 'vision encoder + projector + LLM' recipe |
| Hugging Face · Robotics Course + LeRobot | Course (free) | Robotics track: from classical robotics to learned policies on real hardware |
| OpenVLA | Paper + code | Robotics track: a VLM turned into a robot policy that outputs actions |
Stage 08 · Inference-aware model design
| Resource | Type | Use it for |
|---|---|---|
| Google DeepMind · How To Scale Your Model (inference chapters) | Free online book | KV-cache maths, batching and latency vs throughput |
| Hugging Face · KV cache explained | Blog guide | What the KV cache stores and why it dominates memory |
| vLLM docs | Official docs | Paged KV cache, quantisation, speculative decoding in a real server |
| DeepLearning.AI · Efficiently Serving LLMs | Short course | KV caching, batching, quantisation, LoRA serving |
| MIT 6.5940 · TinyML and Efficient Deep Learning Computing | University course (free lectures) | Quantisation, pruning, efficient attention, on-device and edge, which matters for robots too |
| Philip Kiely · Inference Engineering (book) | Book | Production inference from the Baseten side (see Module 05) |
Stage 09 · Mechanistic interpretability
| Resource | Type | Use it for |
|---|---|---|
| ARENA · Transformer interpretability chapter | Course (free materials) | Hands-on exercises: logit lens, attribution, activation patching, circuits |
| ARENA 3.0 materials (GitHub) | Exercises (free) | The notebooks themselves |
| Neel Nanda · Getting started in mechanistic interpretability | Guide | Reading list and learning path |
| TransformerLens docs | Library + docs | Hooking and patching activations |
| Transformer Circuits Thread | Research publications | The primary research on circuits and features |
Stage 10 · Evaluation-driven development & red teaming
| Resource | Type | Use it for |
|---|---|---|
| Hamel Husain & Shreya Shankar · AI Evals for Engineers & PMs | Paid cohort course | Error analysis, LLM judges and evals that drive product decisions (the authors behind Module 01) |
| UK AISI · Inspect | Eval framework + docs | Write reproducible evals, including agentic ones |
| EleutherAI · lm-evaluation-harness | Eval framework | Standard benchmark runs |
| Hugging Face · Evaluation Guidebook | Free guide | Practical evaluation, including contamination and judges |
| Microsoft PyRIT | Red-teaming toolkit | Automated adversarial testing of your own GenAI system |
| NVIDIA garak | LLM vulnerability scanner | Probe your own model for known failure categories |
Stage 11 · Paper-to-production prototyping
| Resource | Type | Use it for |
|---|---|---|
| labml.ai annotated paper implementations | Annotated code (free) | Many papers implemented with side-by-side notes |
| Karpathy · nanoGPT | Code (free) | A minimal, hackable baseline to drop new ideas into |
| 30u30 (this course's Module 06) | Exercises (free) | Thirty from-scratch reproductions to practise on |
| Stanford CS336 | University course (free materials) | Low-scaffolding assignments: the closest thing to lab work |
Stage 12 · Open-source portfolio & teardowns
| Resource | Type | Use it for |
|---|---|---|
| Hugging Face Hub · Model cards | Official docs | What a rigorous model card contains |
| Hugging Face LLM Course | Course (free) | Fine-tune, then share models and datasets on the Hub |
| Weights & Biases · Reports | Official docs | Publishing training logs and comparisons |
| fast.ai · Practical Deep Learning | Course (free) | Strong emphasis on shipping and writing about your work |
Where this course already covers it
| Stage | Already in the course |
|---|---|
| 01 Tensor maths | 02 TensorTonic (CUDA/Triton tracks), 06 Day 1 (backprop through time by hand) |
| 02 Scaling & data | 06 Day 25 Scaling laws, 04 (data filtering, honest splits) |
| 03 Architectures | 05 Kimi K3 (MoE routing, linear attention), 06 Day 13 |
| 04 Synthetic data | 04 Teacher → student, 07 RLVR verifiers |
| 05 Alignment | 07 Preferences & the KL leash, 06 Day 30, 06 Day 29 |
| 06 Distributed | 06 Day 24 GPipe, 05 serving |
| 07 Multimodal | 05 (K3 trains its vision encoder from scratch), 06 Day 28 |
| 08 Inference | 05 (MLA, KV cache, speculative decoding), 04 latency |
| 10 Evals | 01 LLM evals & calibration, 04 evaluation that doesn't fool you |
| 11–12 Reproduce & publish | 04 (an honest model write-up), 06 (reading papers) |
Reality check
A good map, with some hype. The topics are the real core of LLM research engineering in 2026, and the projects are good practice. A few claims need caveats:
- "Never be unemployed again" is marketing. No skill set guarantees that.
- "Architecture is secondary, data is the secret sauce" overstates it. Data mix matters enormously, but frontier models still compete on architecture too. Module 05 credits Kimi K3's new architecture with about 2.5× scaling efficiency.
- "Ring Attention for infinite context": ring attention spreads a long sequence across devices, so the maximum length grows with the number of GPUs. That's very long, not infinite.
- Several projects assume real hardware (a 7B model across 4 GPUs, 50,000 reasoning traces from a frontier model). Each stage page gives a laptop/Colab-sized version first, and the full version second.
- "Labs hire on HuggingFace repos, not degrees": public work helps a lot, but it isn't the only route.
Suggested order
- Foundations first: 1 → 2 → 8. You can't optimise what you can't see, and inference cost shapes everything.
- Then the model: 3 → 7 → 6.
- Then behaviour: 4 → 5 → 10 → 9.
- Throughout: 11 and 12 aren't stages to "finish". Do one paper reproduction and one public write-up per stage.