LLMs & Robotics
A personal course on two connected fields, built from links as they're added. Each module has a summary, deep dives, a self-check and further reading.
LLM how large language models are built, trained, evaluated and served. Robotics how robots perceive, decide and act. LLM + Robotics shared foundations, or ideas that carry across both. Robot foundation models (VLAs) are built on LLM backbones, so the two tracks keep meeting.
Sources & summaries → every link added, its summary, the primary sources behind it, saved files and all external references.
Modules
LLMLLM evals with decision models: thresholds, trade-offs and calibration
Why the default 0.50 cut-off leaves performance on the table, how to tune thresholds against minimum precision/recall with cross-validation, compare models on cost, and check whether scores are real probabilities. Includes an interactive threshold demo and a robotics lens (perception, safety stops, policy evals).
evaluationclassificationcalibrationLLM + RoboticsTensorTonic: hands-on ML practice for LLMs and robotics
What the 1,300-problem ML coding gym offers and a robotics-specific path through it: maths foundations → 3D geometry & perception → the RL ladder up to PPO → transformers and diffusion (VLA groundwork) → on-robot inference. Includes an interactive two-link arm built from chained homogeneous transforms, and a list of what it doesn't cover.
practicekinematicsreinforcement learningperceptionLLM + RoboticsHow much to help: human–AI and human–robot interaction
An AI tutor for kids (gakku) argues the struggle is the product, which is the same "how much to help" problem assistive robots face. Covers levels of automation, policy blending, Bayesian intent inference when the human changes their mind, ask/act/wait as expected cost, and how not to automate your own learning. Includes an interactive shared-autonomy reach simulation.
HRIshared autonomyintent inferencelearningLLM + RoboticsBig teacher, small student: distilling fast on-device models for apps and robots
Log a big model's outputs on your real task, then fine-tune a tiny one. Rivest's 32M speech model tied a 627M teacher on his voice and answers in ~85 ms instead of ~1 s on a laptop CPU. Covers distillation (and the privileged-teacher version used for legged robots), honest evaluation (day splits, drift cutoffs, paired bootstrap), and pitfalls that transfer to robot data. Includes an interactive "how far does the robot travel after you say stop" calculator.
distillationedge AIdata flywheellatencyLLMLLM architecture at trillion scale: width, depth and length in Kimi K3
Kimi K3 as a case study (via a post quoting Baseten's inference deep-dive). Covers Stable LatentMoE routing 16 of 896 experts with Quantile Balancing, the 3:1 hybrid of fixed-memory linear attention (KDA) and global attention (MLA) for 1M tokens, Attention Residuals across depth, and the serving consequences. Robotics angle: fixed-size memory for streaming sensors, MoE for multi-skill policies. Includes an interactive expert-routing balancer.
architecturemixture of expertslinear attentioninferenceLLM + RoboticsIlya's 30 papers: the shared foundations of LLMs and robotics
A hub plus one page per paper, from RNNs, LSTMs and MDL through AlexNet/ResNet, the Transformer, pointer and graph networks, scaling laws, PPO and RLHF. Every day opens with a Start here box written for a 13-year-old (analogy, simple steps, a try-it activity, words to know), then goes deeper: core maths, mechanism, robotics angle, self-check and an interactive demo. Includes a 12-day robotics fast track and a progress tracker. Code and exercises are in the 30u30 repo.
foundationsdeep learningtransformersRLbeginner-friendlyLLM + RoboticsLLM post-training: demonstrations, comparisons and checkers (and the robot equivalents)
Why a model that has "read the internet" still can't help, and the three post-training teachers that fix it: instruction tuning (copy a demonstration), preference alignment (RLHF/PPO or DPO, with a KL leash) and RLVR (a verifier checks the answer). Mapped onto robot training: behaviour cloning, human-ranked rollouts and success detectors. Opens with a Start-here box. Includes an interactive KL-leash / reward-hacking demo.
post-trainingRLHFbehaviour cloningreward hackingbeginner-friendlyLLMThe LLM research-engineer roadmap: 12 stages, from autograd to open-source teardowns
A hub plus one page per stage: custom autograd and Triton, scaling laws and data mixing, MoE/SSMs/ring attention, synthetic data, DPO/KTO/ORPO, 3D parallelism and ZeRO, multimodal projection (the core of VLAs), GQA and KV-cache design, mechanistic interpretability, evals and red teaming, paper reproduction, and public portfolios. Each stage has a Start-here box, the concepts explained, a laptop-sized project plus the full version, a reality check, and links to where earlier modules already cover it.
careerresearch engineeringsystemsinterpretabilitybeginner-friendlyLLMLLM inference engineering: 25 interview problems, worked through with real numbers
A hub plus ten pages covering every problem: the request lifecycle and streaming, KV-cache sizing (70B × 32K × 100 users), PagedAttention and FlashAttention, continuous batching and chunked prefill, prefix caching and prefill/decode disaggregation, speculative decoding and tail latency, FP8/INT8/INT4/NVFP4 and GPU choice, profiling and a fused Triton kernel, observability and engine benchmarks, rollouts and autoscaling, MoE serving, cost cutting and a 1M-user capstone. Each has a model interview answer and an interactive calculator.
inferenceservingKV cacheGPUsinterview prepbeginner-friendlyLLM + RoboticsSelf-improving agent harnesses: Recursive Language Models, Continual Harness and Prime Agent
Hold context in REPL variables and let the model write code over it and call copies of itself (RLMs). Let the agent make small, reversible, evidence-backed edits to its own prompts, skills, memory and sub-agents without resets (Continual Harness, born from embodied Pokémon agents). Prime Agent combines both. Covers results and how to read them, safety bounds, and the robotics link: reset-free learning, skills libraries, querying robot logs with code. Includes an interactive recursion-tree cost demo.
agentslong contextself-improvementembodied agentsbeginner-friendlyLLMThe LLM self-study stack: CS336, Build a Reasoning Model, the RLHF Book (and a gym plan)
Dongxi's four-keyword answer to "what should young people do in the AI era?", turned into a plan: Stanford CS336 for pre-training and systems, Raschka's reasoning book for inference-time scaling, GRPO and distillation, Lambert's RLHF Book for post-training, and a three-way strength-training split. Chapter-by-chapter mapping onto modules 01–10 and an adjustable time-to-complete planner.
study pathbooksreasoningpost-trainingbeginner-friendlyLLM + RoboticsAI training and inference, in nine videos: watch guides for Jia-Bin Huang's playlist
A hub plus one watch guide per video: ZeRO & FSDP, 5D parallelism, LoRA/QLoRA/DoRA, speculative decoding, on-policy distillation, REINFORCE→PPO→GRPO, AdamW, Muon, and how LLM serving works (vLLM, SGLang). Each guide embeds the video, maps every chapter with timestamps, explains the ideas from the original papers, and links to where the course goes deeper. Suggested learning order and a progress tracker.
video coursetrainingoptimisersservingbeginner-friendlyLLM100 days of inference engineering: a learning journal, the book behind it, and a plan you can follow
A new public journal working through Philip Kiely's Inference Engineering (Baseten). Maps all eight book chapters onto modules 01–12, summarises the journal's first entries, and lays out a 100-day plan with a hands-on build per chapter, plus a clickable 100-day tracker and tips for learning in public.
learning in publicinferencebooksstudy planbeginner-friendlyLLM + RoboticsDeep-ML's Deep-Path: 103 concepts from classical ML to GPU kernels, mapped onto this course (with a placement check)
Deep-ML's ordered route through 1,200+ problems: 14 parts in five depth zones, from Sunlit (classical ML, deep learning) through Twilight (LLMs), Midnight (RL, GNNs) and the Abyss (inference, CUDA, compilers) to the Trench toolboxes. Covers placement plus spaced repetition, a part-by-part crosswalk into modules 01–13 with all 103 concepts listed, a comparison with TensorTonic and 30u30, and a 25-question self-placement check that recommends where to start.
practiceplacementspaced repetitionstudy planbeginner-friendlyLLMInference Engineering, interactive edition: how to use inferenceengineering.tech alongside this course
A free interactive companion to Philip Kiely's book: 8 chapters, 3 guided paths (3–4 h each), 7 calculators (VRAM, roofline, KV cache, quantisation, speculative decoding, GPU selection), quizzes, cheat sheets and a vLLM-vs-SGLang benchmark. Includes a chapter-by-chapter pairing with modules 05, 08, 09, 12 and 13, and a critical read of the benchmark's scope and dates.
inferenceinteractive bookcalculatorsbenchmarksbeginner-friendlyLLMHands-On LLM Serving and Optimization: the book's code as a lab track
Chi Wang and Peiheng Hu's O'Reilly book and its companion repo, turned into eight labs: transformer and KV-cache notebooks, a from-scratch FastAPI serving system (workload manager, worker process, streaming, vLLM backend), multi-model and Triton serving, quantisation, speculative decoding, LMCache, SGLang/TensorRT-LLM/llama.cpp, and a Qwen3-14B optimisation case study. Includes a chapter map to the course, a comparison with Kiely's book, practical gotchas, and an interactive dynamic-batching simulator.
servinglabsvLLMbatchingbeginner-friendlyLLM + Robotics391 free AI courses, checked: a searchable catalog with link health, provenance, and 2026 updates
A viral "free curriculum that replaces your degree" turned into something usable. All 391 courses in a filterable catalog (category, year, search, working links only), with all 733 links checked (610 work; 381 of 391 courses still reachable). Credits the real source (the Deep Learning Drizzle list), gives a reality check (the newest course is from 2022), adds modern courses for what's changed, and adds the missing robotics section (Modern Robotics, MIT Manipulation & Underactuated, CS285, HF Robotics/LeRobot, MuJoCo, Isaac Lab).
coursescataloglink healthrobotics coursesbeginner-friendlyLLM + RoboticsBuild a Robot Brain: 12 weeks, from zero to a robot that understands you (ages 15+)
Our own first-principles course, built from everything in modules 01–17. One lesson a week for 12 weeks, covering robot basics and inference engineering. A learner with basic Python builds Robo-Helper, a simulated robot that understands "put the red block in the bin", plans with A*, stops safely for surprises, and is honestly evaluated (83%). Then they build a tiny neural language model from zero, measure latency, tokens and cost, speed brains up (caching, quantising, distilling, LoRA), plug in Claude as a planner and as a tool-using agent, and race every brain on the same 12 tasks in a simulator with replays. Every lesson has to make the robot measurably better: progress.py shows each one's gain, and the brain the learner builds climbs from 5/12 (word counter) to 8/12 (a tiny LM that writes whole plans) to 10/12 (distilled from a teacher, still 10/12 at 4 bits in 1.4 KB). Extras: attention (how Transformers read), an AI model as judge, and an MCP server so any AI app can drive the robot. Every lesson has a browser playground, real Python homework marked by check.py (100 checks; reference solutions included), hint ladders, and a real-world challenge away from the screen.
LLMEvals: how to know whether an AI system actually works (a field guide, from one test case to a release gate)
Sergii Makarevych's plain-language guide, built on one support assistant: gold answers and dev/test piles, reading outputs before writing graders, code graders vs AI judges, checking a judge with kappa (60% agreement turned out to be kappa 0.15), the five judge biases and why evidence beats a stronger judge, error bars and paired comparisons, grading RAG and agents (pass^k, cost per success), reward hacking, benchmarks, and running evals every week. Three interactive demos (judge quality, error bars, pass@k vs pass^k), plus the same ideas applied to Robo-Helper's bake-off with real numbers.
evalsLLM-as-judgekappaerror barsagentsbeginner-friendlyLLM + RoboticsSix tiny repos that rebuild modern AI from scratch: micrograd, minGPT, nanoGPT, llm.c, tinygrad, minimalRL (run and tested)
The next rung after Robo-Helper. A hub with a suggested path, and one page per repo: how the code works, a "Run it, tested" section with real output from a MacBook (and fixes for what broke in 2026), plus an experiment that uses the repo to try to make Robo-Helper better, measured honestly in the simulator. Some results: micrograd re-derives our hand-written gradients exactly; a nanoGPT brain reads colours buried in long rambling commands (100% vs 52%) but doesn't beat the distilled brain overall; minGPT is too data-hungry for 343 examples; llm.c's GPT-2 is too slow for the robot's 0.2 s reaction budget on a CPU; tinygrad loses one command at a time but wins over 100× on big batches; a DQN learns to drive the room but fails when the cat moves, where A* just replans.
from scratchautogradGPTC/CUDARLbeginner-friendlyLLM + RoboticsThe maths behind robots, learned by playing: ten interactive explainers, each one used to make Robo-Helper better
Linear algebra, Bayes, Markov chains, Kalman filters, A*, Fourier, momentum, reinforcement learning, Transformers and diffusion, each with a free interactive page to play with, the idea in plain words, the Robo-Helper lesson that uses it, and a tested experiment (bonus_robot_math.py) with real before/after numbers. For example, a Kalman filter cuts distance error while driving from 0.56 m to 0.22 m, where averaging barely helps (0.54 m), and a Markov-chain model of a cat with a favourite bed cuts run-ins from 0.83 to 0.06 per trip. Some results are honest surprises (momentum 0.9 overshoots; exploring more slows Q-learning in a tiny maze). Includes an interactive Kalman demo and a summary of how π0 and GR00T are built (a vision-language model plus a flow-matching action head that outputs chunks of motion).
LLM + RoboticsGradient, divergence and curl: three ways a field of arrows moves, and how a robot steers with them
The three animated examples from the post (∇φ = (−2x, −4y), div = ±2, curl = ±2), explained in plain words and checked numerically. Then the robot: a potential-field planner (pull to the goal, push from obstacles) that drives without a map. Following the gradient alone gets stuck behind a sofa from 9 of 11 starts. Adding a swirl (curl) around obstacles reaches the goal from all 11, at the cost of longer paths. Interactive demo of all the fields and of the robots driving, matched exactly to the Python.
vector calculuspotential fieldsnavigationflow matchingbeginner-friendlySummaries at a glance
| # | Track | Topic | Key takeaway |
|---|---|---|---|
| 01 | LLM | LLM evals with decision models | Set minimum precision and recall first, tune the threshold with CV, verify calibration, then pick on the cost/quality frontier. |
| 02 | LLM + Robotics | TensorTonic ML practice | Drill the maths and learning half of robotics from scratch: transforms, CV, the RL ladder to PPO, ViT/DDPM. Kinematics, control, estimation and planning need other sources. |
| 03 | LLM + Robotics | Human–AI & human–robot help | Blend robot and human control by calibrated confidence, keep intent beliefs revisable, choose ask/act/wait by expected cost, and keep the struggle where the learning is. |
| 04 | LLM + Robotics | Big teacher, small student | Log a big model on your narrow task, fine-tune the smallest model that meets your bar, evaluate against hand-corrected data split by day, and measure latency on the target device. Milliseconds are metres. |
| 05 | LLM | LLM architecture at scale (Kimi K3) | Read architectures along three axes: width (sparse experts, balanced routing), depth (layers that reach back), length (fixed-size memory plus some global attention). Every feature also needs a numerical-stability fix and a serving plan. |
| 06 | LLM + Robotics | Ilya's 30 papers | The toolbox behind robot learning in 30 days: memory (RNN → LSTM → attention), seeing (AlexNet → ResNet), relations and graphs, compression as learning, scale, and learning to act (PPO, RLHF). Start each day with the plain-language box. |
| 07 | LLM + Robotics | LLM post-training | Pre-training sets what a model knows; post-training sets how it uses it. Three teachers: demonstrations (SFT / behaviour cloning), comparisons (RLHF, DPO), checkers (RLVR / success detectors). Always keep a leash and watch the true metric. |
| 08 | LLM | Research-engineer roadmap | Twelve skills that turn papers into fast, well-tested models. Learn the internals, the data, the architectures, alignment, systems and inference, then reproduce and publish honestly. Most stages tie back to modules 01–07. |
| 09 | LLM | Inference engineering, 25 problems | Serving is arithmetic plus trade-offs: size the KV cache, keep the GPU busy with continuous batching, protect TTFT with chunked prefill, reuse prefixes, pick precision and GPUs by bandwidth, measure tails, and price it all from explicit assumptions. |
| 10 | LLM + Robotics | Self-improving agent harnesses | Don't stuff the context: keep it in variables and query it with code and sub-calls (RLM). Let the agent refine its own harness in small, reversible steps without resets (Continual Harness). Read harness benchmarks as "model + harness". |
| 11 | LLM | LLM self-study stack | Build it to understand it: CS336 (pre-training and systems) → Raschka (reasoning, GRPO, distillation) → Lambert (post-training). Plan the hours honestly, and keep your body in the plan. |
| 12 | LLM + Robotics | Training & inference videos | Watch the mechanisms move: memory and parallelism, LoRA, GRPO, on-policy distillation, optimisers, speculative decoding and serving, each guide tied back to the course's deeper pages. |
| 13 | LLM | 100 days of inference engineering | One chapter of Kiely's book per block, one build per chapter, one written note per day, shared publicly. Start with the smallest model that passes your evals. |
| 14 | LLM + Robotics | Deep-ML Deep-Path | Find your frontier, then drill a few steps a day with spaced review. Read the course page to understand a concept, and do the Deep-Path problems to make it stick. Robotics and control is the thinnest part of the map. |
| 15 | LLM | Inference Engineering, interactive | Read the book chapter, run its calculator, then cross-check against the course's own demo and explain any gap. Treat single-setup benchmarks as a starting point. |
| 16 | LLM | Hands-On LLM Serving labs | Build a small server yourself (API, scheduler, worker process), then swap in real engines and optimise one change at a time on realistic data. Batching trades latency for throughput, and continuous batching mostly removes that trade. |
| 17 | LLM + Robotics | 391 free AI courses, checked | A huge free list is only useful once it's filtered and verified. Keep the classics for foundations, add post-2022 courses for LLMs and robotics, and prefer courses with assignments over lectures alone. |
| 18 | LLM + Robotics | Build a Robot Brain (our course) | Learn by building: each idea from first principles, then real code, a self-check and a real-world challenge, growing into one working robot that's honestly tested. Then inference engineering: a tiny LM from zero, latency and cost, caching/quantising/distilling, Claude as planner and agent, and a bake-off of every brain. |
| 19 | LLM | Evals field guide | An eval is input, system, grader, number. Read outputs before writing graders, use code for anything definite, and trust an AI judge only after measuring it (kappa, not raw agreement), with the evidence in front of it. Every score gets an error bar; agents get pass^k. |
| 20 | LLM + Robotics | Six from-scratch repos | Read tiny code that does the real thing, run it, then use it to try to make your own robot better, and measure whether it did. Bigger models need more data; a fast brain on a laptop can be too slow on a robot; learned control is great until the world changes. |
| 21 | LLM + Robotics | Robot maths, played with | Play with each idea first, then use it on your own robot and measure it. Estimation (Bayes, Kalman, Fourier) beats plain averaging once the robot moves; a model of the world only helps if the world has patterns; π0 and GR00T are a scene-understanding Transformer plus a noise-to-motion head. |
| 22 | LLM + Robotics | Gradient, divergence, curl | Gradient = uphill, divergence = spreading out or draining in, curl = swirl. A robot can drive by following a field, but pure gradients get stuck in dips; a little curl around obstacles gets it round. Measure from many starts, not one. |
Concept map
Concepts covered so far, and the modules that teach them:
| Concept | Modules |
|---|---|
| Precision / recall / F-beta, Type I/II errors | 01, 02, 19 |
| Decision thresholds | 01 |
| Cross-validation, honest splits (by day/episode, drift cutoff) | 01, 04 |
| Probability calibration (Platt, Brier), MC-dropout uncertainty | 01, 03, 06 |
| Cost / latency / quality trade-offs | 01, 02, 04, 09 |
| LLM judges & evals | 01, 18, 19 |
| Judge agreement (Cohen's kappa, catch rate), judge biases | 18, 19 |
| Error analysis (reading outputs, failure taxonomies) | 19 |
| Agent evaluation (final state, pass^k, reliability, cost per success) | 10, 18, 19 |
| Linear algebra for robots (rotations, SVD, Jacobians) | 02, 21 |
| Homogeneous transforms & forward kinematics | 02 |
| Probability for estimation (Bayes, covariance, Monte Carlo) | 02, 03, 21 |
| State estimation & filtering (Kalman, Fourier, Markov chains) | 21 |
| Vector calculus (gradient, divergence, curl) & potential-field navigation | 21, 22 |
| Vision-language-action models (π0, GR00T: VLM + flow-matching action head, action chunks) | 05, 08, 21 |
| Computer vision for perception (CNNs, ResNet, dilated convs, detection) | 02, 06 |
| Reinforcement learning (Bellman → Q-learning → DQN → PPO/GAE, SAC, RLHF) | 02, 06, 18, 20 |
| Transformers, attention, ViT, diffusion → VLAs / diffusion policy | 02, 05, 06, 18, 20 |
| Autograd & from-scratch implementations (micrograd, minGPT, nanoGPT, llm.c, tinygrad) | 18, 20 |
| On-device inference & GPU (quantisation, ONNX, QAT, speculative decoding) | 02, 04, 05 |
| Human–robot interaction, levels of automation | 03 |
| Shared autonomy & policy blending | 03 |
| Intent inference (noisily rational, goal switching) | 03 |
| Ask / act / wait, mixed initiative | 03 |
| Learning science (scaffolding, desirable difficulty, compounding) | 02, 03, 05 |
| Knowledge distillation (teacher → student, privileged teacher, multi-teacher) | 04, 05, 07 |
| Data flywheels & logging | 04 |
| Speech recognition & voice commands (CTC) | 03, 04, 06 |
| Bootstrap / paired comparisons, detection floor | 04, 18, 19 |
| Mixture of experts & load balancing | 05, 08 |
| Linear attention, delta rule, fixed-size memory | 05 |
| KV cache, MLA, long context (NoPE, 1M tokens) | 05 |
| Residual connections & attention residuals | 05, 06 |
| Forgetting / decay as a design parameter (LSTM gates) | 03, 05, 06 |
| Recurrent networks, LSTMs, external memory (NTM) | 06 |
| Compression as learning (MDL, Kolmogorov complexity) | 06 |
| Graph neural networks & relational reasoning | 06 |
| Regularisation (dropout) | 06 |
| Scaling laws, data mixing & distributed training (ZeRO, FSDP, 3D parallelism) | 05, 06, 08 |
| Learning rewards from human preferences (RLHF, DPO, KTO, ORPO) | 03, 06, 07, 08 |
| Pre-training vs post-training; SFT / behaviour cloning | 07 |
| KL penalty, trust regions & reward hacking | 06, 07, 19 |
| Verifiable rewards (RLVR) & success detectors | 07, 08 |
| Custom autograd, GPU kernels (Triton), memory layout | 02, 08 |
| State-space models (Mamba/S4) & ring attention | 08 |
| Synthetic data & curation (rejection sampling, verifiers, template-generated data) | 04, 07, 08, 18 |
| Multimodal fusion & vision-language projection (VLA core) | 05, 08 |
| KV-cache design: MLA, GQA/MQA, KV quantisation, PagedAttention | 05, 08, 09 |
| Serving: dynamic & continuous batching, chunked prefill, prefix caching, disaggregation | 05, 09, 12, 16 |
| Latency metrics (TTFT, TPOT, p99), observability & autoscaling | 04, 09 |
| GPU selection, roofline & profiling (Nsight) | 08, 09 |
| Agent harnesses: RLMs, programmatic tool calling, sub-agents | 10 |
| Self-improving / reset-free agents (Continual Harness) | 10 |
| Reasoning models: inference-time scaling, GRPO, reasoning distillation | 07, 10, 11 |
| Self-study paths & courses | 02, 06, 08, 11, 12, 13, 14, 17, 18 |
| Hands-on first-principles curriculum (ages 15+) | 18 |
| Robotics courses (kinematics, manipulation, control, sim) | 02, 14, 17 |
| Placement tests & spaced repetition | 03, 14 |
| Optimisers (SGD → Adam → AdamW → Muon) | 08, 12 |
| Parameter-efficient fine-tuning (LoRA, QLoRA, DoRA) | 08, 12 |
| Learning in public | 08, 13 |
| Mechanistic interpretability (logit lens, activation patching) | 08 |
| Red teaming & contamination-aware evals, benchmark saturation | 01, 08, 19 |
| Paper reproduction & honest public write-ups | 04, 06, 08 |