Six tiny repos that rebuild modern AI from scratch, run and tested
micrograd, minGPT, nanoGPT, llm.c, tinygrad and minimalRL: small, readable code that does the real thing. They're the next rung after Robo-Helper, where the learner wrote everything by hand.
Start here (no maths needed)
Big AI programs are huge. These six are tiny versions you can read in an afternoon, and they still work for real.
Think of it like this
A real car has thousands of parts. A go-kart has an engine, wheels, brakes and a steering wheel, and nothing else. You can see every part, and it still drives. Each of these repos is a go-kart version of something huge: the maths that lets AI learn, a ChatGPT-style text model, a game-playing robot brain.
How to use them
- Pick one page. Each starts with a box like this one.
- Read "how the code works", which explains the one big idea.
- Run it on a laptop. Every page shows the exact commands and what happened when we ran them.
- Do the Robo-Helper exercise at the end, which connects it to what you built yourself.
Why robots care
Robots learn the same way: gradients (micrograd), sequence models (GPTs), fast code on small chips (llm.c, tinygrad) and learning by trial and error (minimalRL).
Try it
Before opening any page, guess: how many lines of code do you think it takes to teach a computer to balance a pole on a cart? (Answer on the minimalRL page.)
Words to know
- Repo
- A folder of code shared online, usually on GitHub.
- From scratch
- Built without big helper libraries, so you see every step.
- Gradient
- Which way to nudge each number to make the answer less wrong.
Going deeper: where each repo sits on the ladder, how it connects to the Robo-Helper lessons, and what is out of date.
How to use this module
- After Module 18. Robo-Helper builds everything by hand in the standard library. These repos are the next step up: the same ideas with autograd, tensors, GPUs and real datasets.
- Run, don't just read. Every page has a "Run it, tested" section with real output from a MacBook (Apple Silicon), plus fixes for anything that broke on a 2026 setup.
- Two are officially retired. minGPT is semi-archived, and nanoGPT's README now points to nanochat. They're still the clearest code to learn from; the pages say what to use instead.
Progress
Done: 0 / 6 · Next:
The ladder
| Rung | Repo | What you see built | Needs | After Robo-Helper lesson |
|---|---|---|---|---|
| 1 | micrograd | Autograd: backprop done automatically, about 100 + 50 lines | Python only | Week 3 neuron (gradients by hand) |
| 2 | minGPT | A GPT in about 300 lines, plus GPT-2's tokenizer | PyTorch | Week 6 tiny LM + attention |
| 3 | nanoGPT → nanochat | The same GPT with teeth: trains Shakespeare, reproduces GPT-2 | PyTorch, GPU helps | Week 6 attention stretch |
| 4 | llm.c | GPT-2 forward and backward by hand in C/CUDA, no autograd | C compiler (GPU for speed) | Week 8 speed, week 9 faster |
| 5 | tinygrad | A whole small framework: tensors, laziness, kernel fusion, many chips | Python | Week 9 faster |
| 6 | minimalRL | 11 RL algorithms (REINFORCE → PPO, SAC), one file each | PyTorch, gymnasium | Q-learning bonus, week 12 RLHF |
The pages
A scalar Value that remembers how it was made, so backward() can apply the chain rule for you.
The GPT recipe in three files: model, tokenizer, trainer. Small toy projects (an adder, a character model).
minGPT rewritten to be fast: Shakespeare in minutes, GPT-2 (124M) in days. Its successor nanochat trains a GPT-2-level chatbot for about $48.
No PyTorch, no Python: every layer and its gradient written out in C. What frameworks hide.
Between micrograd and PyTorch: lazy tensors, operations fused into a few fast kernels, many hardware backends.
Learning by trial and error on CartPole, each algorithm in one file of about 70–190 lines (REINFORCE is 66).
What each one did for Robo-Helper (measured)
Each page tries to use its repo to make the Module 18 robot better, and reports what really happened. Baselines on the 12 bake-off tasks: word counter 5/12, tiny LM 8/12, distilled 10/12.
| Repo | Experiment | Result |
|---|---|---|
| micrograd | Re-derive the robot's hand-written gradients; an MLP stop detector | Gradients match exactly. On Lesson 4's data the MLP is no better (the true rule is a straight line). On a curved braking case it misses 7 stops instead of 17. |
| minGPT | A character-level GPT brain | 7–8/12. Typos barely hurt it, but our few hundred examples are too few for a Transformer. |
| nanoGPT | A word-level GPT brain | 9/12 (one pass was luck); 10/12 with teacher labels, the same as distilled. Clear win on colours buried in 14 filler words: 100% vs 52%. |
| llm.c | Could GPT-2 (124M) run on the robot's own computer? | Not in time: about 245–323 ms per 3-word decision on a laptop CPU, over the 0.2 s budget. The tiny brains take under 1 ms. |
| tinygrad | The tiny LM on tensors and the Mac GPU | Same plans on 50/50 commands. Slower for one command (about 2 ms vs 0.5 ms), 76–333× faster on 4,096 at once. |
| minimalRL | DQN drives the room vs A* | DQN finds the shortest path and copes with wheel slip, but fails when the cat moves (53 bumps), where A* replans in 7 steps. |
What they share
- One idea per file, nothing hidden. If you can't find where something happens, it doesn't happen.
- They reproduce something real. micrograd's gradients are tested against PyTorch, nanoGPT reproduces GPT-2's loss, and llm.c checks its outputs against a PyTorch reference. A from-scratch version is only worth trusting once it matches a reference. That's the same rule as the numerical gradient check in Robo-Helper's attention stretch.
- Each one adds exactly one layer of help on top of what Robo-Helper did by hand: automatic gradients, then tensors, then GPUs, then compilers.
Suggested path (one a week, after Module 18)
- micrograd. Rebuild Robo-Helper's neuron with a Value class and compare the gradients.
- minGPT. Train the adder, then compare its tokenizer with Robo-Helper's word lists.
- nanoGPT. Train the character-level Shakespeare model on a laptop; read nanochat's README for the full modern pipeline.
- minimalRL. Run DQN and compare it with tabular Q-learning from the bonus lesson.
- tinygrad. Watch operations fuse into kernels (
DEBUG=3), and tie it to week 9's speed tricks. - llm.c. Read one layer (layernorm) forward and backward in C; build the CPU version.