Ilya's 30 papers: the shared foundations of LLMs and robotics
The 30 foundational papers Ilya Sutskever reportedly recommended to John Carmack, one per day. Each has its own page with a summary, core maths, deep dive, robotics angle, self-check and a link to code you can run.
How to use this module
- One paper a day. Read the course page first (20 min), then the original paper, then run the repo's
train_minimal.pyand do its exercises. - The repo is licensed CC BY-NC-ND, so these pages don't copy it. They're written from the original papers and link to the repo for code and exercises.
- Tick papers off below. Progress is saved in this browser only.
New to all this? Start here
These 30 papers are some of the most important ideas behind today's AI, and behind the robots that learn.
A "paper" is a write-up where scientists explain a new idea and test whether it works. Many of these papers are hard, so every day in this module begins with a Start here box written for someone around 13. You don't need any maths for it.
How to do one day
- Read the Start here box (5–10 minutes).
- Try the "Try it" activity, and play with the interactive demo on the page.
- Check your understanding with the Self-check questions near the bottom. Click a question to see the answer.
- Only if you're curious: read the "Going deeper" parts and look at the code.
Stuck on a word? Every Start-here box has a Words to know list.
Two tracks, one toolbox: these papers are the shared foundations of both today's LLMs and robot learning. Each day has a badge: LLM (mainly language models), Robotics (mainly perception and control) or LLM + Robotics. Every page also has a "Why it matters for robotics" box. Some links are direct (PPO, RLHF, ResNet), and some are honestly conceptual (complexodynamics, Kolmogorov complexity).
Progress
Three ways through
| Path | Order | Why |
|---|---|---|
| The full 30 days | 01 → 30 in order | The repo's intended path: foundations → vision → transformers → specialised architectures → generative and scaling → RL. |
| LLM fast track (12 days) | 01 → 02 → 15 → 13 → 14 → 24 → 25 → 12 → 05 → 26 → 29 → 30 | Language modelling, attention, the Transformer, training giant models, scaling laws, compression as prediction, then PPO and RLHF, the core of post-training (Module 07). |
| Robotics fast track (12 days) | 01 → 02 → 08 → 09 → 28 → 12 → 13 → 14 → 21 → 25 → 29 → 30 | Sequence models, perception backbones, uncertainty, the Transformer, graphs, scaling, then RL and preference learning: the pieces of a modern robot-learning stack. Come back for the rest later. |
Unit 1 · The foundations (Days 1–7)
Next-character prediction learns structure for free.
Gates and a cell state that let memory survive long sequences.
Dropout on non-recurrent connections only.
Noisy weights carry fewer bits, which gives better generalisation.
The best model is the shortest description of the data.
Why "interesting" complexity rises and then falls while entropy only rises.
Measuring that rise and fall in a simulated mixing system.
Unit 2 · Deep learning explosion (Days 8–12)
GPUs + ReLU + dropout + data: the 2012 breakthrough.
Learn a residual on top of the identity, and 100+ layers train.
Pre-activation keeps a clean signal highway through 1,000 layers.
Exponentially growing context without losing resolution.
Randomly silence units: an ensemble for the price of one, and an uncertainty estimate.
Unit 3 · The Transformer era (Days 13–16)
Self-attention replaces recurrence.
The paper, line by line in code.
The original attention: let the decoder look back.
Handling inputs and outputs that are sets, not sequences.
Unit 4 · Specialised architectures (Days 17–22)
A network with differentiable read/write memory.
Outputs that point at inputs: convex hull, TSP.
Reason over every pair of objects.
Memory slots that attend to each other at every step.
One framework for graph neural networks.
End-to-end speech recognition with CTC.
Unit 5 · Generative models & scaling (Days 23–28)
Make the latent code carry the global structure.
Pipeline parallelism with micro-batches.
Loss falls as a power law in parameters, data and compute.
The length of the shortest program as a measure of information.
A formal definition of intelligence across environments.
The working toolkit of convolutional vision.
Unit 6 · Deep RL & alignment (Days 29–30)
Clipped policy updates: the workhorse of sim-trained locomotion.
Learn the reward from human comparisons.
How the 30 connect
| Thread | Papers | Where it leads in robotics |
|---|---|---|
| Memory over time | 01 → 02 → 03 → 17 → 20 → 13 | Recurrent policies → transformer policies with long context (Module 05) |
| Seeing | 08 → 09 → 10 → 11 → 28 | Perception backbones, segmentation for grasping, ViT encoders in VLAs |
| Attention | 15 → 13 → 14 → 16 → 18 | Transformer policies, picking which object to act on |
| Relations & graphs | 19 → 20 → 21 | Scene graphs, multi-robot coordination, learned physics |
| Compression = learning | 04 → 05 → 26 → 27 → 06 → 07 | Why prediction-based world models work. Simplicity priors. |
| Uncertainty | 04 → 12 → 23 | Knowing when the robot doesn't know (Module 01) |
| Scale | 24 → 25 | Training big robot models (Module 05, Module 04) |
| Learning to act | 29 → 30 | Sim-to-real locomotion, learning rewards from people (Module 03, Module 02's RL ladder) |
Before you start
- Maths: matrix multiplication, gradients and the chain rule, softmax, log-probabilities. Module 02 Stage A covers them.
- Setup: clone the repo, then for each day run
pip install -r requirements.txtin that paper's folder. Most code is NumPy or PyTorch and CPU-friendly. - Habit: attempt each exercise before opening the solution (Module 03, don't automate your own learning).