LLM + RoboticsModule 20 · From-scratch code · 1 of 6

micrograd: backpropagation in about 150 lines

A tiny engine that works out gradients for you. It does automatically what you did by hand in Lessons 4 and 15, and it is the same idea that sits inside PyTorch.

Repokarpathy/micrograd

LicenceMIT

Stars18,738 (7 Nov 2026, GitHub API)

Core codeengine.py 94 lines, nn.py 60 lines (counted with wc -l)

StatusSmall and finished. Not marked deprecated. Last commit 3 Aug 2026. The README now points to microgpt for a faster engine.

NeedsPlain Python for the engine. The demo uses numpy, scikit-learn and matplotlib. The tests need PyTorch.

Tested at commit 7bc720e on 7–8 Nov 2026, on a MacBook with an Apple M4 (16 GB), Python 3.14.5. Short code excerpts are quoted with credit to Andrej Karpathy (MIT).

Track: LLM + Robotics. Backpropagation trains every neural network, whether it writes text or steers a robot. On this page micrograd trains Robo-Helper's stop detector (robotics), and the same engine idea trains GPTs (LLMs).

On this page

Start here (no maths needed)
Summary
How the code works
Interactive: watch the gradients flow
Run it, tested
Connect to Robo-Helper (Module 18)
Why it matters for robotics
Reality check
Self-check
Further reading

Start here (no maths needed)

micrograd is a little program that remembers every sum it does, so that afterwards it can tell you which number to nudge, and which way, to make the answer better.

Think of it like this

Imagine a row of dominoes that split and join. You push the first ones and the last one falls. Now you want to know: "if I had pushed domino 3 a bit harder, how much faster would the last one fall?" micrograd keeps a map of which domino knocked over which. Then it walks the map backwards, from the last domino to the first, and works out how much each one mattered. That backwards walk is called backpropagation.

How it works

Every number is wrapped in a little box called a Value.
When you add or multiply two boxes, the new box remembers which boxes made it, and how.
At the end you have one number, the "loss", that says how wrong the answer is.
Call backward(). micrograd walks the map backwards and writes into every box how much it changed the loss. That's its gradient.
Nudge every weight a little bit against its gradient. The loss goes down. Repeat.

Why robots care

In Lesson 4 you worked out by hand how to change Robo-Helper's stop-detector weights. That works for one neuron. A brain with 100 or a million weights needs a machine to do it. micrograd is that machine, small enough to read in an afternoon.

Try it

In the demo below, drag the distance slider and watch the gradient of each weight change sign.
Write down a = 2, b = 3, c = a × b. If you add 1 to a, how much does c change? (It changes by 3, which is b. That's the gradient of c with respect to a.)

Words to know

Gradient: How much the loss changes if you nudge one number up a tiny bit.
Backpropagation: Walking backwards through the sums to get every gradient in one go.
Autograd: Software that does backpropagation for you, automatically.
MLP: "Multi-layer perceptron": layers of neurons, each layer feeding the next.
Loss: One number that says how wrong the model is. Smaller is better.

Going deeper: everything below is the full version, with some code and maths. On a first read, skim the headings, look at the demo and the Robo-Helper results, and come back for the rest later.

1. Summary

TL;DR

micrograd is a scalar autograd engine (engine.py, 94 lines) plus a tiny neural-network library on top (nn.py, 60 lines). Each Value stores a number, its gradient, its parents, and a small function that knows the local derivative. backward() sorts the graph and applies the chain rule from the output back to the inputs. We ran the README example (24.7041, 138.8338, 645.5773, as promised) and trained the README's 2-layer MLP on two moons: 100% training accuracy in 100 steps, 74.5 s on a MacBook CPU.

Everything is a scalar. A neuron is chopped into single adds and multiplies. That is slow, but every step is visible.
The API copies PyTorch. loss.backward(), p.grad, zero_grad() mean the same thing in both.
It checks out against PyTorch. The two unit tests compare micrograd's gradients with PyTorch's. Both passed on our Mac.
Robo-Helper result: on Lesson 4's own data a micrograd MLP is no better than the single neuron, because the true rule is a straight line. On braking data with a curved rule it misses 7 stops instead of 17, out of 500 test readings.
The README now points onward to microgpt, a whole GPT in one dependency-free Python file built on a faster version of this engine.

2. How the code works

The `Value` box

A Value holds data (the number), grad (starts at 0), _prev (the Values that made it) and _backward (a small function). Here is multiplication, quoted from engine.py:

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

From karpathy/micrograd, micrograd/engine.py (MIT).

Read the two grad lines as: "if out moves by 1, self was pushing it with strength other.data". That is the derivative of a × b with respect to a, which is b. It's multiplied by out.grad, how much out matters to the final loss. That multiplication is the chain rule.

Notice the +=. If one Value is used twice (like b in the README example), the gradient from each use is added up. Forgetting this is the most common bug in hand-written backprop.

The one idea that makes it click: walk the graph backwards, in order

backward() first builds a topological order: a list where every Value comes after the Values that made it. Then it sets the output's gradient to 1 and calls each node's _backward in reverse order:

self.grad = 1
for v in reversed(topo):
    v._backward()

From karpathy/micrograd, micrograd/engine.py (MIT).

Reverse order guarantees that by the time a node passes its gradient back to its parents, it has already collected the gradient from everything that used it. That's the whole algorithm. PyTorch does the same thing, but on big arrays (tensors) instead of single numbers, and with fast C++ and GPU code.

The neural-network layer

nn.py builds on that: a Neuron is a list of weight Values and a bias, giving relu(w·x + b). A Layer is a list of Neurons, and an MLP is a list of Layers. parameters() collects every weight so the training loop can do p.data -= lr * p.grad. The demo's MLP(2, [16, 16, 1]) has 337 parameters.

3. Interactive: watch the gradients flow

Robo-Helper's stop neuron, as a micrograd graph

The Lesson 4 neuron: p = sigmoid(w1·distance + w2·speed + b), loss = cross-entropy. Each row is one Value, listed in topological order. Press Backward to run the reversed loop one node at a time. Gradients are computed by a 30-line JavaScript copy of micrograd's Value.

distance x1 (m): closing speed x2 (m/s):

label: should stop (1)

p(stop)

–

dLoss/dw1

–

(p − y)·x1, by hand

–

dLoss/dw2

–

Weights are fixed at w1 = −0.9, w2 = 1.3, b = 2.1. At distance 1.5 and speed 1.0 the page gives p = 0.885948 and dLoss/dw1 = −0.171079, the same digits as our Python check in section 6.

4. Run it, tested

Everything ran in a fresh venv in a scratch folder, with the repo cloned into its own empty folder. Our scripts live in a separate scripts/ folder and run by path.

# 8 Nov 2026 · MacBook (Apple M4), CPU · Python 3.14.5
git clone --depth 1 https://github.com/karpathy/micrograd micrograd-src/micrograd
python3 -m venv venv
venv/bin/pip install numpy scikit-learn torch pytest     # numpy 2.5.4, scikit-learn 1.9.1, torch 2.15.0
venv/bin/pip install -e micrograd-src/micrograd

# 1) The README example, copied into scripts/readme_example.py
$ venv/bin/python scripts/readme_example.py
24.7041
138.8338
645.5773

# 2) The unit tests (they compare against PyTorch)
$ venv/bin/python -m pytest -q micrograd-src/micrograd/test
..                                                    [100%]
2 passed in 6.22s

All three README numbers match exactly. Python 3.14 caused no problems for micrograd.

Two moons, as a plain script

demo.ipynb is a notebook. We copied its steps into scripts/moons.py: the same seeds (1337), make_moons(n_samples=100, noise=0.1), MLP(2, [16, 16, 1]), the same hinge ("max-margin") loss with L2 1e-4, and 100 steps of SGD with the learning rate falling from 1.0 to 0.1. We added a timer and a check on 200 fresh moons points.

# 7 Nov 2026 · MacBook (Apple M4), CPU
$ venv/bin/python scripts/moons.py
params 337
step 0 loss 0.8958, accuracy 50%
step 10 loss 0.2451, accuracy 91%
step 20 loss 0.1898, accuracy 91%
step 30 loss 0.1173, accuracy 95%
step 40 loss 0.0602, accuracy 100%
step 50 loss 0.0988, accuracy 96%
step 60 loss 0.0326, accuracy 99%
step 70 loss 0.0142, accuracy 100%
step 80 loss 0.0123, accuracy 100%
step 90 loss 0.0110, accuracy 100%
step 99 loss 0.0110, accuracy 100%
time 74.5s
held-out accuracy 100.0%

The losses match the notebook's saved output line for line (it also ends at step 99, loss 0.01098, accuracy 100%). The cost is the point to notice: 74.5 seconds for 100 steps of a 337-weight network on 100 points. Every multiply is a Python object with a closure. PyTorch would do this in well under a second.

5. Connect to Robo-Helper (Module 18)

Module 18 uses the standard library only, and every gradient is written by hand. micrograd is the next rung up. It's still pure Python, but the gradients are worked out for you.

a) Check our hand-derived gradients with a 27-line engine

In Lesson 4 the update rule is w −= lr × (p − label) × x. In Lesson 15's attention stretch, lesson15_attention.py back-propagates through softmax with d_s = a × (d_a − Σ a·d_a). We wrote a micrograd-style V class (27 lines with + × ** exp log) and let it compute both gradients automatically:

# 7 Nov 2026 · MacBook (Apple M4), CPU
$ venv/bin/python scripts/tiny_value.py
lesson04 neuron, p = 0.885948
  autodiff dw: [-0.171079, -0.114052] db: -0.114052
  by hand  dw: [-0.171079, -0.114052] db: -0.114052
lesson15 softmax backward
  autodiff d_s: [0.022903, -0.059523, 0.054539, -0.017918]
  by hand  d_s: [0.022903, -0.059523, 0.054539, -0.017918]

They agree to six decimal places. Lesson 15 checked its attention backprop against a numerical gradient (nudge and measure). This is a second, independent check: an autograd engine never sees our formula, yet reaches the same numbers.

b) A better stop detector? Tested honestly

The robot's stop decision comes from Lesson 4's single neuron, judged with Lesson 5's precision and recall. We trained a micrograd MLP(2, [8, 8, 1]) (105 weights, hinge loss as in the demo, 300 SGD steps of 64 examples) and compared it with the neuron, using Lesson 5's own split and rules. "Safe cut" is Lesson 5's rule: the highest threshold whose training recall is at least 0.97.

# 8 Nov 2026 · MacBook (Apple M4), CPU
$ venv/bin/python scripts/micrograd_stop.py <Module 18 solutions folder>

== Lesson 4/5 data (straight-line rule: stop if d - v < 1): 450 train, 150 test ==
single neuron (Lesson 4)
   default cut  : accuracy 0.973  precision 0.946  recall 0.946  missed stops 2
   safe cut +0.22: accuracy 0.947  precision 0.837  recall 0.973  missed stops 1
micrograd MLP 2-8-8-1 (105 params)
   default cut  : accuracy 0.980  precision 1.000  recall 0.919  missed stops 3
   safe cut -1.10: accuracy 0.753  precision 0.500  recall 0.973  missed stops 1

== Braking data (curved rule: stop if d < 0.3 + v^2/1.6): 1500 train, 500 test ==
single neuron (Lesson 4)
   default cut  : accuracy 0.946  precision 0.937  recall 0.898  missed stops 17
   safe cut +0.11: accuracy 0.912  precision 0.811  recall 0.958  missed stops 7
micrograd MLP 2-8-8-1 (105 params)
   default cut  : accuracy 0.966  precision 0.941  recall 0.958  missed stops 7
   safe cut -0.02: accuracy 0.966  precision 0.941  recall 0.958  missed stops 7

Data	What happened
Lesson 4/5 data	No real gain. The true rule (`d − v < 1`) is a straight line, which is exactly what one neuron draws. The MLP's extra accuracy is one reading out of 150, and it misses one more stop. At Lesson 5's safe setting it is much worse: precision 0.500 against 0.837. Two more training seeds gave the same picture (safe-cut precision 0.837 and 0.818).
Braking data	A real gain. Stopping distance grows with speed squared, so the true boundary is a curve. A straight line can't follow it. At the default cut the MLP misses 7 stops instead of 17. At the same safe recall (0.958) its precision is 0.941 against 0.811, so it gives far fewer false alarms. Seeds 1 and 2 reached 0.972 and 0.974 accuracy, against the neuron's 0.946.

For this test we made the braking data ourselves. Speeds go up to 3.5 m/s, and the sensors are less noisy than Lesson 4's (0.1 m and 0.05 m/s). With Lesson 4's noisier sensors, the noise blurred the curve and the two models tied. Cost: in micrograd one MLP decision took about 0.2–0.5 ms, against 0.0005 ms for the neuron. In the simulator's bake-off the stop detector is trained on Lesson 4's straight-line data, so swapping in the MLP would not be expected to help there.

Your next step

Copy the 27-line V class from the code above (the __init__, __add__, __mul__, __pow__, exp, log and backward methods, written in your own words). Use it to recompute the gradient of W2 in lesson15_tiny_lm.py's train_step for one example, and check it matches d_scores[k] × hidden[j].
Then make your own curved stop rule (try d < 0.3 + v²/1.6). Train Lesson 4's neuron and a 2-8-8-1 micrograd MLP on it. Compare missed stops on a test set you never trained on.

6. Why it matters for robotics

Every learned robot controller is trained by backprop. Policies, perception networks and world models all use this same algorithm, run by PyTorch or JAX on tensors.
Physics is often curved. Braking distance grows with speed squared. Torque needed changes with arm angle. Our braking test shows that one neuron can't draw these boundaries and a small MLP can.
Gradients can run through a simulator. "Differentiable physics" research builds simulators out of operations like micrograd's, so a gradient can flow from "the robot missed the target" back to the control inputs. micrograd shows the mechanism, not a tool you'd use for that.
Gradient checking is a safety habit. Before trusting a hand-written gradient in robot code, compare it against autograd or a numerical estimate, as we did above.

7. Reality check

Far too slow for real use. 74.5 s for 100 steps of a 337-weight net on 100 points. Real work uses tensors: PyTorch, JAX or tinygrad (page 5).
Only a few operations. +, ×, ** with a number, and relu. No exp or log, so no sigmoid or cross-entropy. That's why the demo uses a hinge loss. Our 27-line class adds exp and log.
Deep recursion. build_topo is recursive. A very long chain of operations can hit Python's recursion limit.
Not deprecated, but not the newest. The README sends readers who want more to microgpt, which keeps local gradients at forward time instead of one closure per operation.
Pitfalls when copying it: forgetting zero_grad() between steps (gradients pile up because of +=), and using = instead of += in a _backward.

8. Self-check

Why does _backward use += and not =?

A Value can feed several later operations. Its total gradient is the sum of the gradients from every use, so each use adds its share.

Why must backward() visit nodes in reverse topological order?

A node can only pass its gradient back once it has received gradient from every node that used it. Reverse topological order guarantees all of those come first.

In c = a × b, what is a.grad after c.backward()?

b.data. The output's gradient is 1, and the local derivative of a × b with respect to a is b.

Why didn't the MLP beat the single neuron on Lesson 4's data?

The true stop rule there is a straight line, which one neuron can already draw. Extra layers only help when the boundary is curved, as in the braking test.

What is the difference between micrograd and PyTorch's autograd?

The same algorithm. micrograd works on single numbers in pure Python. PyTorch works on whole tensors with fast compiled kernels on CPU or GPU.

9. Further reading

karpathy/micrograd: the repo, with demo.ipynb and trace_graph.ipynb
microgpt, a whole GPT in pure Python, and its explainer post (both linked from the README)
PyTorch, which the tests use as the reference
Back in this course: Lesson 4, the neuron, Lesson 5, precision and recall, Lesson 15, attention
Next: minGPT, which uses PyTorch's autograd to train a whole Transformer