nanoGPT: a real GPT in two 300-line files

The simplest code that can train a proper GPT, from a baby Shakespeare model on a laptop up to GPT-2 on a GPU node. Now retired in favour of nanochat, which covers the whole path to a chatbot.

Tested at commit 3adf61e on 7 Nov 2026, on a MacBook with an Apple M4 (16 GB), Python 3.14.5, PyTorch 2.15.0. Short code excerpts are quoted with credit to Andrej Karpathy (MIT).

Going deeper: everything below is the full version. On a first read, look at the real output in Run it, tested and the Robo-Helper link, then come back.

1. Summary

• Two files do the work. The model (embeddings, Transformer blocks, output head) and the training loop (batches, AdamW, learning-rate schedule, checkpoints).
• Autograd does the calculus. Unlike our Module 18 lessons, there is no hand-written backward pass. One call, loss.backward(), computes every gradient.
• Configs are just Python files plus --key=value flags that override them. No framework.
• It scales by changing numbers, not code. The same files train a 0.8M-parameter toy and the 124M-parameter GPT-2.
• As a Robo-Helper brain (61K parameters, the same training data as Lesson 15), it scored 9/12 in the bake-off against tinylm's 8/12. It kept the colour in 100% of commands with 14 filler words, against 52%, and decided in under 2 ms on the CPU. Details.
• It's deprecated. The Nov 2025 note says you probably want nanochat, which covers tokenizer, pretraining, finetuning, evaluation and chat.

2. How the code works

model.py: the GPT

The model is four small classes and one big one:

• CausalSelfAttention: one linear layer makes a query, key and value for every token, for every head at once. The tokens are split into n_head groups, each head does attention on its own slice, and the results are glued back together.
• MLP: two linear layers with a GELU between them. The middle is 4× wider than the embedding.
• Block: layer norm → attention → add back; layer norm → MLP → add back.
• GPT: token embeddings (wte) plus position embeddings (wpe), a stack of n_layer Blocks, a final layer norm, and lm_head, which turns each position into a score for every token in the vocabulary.

The whole Transformer block fits in two lines. This is the one idea that makes it click:

# model.py (MIT, Andrej Karpathy): class Block
def forward(self, x):
    x = x + self.attn(self.ln_1(x))
    x = x + self.mlp(self.ln_2(x))
    return x

Each sub-layer only adds a correction to x; it never replaces it. That "add back" is the residual connection. Gradients can flow straight down the x = x + … path, which is why you can stack many blocks and still train them. Layer norm (ln_1, ln_2) rescales the numbers going into each sub-layer so they stay in a sensible range.

Inside attention, the line that makes it "causal" asks PyTorch's fused attention to mask out the future:

# model.py: CausalSelfAttention.forward (trimmed)
q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
y = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=None,
        dropout_p=self.dropout if self.training else 0, is_causal=True)

The file also keeps a slow, written-out version (used only on old PyTorch): q @ k.transpose, times 1/√(head size), fill the future with −∞, softmax, then @ v. That's exactly the recipe from our Lesson 15 stretch, done for every position and every head in one matrix multiply.

Two more details worth spotting: the output head shares its weights with the token embedding (wte.weight = lm_head.weight, "weight tying"), and generate() is a plain loop: crop to block_size, run the model, take the last position's scores, divide by temperature, optionally keep the top k, softmax, sample, append.

train.py: the loop

1. Config. Defaults at the top of the file, then a config file, then command-line flags (via the tiny configurator.py), each overriding the last.
2. Data. get_batch memory-maps train.bin (a long list of token ids as 16-bit integers), picks batch_size random start points, and takes block_size tokens from each. The targets are the same slices shifted by one.
3. Learning rate. get_lr warms up linearly for warmup_iters, then follows a cosine curve down to min_lr at lr_decay_iters.
4. Step. Forward, loss.backward(), clip the gradient norm to 1.0, AdamW step, zero the gradients. Gradient accumulation lets a small GPU fake a big batch.
5. Evaluate. Every eval_interval steps, average the loss over eval_iters batches of train and val data, and save a checkpoint if val improved.
6. Scale up. The same file runs on 8 GPUs with PyTorch DDP via torchrun, and can use torch.compile and bfloat16 on GPUs.

3. Run it, tested

All commands were run on 7 Nov 2026 on a MacBook (Apple M4, 16 GB), Python 3.14.5, in a private venv in a scratch folder. PyTorch reported torch.backends.mps.is_available() == True. train.py uses relative paths, so it is run from inside the cloned folder, as the README does.

Install and prepare the data

git clone --depth 1 https://github.com/karpathy/nanoGPT.git
python3 -m venv venv
./venv/bin/pip install torch numpy tiktoken requests
cd nanoGPT
../venv/bin/python data/shakespeare_char/prepare.py

length of dataset in characters: 1,115,394
vocab size: 65
train has 1,003,854 tokens
val has 111,540 tokens

Everything installed on Python 3.14 with no fixes. (The README's install line also lists transformers datasets wandb tqdm; they're only needed for loading OpenAI's weights, OpenWebText and logging.)

Run 1: the README's MacBook command, with --device=mps

This is the README's CPU command with the device switched to the Mac GPU, as the README suggests for Apple Silicon. It already finishes in well under 10 minutes, so max_iters stays at 2000.

$ ../venv/bin/python train.py config/train_shakespeare_char.py --device=mps --compile=False \
    --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 \
    --n_embd=128 --max_iters=2000 --lr_decay_iters=2000 --dropout=0.0
# 7 Nov 2026 · MacBook (Apple M4) · MPS
number of parameters: 0.80M
step 0: train loss 4.1676, val loss 4.1649
step 500: train loss 2.2732, val loss 2.3141
step 1000: train loss 1.9714, val loss 2.0528
step 1500: train loss 1.8557, val loss 1.9225
step 2000: train loss 1.7648, val loss 1.8857
iter 1999: loss 1.9002, time 12.38ms, mfu 0.11%
total wall time 38.3 s · median 11.9 ms per iteration

The README reports a loss of "only 1.88" for this command on CPU in about 3 minutes. We got val 1.8857 in 38 seconds. The run also prints two harmless warnings about torch.cuda.amp.GradScaler being deprecated and disabled (there's no CUDA).

The same command with --device=cpu took 33.5 s (median 13.6 ms per iteration) and ended at the same losses to 4 decimal places. This model is so small that the Mac GPU doesn't help; the CPU is just as fast.

$ ../venv/bin/python sample.py --out_dir=out-shakespeare-char --device=mps --num_samples=1 --max_new_tokens=300

That not and of this neht a lord shall stell the maid
not for sembly in blow the she were no the broth
sace crear in what hus from hourt canter here!
Thy post a coursel my happhing fond esend
Of the me blod I have at affor's o' for werer.

YELEO:
Went nacte king it you hath Cliffsly all tiser mut
To

It has learned the shape of a play: a speaker name in capitals with a colon, short lines, real small words ("a lord shall", "I have"). Most long words are invented.

Run 2: the full "baby GPT" config on MPS, shortened

The config file's own model (6 layers, 6 heads, 384 channels, 256-character context, batch 64, dropout 0.2) is meant for a GPU: the README says 5000 iterations take about 3 minutes on one A100 and reach val loss 1.4697. A 30-iteration probe on MPS took about 0.85 s per iteration, so 5000 iterations would take over an hour. We cut it to 500.

$ /usr/bin/time -p ../venv/bin/python train.py config/train_shakespeare_char.py --device=mps --compile=False \
    --max_iters=500 --lr_decay_iters=500 --eval_iters=20 --out_dir=out-big
# 7 Nov 2026 · MacBook (Apple M4) · MPS
number of parameters: 10.65M
step 0: train loss 4.2885, val loss 4.2831
step 250: train loss 1.9712, val loss 2.0761
step 500: train loss 1.6049, val loss 1.7723
iter 400: loss 1.7681, time 1269.03ms, mfu 0.29%
real 664.08          (median 1.21 s per iteration)

$ ../venv/bin/python sample.py --out_dir=out-big --device=mps --num_samples=1 --max_new_tokens=300

That not and bect of me
That the shall stell the maid not for seess and blood.

AUTOLYCUS:
I cark that say they
Heir what nuke to the good on me such brone,
He should like happ it then less in him to thy
Which which daught omon for werer a grace of then comple,
And have their somell all of rangue

After only 500 of its 5000 steps, the 10.65M model already beats the small model's final score (val 1.7723 vs 1.8857). It also uses a real Shakespeare name, AUTOLYCUS. The run took 11 minutes, a little over our 10-minute target. Steps slowed from about 0.85 s at the start to a median of 1.21 s, and each evaluation took 15–19 s. For the last minute, the Robo-Helper experiment below was also running on the CPU, which may have slowed it a little.

What failed

Nothing failed outright. --compile=False is still needed on the Mac (as the README says), and torch.compile was not tried.

4. The successor: nanochat

5. Connect to Robo-Helper (Module 18): a nanoGPT brain, measured

Measured on 8 Nov 2026, with the first version of Robo-Helper's training data: 350 examples, which by accident included 3 of the 12 bake-off commands. On 9 Nov those were removed (343 examples). The small Module 18 brains' bake-off scores didn't change, and the tiny LM's held-out score went from 86% to 92%. Every brain here was compared on the same data, but if you re-run, expect the exact numbers to move by a few points.

Our Lesson 15, a tiny language model, is the hand-built version of what nanoGPT does. It writes a plan one word at a time, like put red blue <end>. Its Stretch 2: attention is the exact bridge.

The experiment: swap Robo-Helper's brain for a nanoGPT

We used nanoGPT's model.py unchanged, at a small size: 2 layers, 4 heads, 48 channels, a 24-word window, 61,488 parameters. Each word is one token. A training example looks like command words <sep> plan words <end>, and the loss only counts the plan words. It trains on exactly the same data as the Lesson 15 model, lesson15_tiny_lm.training_data() (350 examples), and runs inside the same TinyLMBrain wrapper. That means the same typo fixer and the same Lesson 10 safety check on every action. We added one trick: in training, each example is shifted right by a random number of padding tokens, so every position learns to hold real words. Training 1,500 steps of batch 32 on the CPU took about 14 s per model. We trained 3 seeds.

$ ./nanogpt/venv/bin/python nanogpt_brain.py        # 8 Nov 2026 · MacBook (Apple M4)
BAKE-OFF (12 tasks)            words  tinylm  attention  distilled  nanogpt(s0,s1,s2)  nanogpt+teacher(s0,s1,s2)
 1. put the red block in the bin     Y     Y      Y          Y         Y Y Y              Y Y Y
 2. where's the green one?           .     .      .          Y         . . .              Y Y Y
 3. make me a sandwich               Y     Y      Y          Y         Y Y Y              Y Y Y
 4. put the red and blue blocks away .     Y      Y          Y         Y Y Y              Y Y Y
 5. tidy up the room                 .     .      .          .         . . .              . . .
 6. bin everything except the green  .     Y      Y          Y         Y Y Y              Y Y Y
 7. put away the block closest to... .     .      .          .         . . .              . . .
 8. put the block away               .     Y      Y          Y         Y Y Y              Y Y Y
 9. go to the blue block             Y     Y      Y          Y         Y Y Y              Y Y Y
10. stop                             Y     Y      Y          Y         Y Y Y              Y Y Y
11. hmm what about the blue          .     .      .          Y         Y Y Y              Y Y Y
12. put the purple block in the bin  Y     Y      Y          Y         Y Y Y              Y Y Y
PASSED                              5/12  8/12   8/12      10/12      9/12 (all 3 seeds)  10/12 (all 3 seeds)

HELD-OUT test_data() exact-plan accuracy (50 examples)
   tinylm 86.0%   attention 94.0%   distilled 94.0%   nanogpt 94.0% / 94.0% / 88.0%

COLOUR BURIED IN FILLER (padded_commands, 9 commands, mean of 3 seeds)
   filler   tinylm  attention  nanogpt
        0     100%       100%     100%
        6      89%        85%     100%
       10      78%        85%     100%
       14      52%        52%     100%

TIME PER DECISION: model.generate("put the red block in the bin"), 200 runs
   tinylm (pure Python)   p50   0.172 ms  p95   0.193 ms   travels 0.19 mm at 1 m/s
   nanogpt CPU            p50   0.910 ms  p95   1.626 ms   travels 1.63 mm
   nanogpt MPS            p50   5.281 ms  p95   6.451 ms   travels 6.45 mm

(The table is lined up by hand from the script's output. "nanogpt+teacher" adds the Lesson 17 teacher's labels, distill_examples(UNLABELLED, best_teacher()), to the training data, just as the "distilled" brain does.)

What got better, and what didn't

• Bake-off: 9/12 against 8/12 for tinylm and attention trained on the same data. With the teacher's labels it reaches 10/12, the same as the distilled brain, not better.
• The extra pass is task 11, and it's luck. On "hmm what about the blue" the model wrote go blue green, which isn't a valid plan, so the robot stayed put and the task (bin nothing) passed. That's a safe failure, not understanding.
• Buried colours: the clearest win. With 14 filler words, nanoGPT still found the colour 100% of the time, against 52% for both Lesson 15 models. Attention over every position, plus position embeddings, keeps the colour from being averaged away. The random padding in training helps too: without it, positions 15–18 would never have seen a real word.
• Held-out test: 94% for two seeds and 88% for the third. That's level with the attention model, not ahead of it.
• Still failing: "tidy up the room" (it asks "which block?") and "the block that's closest to the bin" (it says it can't). Neither is in the training templates, and no model this size can work out "closest" without seeing the room.
• Speed: nanoGPT is about 5–8× slower than our pure-Python tinylm, because PyTorch has a fixed cost per call. On the Mac GPU (MPS) it is slower again, about 5 ms, because launching GPU work costs more than the sums themselves. Even so, the slowest case, 6.45 ms (8.6 ms for a 3-word plan), is far inside the 0.2 s "stop!" budget from Lesson 16. The robot rolls less than 1 cm while it thinks.

So nanoGPT is the next rung up. It isn't magic on 350 examples. What it adds is robustness to long, messy commands and a way to scale: the same file trains a 61K model in seconds or a 124M GPT-2 on a GPU node, just by changing numbers. Our hand-built model would need new hand-written backprop for every change.

Next-step exercise. Open nanogpt_brain.py and make the model bigger (n_layer=4, n_embd=96). Then remove the random padding by setting k = 0 in _example. Re-run it and record three numbers each time: bake-off score, filler accuracy at 14 words, and CPU p95. Which change matters more for the buried colours: the size or the padding?

6. Why it matters for robotics

7. Reality check

• Deprecated. The README itself sends you to nanochat. Expect no fixes. We saw deprecation warnings (torch.cuda.amp.GradScaler) on PyTorch 2.15.0, though everything still ran.
• The GPU numbers are for GPUs. "3 minutes, val loss 1.4697" is on an A100. The same config on the M4's MPS ran at a median of 1.21 s per iteration, so the full 5000 iterations would take well over an hour.
• MPS isn't always faster. For the tiny 0.8M model, CPU and MPS took about the same time per step. The GPU only pays off once the model and batch are big enough to keep it busy.
• Character-level is a toy. Real models use tokens (GPT-2's BPE has 50,257). The shakespeare (not _char) data folder and the finetuning config show that path.
• The GPT-2 reproduction needs OpenWebText and about 4 days on 8×A100 40GB. nanochat now gets past GPT-2's CORE score in under 2 hours on 8×H100, thanks to better data, optimisers and precision.
• Overfitting. The small Shakespeare set is about 1 MB. The config's comment says "we expect to overfit", which is why it only saves when val loss improves.

8. Self-check

9. Further reading

• karpathy/nanoGPT on GitHub, and its predecessor minGPT (page 2)
• karpathy/nanochat, its original post (13 Oct 2025) and "Beating GPT-2 for <<$100: the nanochat journey" (1 Feb 2026)
• Neural Networks: Zero to Hero, especially the GPT video that builds this model step by step
• OpenAI GPT-2, the model nanoGPT reproduces, and OpenWebText, the open copy of its training data
• modded-nanogpt, the speedrun fork that inspired nanochat's leaderboard
• Back in this course: Lesson 15, Stretch 2: attention