LLMModule 20 · From-scratch code · 2 of 6

minGPT: a whole GPT in about 300 lines of PyTorch

Karpathy's clean, teaching-first GPT. It's the same kind of model as GPT-2, and small enough to learn two-digit addition on a laptop in four minutes. It's semi-archived now, and nanoGPT is its successor.

Repokarpathy/minGPT
LicenceMIT
Stars25,006 (7 Nov 2026, GitHub API)
Core codemodel.py 310 lines, trainer.py 109, bpe.py 319, utils.py 103 (counted with wc -l)
StatusSemi-archived. The README's Jan 2023 note says minGPT "is in a semi-archived state" and points to nanoGPT. Last commit 8 Jan 2023.
NeedsPython and PyTorch. bpe.py also needs regex and requests. A MacBook CPU is enough for the demos.

Tested at commit 37baab7 on 7–8 Nov 2026, on a MacBook with an Apple M4 (16 GB), Python 3.14.5, PyTorch 2.15.0. Short code excerpts are quoted with credit to Andrej Karpathy (MIT).

On this page
  1. Start here (no maths needed)
  2. Summary
  3. How the code works
  4. bpe.py versus our word tokenizer
  5. Run it, tested
  6. Connect to Robo-Helper (Module 18)
  7. Why it matters for robotics
  8. Reality check
  9. Self-check
  10. Further reading

Start here (no maths needed)

minGPT is a real GPT, the same design as the model behind early ChatGPT, written so plainly that you can read all of it in one evening.

Think of it like this

Picture a class where every pupil holds one word of a sentence. Before guessing the next word, each pupil may look at the pupils to their left, never the right, and decide whom to listen to most. They pass notes through a few rounds of this. Then the last pupil shouts a guess for the next word. That "who should I listen to?" step is called attention. A GPT is a stack of these rounds.

How it works

  1. Text is cut into small pieces called tokens, and each token becomes a number.
  2. Each number is looked up as a list of numbers (an embedding), plus a note of its position.
  3. A few Transformer blocks let every position look back at earlier positions (attention), then think on its own (a small neural network).
  4. The last layer gives a score for every possible next token. Pick a likely one, add it, repeat.
  5. Training means showing it real text and nudging its weights whenever it guesses wrong.

Why robots care

Robots are starting to use the same machinery. Some robot models turn camera pictures and commands into tokens, then write out actions as tokens, one at a time. On this page we train a tiny minGPT to read Robo-Helper's commands letter by letter and write its plan.

Try it

Words to know

Token
One piece of text the model reads: a letter, part of a word, or a whole word.
Transformer
The design inside GPTs: blocks of attention plus small neural networks.
Attention
Each position deciding how much to listen to each earlier position.
BPE
Byte Pair Encoding: building tokens by repeatedly gluing together the most common pairs of pieces.
Iteration
One training step: one batch of examples, one nudge of the weights.

Going deeper: everything below is the full version, with code. On a first read, skim the headings, the Run it section and the Robo-Helper results, and come back for the rest later.

1. Summary

TL;DR

minGPT is a PyTorch re-implementation of GPT in three library files: model.py (the Transformer, about 300 lines as the README says; we counted 310), bpe.py (GPT-2's byte-pair tokenizer) and trainer.py (a general training loop). We ran projects/adder on a MacBook CPU. After 6,001 iterations (243 s) the 0.09M-parameter model got all 500 held-out two-digit sums right. We also trained a character-level minGPT as a Robo-Helper brain, with honest results in section 5.

2. How the code works

model.py: four small classes

The heart of attention is these lines, quoted from CausalSelfAttention.forward:

att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1)))
att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf'))
att = F.softmax(att, dim=-1)
y = att @ v

From karpathy/minGPT, mingpt/model.py (MIT). The dropout line between softmax and att @ v is left out here.

This is Lesson 15's attention, done for every position, every head and every example in the batch at once. Score = dot product ÷ √size, softmax, weighted average. The one new idea is the mask. Setting future scores to −∞ makes their softmax weight exactly 0, so position 5 can only read positions 0–5. That is what lets one forward pass train on every next-token prediction in a sequence at the same time. The README says most of the complexity is "just being clever with batching".

The block, quoted whole

def forward(self, x):
    x = x + self.attn(self.ln_1(x))
    x = x + self.mlpf(self.ln_2(x))
    return x

From karpathy/minGPT, mingpt/model.py (MIT).

Each sub-layer adds a correction to x instead of replacing it. That's the GPT-2 "pre-norm" layout the README describes: LayerNorm moved to the input of each sub-block.

trainer.py and the projects

Trainer.run() is a plain loop: get a batch, forward, loss.backward(), clip the gradient norm to 1.0, AdamW step, call any callbacks, stop at max_iters. Nothing in it is specific to GPT. The projects/adder script defines a dataset of n-digit sums. The answer is written reversed and the loss is masked (target −1) on the question digits, so the model only learns to write answers.

3. bpe.py versus our word tokenizer

bpe.py is a lightly refactored copy of OpenAI's GPT-2 encoder. It first splits text with a regex (words with their leading space, contractions like 's, numbers, punctuation). Then it maps every byte to a printable character and repeatedly merges the adjacent pair with the best rank from OpenAI's list of 50,000 merges. The vocabulary is 50,257 tokens: 256 bytes, 50,000 merges and one end-of-text token. Our Module 18 tokenizer (Lesson 8's tokenize) is one regex: lowercase, then whole words and full stops.

We ran both on the same commands (the GPT-2 merge files are 1.4 MB, downloaded by bpe.py):

# 7 Nov 2026 · MacBook (Apple M4), CPU
'put teh red blok in the bin'
  BPE   ['put', 'Ġte', 'h', 'Ġred', 'Ġbl', 'ok', 'Ġin', 'Ġthe', 'Ġbin']   (Ġ = a leading space)
  ours  ['put', 'teh', 'red', 'blok', 'in', 'the', 'bin']   unknown to tinylm: ['teh', 'blok']
"where's the green one?"
  BPE   ['where', "'s", 'Ġthe', 'Ġgreen', 'Ġone', '?']
  ours  ["where's", 'the', 'green', 'one']                  unknown to tinylm: ["where's"]
'Put the RED block!'
  BPE   ['Put', 'Ġthe', 'ĠRED', 'Ġblock', '!']               (ids differ from 'put', ' red')
  ours  ['put', 'the', 'red', 'block']
GPT-2 BPE (bpe.py)Our word tokenizer (Lesson 8)
Vocabulary50,257, fixed, learned from web textWhatever words appear in our training commands
Unknown wordsNever. Anything falls back to smaller pieces, down to single bytesDropped: "teh" and "blok" are simply ignored by the tiny LM
Capitals, punctuationKept: "RED" and "red" are different tokensLowercased, punctuation removed except "."
SpacesPart of the token ("Ġred")Thrown away
Good forAny text in any languageA tiny, known command vocabulary

For the Robo-Helper brain in section 5 we used neither. We used characters: 30 symbols (a–z, space, apostrophe, question mark, and two control symbols). That's the simplest way to never meet an unknown word, which BPE gets with a big learned vocabulary.

4. Run it, tested

# 7 Nov 2026 · MacBook (Apple M4) · Python 3.14.5 · torch 2.15.0
git clone --depth 1 https://github.com/karpathy/minGPT mingpt-src/minGPT
python3 -m venv venv
venv/bin/pip install torch
venv/bin/pip install -e mingpt-src/minGPT
venv/bin/pip install regex requests      # needed by bpe.py, but not listed in setup.py

mkdir -p runs/cpu6k && cd runs/cpu6k
../../venv/bin/python ../../mingpt-src/minGPT/projects/adder/adder.py \
    --trainer.max_iters=6001 --trainer.device=cpu

The adder trains gpt-nano (3 layers, 3 heads, 48 wide: 0.09M parameters) on 9,500 of the 10,000 two-digit sums and tests on the other 500. It scores the whole train and test sets every 500 iterations. Real output, trimmed:

# 7 Nov 2026 · MacBook (Apple M4), CPU · adder, 6,001 iterations
number of parameters: 0.09M
iter 0: train loss 2.31454
train final score: 17/9500 = 0.18% correct     test final score: 0/500 = 0.00% correct
iter 1000: train loss 0.53160
train final score: 7323/9500 = 77.08% correct  test final score: 370/500 = 74.00% correct
iter 2000: train loss 0.22252
GPT claims that 6 + 94 = 90 but gt is 100
GPT claims that 96 + 4 = 90 but gt is 100
GPT claims that 37 + 92 = 139 but gt is 129
train final score: 9181/9500 = 96.64% correct  test final score: 481/500 = 96.20% correct
iter 3000: train loss 0.22115
train final score: 9489/9500 = 99.88% correct  test final score: 499/500 = 99.80% correct
iter 5000: train loss 0.03020
train final score: 9500/9500 = 100.00% correct test final score: 500/500 = 100.00% correct
iter 6000: train loss 0.02295
train final score: 9500/9500 = 100.00% correct test final score: 500/500 = 100.00% correct
real 242.67 s

Look at the mistakes at iteration 2000: they are nearly all carry errors (6 + 94 = 90, missing the 1 that should carry into the hundreds). The model learns digit-by-digit addition first and carrying last, much as a child does. From iteration 4,500 to the end it scored 100% on both sets at every check.

CPU or MPS?

# 7 Nov 2026 · same adder, 2,001 iterations each
--trainer.device=cpu   real 37.25 s    test 481/500 = 96.20%
--trainer.device=mps   real 183.67 s   test 478/500 = 95.60%

On the Apple GPU (MPS) it was about 5× slower. With batches of 64 sequences of 6 tokens and a 0.09M-parameter model, the cost of sending each small job to the GPU outweighs the maths. The two runs also differ slightly in accuracy because GPU and CPU arithmetic don't give bit-identical results.

What failed, and the fixes

5. Connect to Robo-Helper (Module 18)

Measured on 8 Nov 2026, with the first version of Robo-Helper's training data: 350 examples, which by accident included 3 of the 12 bake-off commands. On 9 Nov those were removed (343 examples). The small Module 18 brains' bake-off scores didn't change, and the tiny LM's held-out score went from 86% to 92%. Every brain here was compared on the same data, but if you re-run, expect the exact numbers to move by a few points.

Lesson 15's tiny language model is minGPT's little cousin. It reads the command's words, writes a plan one word at a time ("put red blue"), and has its backprop written by hand in the standard library. Its attention stretch computes exactly minGPT's score → softmax → weighted average, for one head and one query. minGPT is the next rung up: many heads, every position at once, a causal mask, residual blocks, and PyTorch working out the gradients.

The experiment: a character-level minGPT as Robo-Helper's brain

We built a brain called minigpt. It's a minGPT with 2 layers, 4 heads and 64-wide embeddings (110,208 weights), reading characters. It trains on exactly the same 350 examples as the tiny LM (lesson15_tiny_lm.training_data(), as (command, plan words) pairs). Each example becomes one string, with the loss masked on the command part, as in the adder:

put the red and blue blocks away|put red blue⏎

At run time it writes characters greedily until ⏎, turns the words into a plan with Lesson 15's words_to_plan, and passes every action through Lesson 10's validate_action, just like TinyLMBrain. Because it reads characters, we could also add typo augmentation: half the training commands get one random typo (swapped, dropped, doubled or neighbouring-key letter). Training: 1,500 iterations, batch 64, on the Mac CPU, about 75–185 s per model. We trained 3 seeds of each version and scored them on the held-out test_data() (50 plans), typo'd copies of it (150 with one typo, 150 with two), 10 hand-written typo commands, and the Lesson 19 bake-off.

# 8–9 Nov 2026 · MacBook (Apple M4), CPU · scripts/eval_minigpt.py · "+clean" = Lesson 8's typo fixer runs first
brain                       held-out  1-typo  2-typo  hand-10  bake-off  decide p50
words (Lesson 9)               34%      34%     33%     70%      5/12     0.05 ms
tinylm (Lesson 15)             86%      86%     81%    100%      8/12     0.24 ms
attention (Lesson 15)          92%      92%     87%    100%      8/12     0.25 ms
distilled (Lesson 17)          94%      93%     90%    100%     10/12     0.25 ms
minigpt, no typo-aug, seed 0   28%      25%     27%     50%      5/12     6.6 ms
minigpt, no typo-aug, seed 1   28%      28%     26%     60%      7/12     3.5 ms
minigpt, no typo-aug, seed 2   28%      27%     23%     60%      5/12     9.7 ms
minigpt, typo-aug, seed 0      60%      61%     61%     90%      7/12     6.7 ms
minigpt, typo-aug, seed 1      58%      58%     59%    100%      8/12     8.9 ms
minigpt, typo-aug, seed 2      52%      53%     51%     80%      7/12    10.1 ms

We also asked: how much do the word-level brains lean on Lesson 8's typo fixer? We scored them on the same typo sets with the fixer switched off:

# 9 Nov 2026 · MacBook (Apple M4), CPU · scripts/raw_typos.py
                         held-out  1-typo  2-typo
tinylm     no fixer         86%      73%     53%
attention  no fixer         94%      77%     59%
distilled  no fixer         94%      76%     59%
minigpt (typo-aug) no fixer, mean of 3 seeds:  57%  58%  56%

What the numbers say

The honest verdict: as a Robo-Helper brain, minGPT is a step sideways, not up. It shows why real LLMs need huge datasets. The architecture is not the bottleneck here. The data is.

Your next step

  1. Give minigpt more data the cheap way. Add Lesson 17's teacher labels (training_data(extra=...), as the distilled brain does). Then write 5 new PUT_TEMPLATES of your own. Retrain and check whether held-out accuracy climbs towards the tiny LM's 86%.
  2. Then shrink it: try n_layer=1, n_embd=32 and see whether a smaller Transformer overfits less on 350 examples.

6. Why it matters for robotics

7. Reality check

8. Self-check

What does the causal mask do, and why is it needed for training?

It sets attention scores to the future to −∞, so their softmax weight is 0. Each position can only use earlier tokens, so one forward pass can train every next-token prediction in the sequence at once without cheating.

Why does the adder write the answer backwards?

When you add, the lowest digit is worked out first and the carry moves left. Writing the answer reversed lets a left-to-right model produce digits in the same order the carries happen.

What does the target value −1 mean in the adder's dataset?

It is ignore_index for the cross-entropy loss. Positions marked −1 add nothing to the loss, so the model isn't trained to predict the question digits, only the answer.

Why can't GPT-2's BPE ever meet an unknown word?

Its base vocabulary includes all 256 bytes. Anything it can't merge into bigger pieces falls back to single bytes.

Why was MPS slower than CPU for the adder?

The model and batches are tiny. The fixed cost of launching work on the GPU is larger than the arithmetic saved.

9. Further reading