LLMModule 20 · From-scratch code · 4 of 6

llm.c: GPT-2 training in plain C, with every gradient written by hand

No PyTorch, no autograd, no Python. One C file runs GPT-2's forward pass, works out every gradient by hand, and updates 124 million numbers. Next to it sits a PyTorch version that checks the C gets the same answers.

Repokarpathy/llm.c
LicenceMIT
Stars31,406 (7 Nov 2026, GitHub API)
Core codetrain_gpt2.c 1,182 lines (CPU, fp32), test_gpt2.c 195, PyTorch reference train_gpt2.py 860, CUDA mainline train_gpt2.cu 1,904 (counted with wc -l)
StatusNot marked deprecated. The README calls the CPU file "an even earlier checkpoint" and the fp32 CUDA files "legacy". Last commit 10 May 2025.
NeedsA C compiler. OpenMP for speed (optional). The GPU version needs an NVIDIA GPU and CUDA. Python + PyTorch only to rebuild the data files.

Tested at commit f1e2ace on 7–8 Nov 2026, on a MacBook with an Apple M4 (4 performance + 6 efficiency cores, 16 GB) and Apple clang 17. Short code excerpts are quoted with credit to Andrej Karpathy and the llm.c contributors (MIT).

On this page
  1. Start here (no maths needed)
  2. Summary
  3. How the code works
  4. Run it, tested
  5. Connect to Robo-Helper (Module 18): does it fit the time budget?
  6. Why it matters for robotics
  7. Reality check
  8. Self-check
  9. Further reading

Start here (no maths needed)

Most AI code leans on a big helper library that does the hard calculus for it. llm.c throws the helper away and writes every single step by hand, in C. So you can see exactly what a language model does.

Think of it like this

Most people bake a cake from a packet mix: add eggs, stir, done. The packet hides what's inside. llm.c is like baking from scratch: you weigh the flour, sugar and butter yourself. It's more work, but you know what every ingredient does. And because nothing is hidden, a good baker can make it faster. PyTorch is the packet mix. llm.c is the from-scratch recipe, and it comes with the packet version next to it so you can check your cake tastes the same.

How it works

  1. Load GPT-2's 124 million numbers from a file straight into one big block of memory.
  2. Forward pass: push the text through each layer, one hand-written function per layer: layer norm, matrix multiply, attention, GELU, add back.
  3. Backward pass: go through the same layers in reverse. Each one has a hand-written function that works out how much every number was to blame for the mistake.
  4. Nudge all the numbers a little (the AdamW optimiser), and repeat.
  5. A test program checks that the C answers match PyTorch's answers.

Why robots care

A robot's computer is often small, and it can't always run Python and a 200 MB library. C runs almost anywhere, from a laptop to a tiny board. Knowing what a model costs, step by step, tells you whether it can think fast enough to stop before it hits the cat.

Try it

Words to know

C
An old, fast programming language that runs close to the hardware.
Autograd
A library feature that works out gradients for you automatically. llm.c doesn't use one.
Forward / backward pass
Running the model to get an answer, then running backwards to work out how to fix each number.
Kernel
One small, fast function that does one job, like "layer norm" or "multiply these matrices".
fp32 / bf16
Two ways to store a number: 32 bits (precise) or 16 bits (half the size, less precise).
OpenMP
A way to split a C loop across several CPU cores at once.

Going deeper: everything below is the full version. On a first read, look at the real outputs in Run it, tested and the time-budget table in the Robo-Helper section, then come back.

1. Summary

TL;DR

llm.c trains GPT-2 "in simple, pure C/CUDA with no need for 245MB of PyTorch or 107MB of cPython" (README). The CPU file train_gpt2.c is about 1,000 lines of fp32 code. Every layer has a forward function and a backward function written by hand, and test_gpt2.c checks them against the PyTorch reference train_gpt2.py. The CUDA mainline is mixed precision (bf16) and, per the README, "a bit faster than PyTorch Nightly (by about 7%)". On this M4, the CPU version fine-tuned GPT-2 (124M) on Tiny Shakespeare at 3–7 s per step. One forward pass for a 16-token command took about 60–110 ms, too slow for Robo-Helper's 0.2 s "stop!" budget once a plan needs 3 tokens.

2. How the code works

train_gpt2.c reads top to bottom like a textbook. First come the layer pairs: encoder_forward/backward (token + position embeddings), layernorm_*, matmul_*, attention_*, gelu_*, residual_*, softmax_forward and crossentropy_*. Then the GPT2 struct, the forward and backward passes, gpt2_update (AdamW), a data loader, a sampler and main.

The forward pass for one layer is just a list of calls. It's the same Block as nanoGPT's x = x + attn(ln_1(x)), spelled out:

// train_gpt2.c (MIT, Andrej Karpathy): inside gpt2_forward, one layer
layernorm_forward(l_ln1, l_ln1_mean, l_ln1_rstd, residual, l_ln1w, l_ln1b, B, T, C);
matmul_forward(l_qkv, l_ln1, l_qkvw, l_qkvb, B, T, C, 3*C);
attention_forward(l_atty, l_preatt, l_att, l_qkv, B, T, C, NH);
matmul_forward(l_attproj, l_atty, l_attprojw, l_attprojb, B, T, C, C);
residual_forward(l_residual2, residual, l_attproj, B*T*C);

The idea that makes it click: the backward pass is the forward pass read in reverse, one line at a time. Every forward line of the form out += a * b becomes two backward lines, "a's gradient gets b × out's gradient" and "b's gradient gets a × out's gradient". The attention backward even keeps the forward line as a comment:

// train_gpt2.c: attention_backward, through the value accumulation
// in the forward pass this was:
// out_bth[i] += att_bth[t2] * value_t2[i];
// so now we have:
datt_bth[t2] += value_t2[i] * dout_bth[i];
dvalue_t2[i] += att_bth[t2] * dout_bth[i];

That's exactly what you did by hand in Lesson 15's attention stretch, for one head and one query. llm.c does it for 12 heads, every position (with a causal loop t2 <= t), 12 layers and a batch. To make the backward pass possible, the forward pass saves what it will need, such as the attention weights att and layer norm's mean and rstd. That's why the activation memory (73,347,840 floats for B=4, T=64) is about half the size of the model itself.

OpenMP is the only speed trick in the CPU file. A line like #pragma omp parallel for collapse(2) above a loop splits it across cores. Without OpenMP, the pragmas are ignored and the code still runs, on one core.

3. Run it, tested

All commands were run on 7–8 Nov 2026 on a MacBook (Apple M4, 16 GB), in a scratch folder. Everything is CPU: llm.c's GPU code needs CUDA, which Macs don't have.

Starter pack: checked first, then a subset

The README's download_starter_pack.sh fetches 7 files. We checked their sizes with HTTP HEAD requests before downloading anything:

gpt2_124M.bin               497,904,640 bytes   (GPT-2 weights, fp32)
gpt2_124M_bf16.bin          248,952,832         (same weights, bf16)
gpt2_124M_debug_state.bin   549,369,860         (only for test_gpt2)
gpt2_tokenizer.bin              372,108
tiny_shakespeare_train.bin      611,544
tiny_shakespeare_val.bin         66,560
hellaswag_val.bin             3,762,386
total                     ≈ 1.30 GB

That's over our 1 GB limit, so we skipped the script and downloaded only what train_gpt2 needs: the fp32 weights, the tokenizer and Tiny Shakespeare (about 499 MB). That means test_gpt2 was not run: it needs the 549 MB debug state, which would have taken the total to about 1.05 GB. All the .bin files were deleted afterwards.

Build: the missing OpenMP, handled two ways

$ make train_gpt2
✗ nvcc not found, skipping GPU/CUDA builds
✗ OpenMP not found
cc -Ofast ... -march=native train_gpt2.c -lm -o train_gpt2

Apple's clang has no OpenMP library, and Homebrew's libomp wasn't installed. We didn't install anything system-wide. Instead:

  1. Single-thread build: make train_gpt2 NO_OMP=1. It works, and uses one core.
  2. Multi-thread build, no install: the PyTorch wheel in our scratch venv ships its own libomp.dylib and omp.h. We copied both into the scratch folder and compiled with the same flags the Makefile uses for Homebrew's libomp. The first attempt, linking straight to the copy inside PyTorch, failed at launch (Library not loaded: /opt/llvm-openmp/lib/libomp.dylib), because that library records a build path that doesn't exist on this Mac. Giving the copy an @rpath name fixed it:
cp venv/.../torch/lib/libomp.dylib venv/.../torch/include/omp.h omp/
install_name_tool -id @rpath/libomp.dylib omp/libomp.dylib && codesign -f -s - omp/libomp.dylib
clang -Ofast -march=native -Xclang -fopenmp -DOMP -Iomp train_gpt2.c \
      -Lomp -lomp -Wl,-rpath,$PWD/omp -lm -o train_gpt2_omp

Train: fine-tune GPT-2 (124M) on Tiny Shakespeare, 40 steps

$ OMP_NUM_THREADS=8 ./train_gpt2_omp          # 7 Nov 2026 · MacBook (Apple M4) · CPU, 8 threads
[GPT-2]
max_seq_len: 1024
vocab_size: 50257
padded_vocab_size: 50304
num_layers: 12
num_heads: 12
channels: 768
num_parameters: 124475904
train dataset num_batches: 1192
val dataset num_batches: 128
num_activations: 73347840
val loss 5.325528
step 0: train loss 4.677778 (took 3329.796000 ms)
step 1: train loss 5.191575 (took 4559.297000 ms)
...
val loss 4.416557
...
val loss 4.329176
generating:
---
I am pale and ill-smacking
And just 'with the eyes;
<|endoftext|>MONTAGUE:
Fication
Where we come, feast,
...
---
...
val loss 4.291537
generating:
---
Being barren savour, grant
Everyone, every man and woman,
Is heir to his life in unwritten words:
The Salmon of the family
How would gentlerish words indeed bring them
---
step 40: train loss 3.953056 (took 3205.787000 ms)
real 254.29

Val loss fell from 5.33 to 4.29 in 40 steps (batch 4 × 64 tokens, AdamW, learning rate 1e-4). The whole run, including two rounds of 63-token generation, took 4 min 14 s. Step times wandered between 2.9 and 7.0 s. With 8 threads on a chip with only 4 performance cores, some threads land on the slower efficiency cores.

The same program with 4 threads took 2.9–4.3 s per step (first 10 steps). The single-thread NO_OMP build took 10.7, 11.5 and 11.3 s for steps 0–2. The losses were identical to 6 decimal places in all three builds. For comparison, the README's M3 Max run shows about 1.3–1.5 s per step with 8 threads, and slightly different starting losses (val 5.252026). We didn't find out why the starting losses differ. A different version of the starter-pack files is one possibility, but we didn't check.

One measurement we threw away: a first attempt at the 4-thread and single-thread timings overlapped with the Mac going to sleep, and printed steps "taking" 974 and 2,985 seconds. Those are sleep, not compute. We reran them under caffeinate -i.

Inference latency: our own small harness

train_gpt2 only times training steps. To time one forward pass at batch 1, we wrote a 50-line harness, llmc_latency.c. It does #define TESTING and then #include "train_gpt2.c", exactly like the repo's own test_gpt2.c, then times gpt2_forward on T dummy tokens:

                                  T = 16 tokens            T = 64 tokens
                                  p50       p95            p50       p95
1 thread (NO_OMP build)           95.3 ms   113.4 ms       371.7 ms  412.1 ms
OpenMP, 4 threads                 57.9 ms    81.5 ms       143.5 ms  160.1 ms
OpenMP, 8 threads                 64.5 ms   129.4 ms       112.5 ms  190.7 ms
OpenMP, 4 threads (second run)    88.1 ms   107.7 ms       258.7 ms  320.6 ms

The two 4-thread runs differ by up to 1.8×. The efficiency cores, heat and whatever else the Mac was doing all add noise, so treat these as a range, not one number.

4. Connect to Robo-Helper (Module 18): does it fit the time budget?

Two Module 18 lessons meet here. Lesson 16, How fast is a brain? says a brain only fits if its p95 fits the budget: 0.2 s for "stop!". Lesson 17, Faster and cheaper showed that storing numbers with fewer bits (quantising) makes a model smaller and faster.

The measurement

Could Robo-Helper run a GPT-2-sized brain on its own small computer? We took llm.c's CPU GPT-2 (124M) as the candidate and used Lesson 16's own functions (time_it, summarize, thinking_distance, fits_budget). One Robo-Helper decision is a short plan such as put red <end>, which is 3 tokens. llm.c's CPU code has no KV cache, so it reruns the whole sequence for every token: one decision = 3 forward passes. Note that GPT-2 has never been trained on robot commands. This is only about what a model of that size costs.

$ python3 llmc_budget.py          # 8 Nov 2026 · MacBook (Apple M4) · CPU, llm.c with 4 OpenMP threads
brain                                          p50         p95       robot travels (p95, 1 m/s)  fits 0.2 s
tinylm (Lesson 15, pure Python)                0.27 ms     0.37 ms       0.04 cm                  yes
llm.c GPT-2 124M, 1 forward, T=16             88.10 ms   107.70 ms      10.77 cm                  yes
llm.c GPT-2 124M, decision = 3 forwards, T=16  264.30 ms  323.10 ms     32.31 cm                  NO
llm.c GPT-2 124M, 1 forward, T=64            258.70 ms   320.60 ms      32.06 cm                  NO
llm.c GPT-2 124M, decision = 3 forwards, T=64  776.10 ms  961.80 ms     96.18 cm                  NO
network round trip to Claude API (no thinking) 37.43 ms   135.62 ms     13.56 cm                  yes

For comparison, from the nanoGPT page: the 61K-parameter nanoGPT brain decided in 1.63 ms p95 on the CPU (6.45 ms on MPS). In the same script run, tinylm took 0.19 ms p95.

Which brain fits the robot's time budget?

fp32 vs bf16, and our quantising lesson

llm.c's CPU file is all fp32: 4 bytes per number, so 124,475,904 × 4 = 497.9 MB, which matches the starter-pack file plus its 1,024-byte header. The CUDA mainline trains in bf16 (2 bytes), and the bf16 file is 249.0 MB, exactly half. That's the same trade as Lesson 17's slider. Lesson 17 went from 32-bit to 8 and 4 bits on our 2,225-number model and kept 100% test accuracy, but broke at 2 bits (75%). By the same sum, GPT-2 at 4 bits would be about 62 MB. We didn't test GPT-2's accuracy at 4 bits.

The difference between bf16 and Lesson 17: bf16 is still a floating-point number (it keeps fp32's range and drops precision), while our quantiser rounds to a few whole numbers with a scale. On a CPU, fewer bytes mostly means less memory to read for every token, which is why quantised models are faster as well as smaller.

Next-step exercise. Run llmc_budget.py yourself (its header shows the build commands). Then answer with numbers: (1) How many tokens could GPT-2 write within 0.2 s on your machine? (2) If a KV cache made each extra token cost one position instead of the whole sequence, roughly how much would the T=16 decision shrink? (3) Lesson 17's quantised tinylm-4bit is a fraction of tinylm's size. Time both with time_it. Does fewer bits make pure Python faster, or only smaller? Why?

5. Why it matters for robotics

6. Reality check

7. Self-check

What does "no autograd" mean in practice in llm.c?

Every layer has a hand-written *_backward function, and gpt2_backward calls them in reverse order. Nothing records operations or works out derivatives automatically.

Why does the forward pass store things like att, mean and rstd?

The backward pass needs them to compute gradients. That's why activation memory grows with batch size and sequence length.

How does llm.c prove the C code is right?

The PyTorch reference train_gpt2.py writes a debug state (inputs, expected logits, loss and gradients). test_gpt2.c runs the C forward and backward on it, compares the results within a tolerance, and checks 10 training steps against PyTorch's losses.

Why did a 3-token GPT-2 decision miss the 0.2 s budget when one forward pass at T=16 fit?

Without a KV cache, each new token reruns the whole forward pass, so a 3-token plan costs 3 forwards, about 245–323 ms p95 here, over 0.2 s.

How big is GPT-2 (124M) in fp32 vs bf16, and why exactly half?

497.9 MB vs 249.0 MB. Same number of parameters, but 4 bytes each vs 2 bytes each.

8. Further reading