LLMUnit 1 · The foundations · Day 01

The Unreasonable Effectiveness of Recurrent Neural Networks

Train a network only to predict the next character, and it learns spelling, syntax and document structure for free.

AuthorAndrej Karpathy

Year / venue2015 · blog post

Core ideaNext-token prediction

Study time2–4 h

Original: karpathy.github.io · Code, notebook and exercises: 30u30 Day 1

Start here (no maths needed)

If a computer gets really good at guessing the next letter, it ends up learning how words, sentences and even computer code are put together.

Think of it like this

Play a game with a friend: you read out a sentence one letter at a time, and they guess the next letter. "The cat sat on th…" Your friend shouts "e!" To be good at this game, your friend has to know spelling, grammar and what usually comes next. Nobody taught them those rules for the game. They just know them from reading a lot. This computer program learns the same way, by playing the guessing game millions of times.

How it works

The program reads one letter.
It keeps a little "memory note" about everything it has read so far.
It uses the letter plus the note to guess the next letter.
If it guessed wrong, it adjusts itself a tiny bit so it does better next time.
Repeat this millions of times. Now it can write its own text, one letter at a time.

Why robots care

A robot also sees the world one moment at a time: camera picture, then the next one, then the next. Guessing "what happens next?" is how a robot can learn how the world works. For example, it can learn that a ball rolling toward the table edge is about to fall.

Try it

Cover a sentence in a book and reveal it one letter at a time. Guess each next letter. Which letters are easy to guess? (Hint: the one after "q".)
In the demo further down, slide the Temperature to the far left, then the far right. Left means the computer always picks its safest guess. Right means it takes wild guesses.

Words to know

Neural network: A computer program that learns from examples instead of being given rules.
RNN: A neural network with a memory note it carries from one step to the next.
Training: Showing the program lots of examples so it slowly gets better.
Temperature: A dial for how adventurous the guesses are.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

Karpathy trained character-level LSTMs on Shakespeare, Wikipedia, LaTeX and the Linux kernel source. With nothing but "predict the next character", the models produced text with correct form: speaker names in capitals, balanced brackets, indented code, plausible markup. They didn't get the meaning right. The post made a broad audience see that a simple predictive objective can force a network to learn deep structure. That's the same bet modern LLMs scale up.

Task: given the characters so far, output a probability for every possible next character. Train with cross-entropy.
Model: a recurrent network keeps a fixed-size hidden state that's updated once per character.
Result: syntax and layout emerge from data. Semantics mostly don't, at this scale.
Interpretability bonus: individual hidden units turned out to track things like "inside a quote" or "position in line".
The 30u30 repo builds a vanilla RNN in pure NumPy for clarity. Karpathy's experiments used LSTMs (that's Day 2).

2. The core idea

At each time step t, the RNN reads one input x_t (a one-hot character) and updates its hidden state:

h_t = tanh(W_xh · x_t + W_hh · h_{t−1} + b_h)
y_t = W_hy · h_t + b_y
p_t = softmax(y_t / T)        # T = sampling temperature

Three weight matrices, reused at every step. h_t is the network's entire memory of the past. Training unrolls the loop over a window of characters and back-propagates through all of it. This is backpropagation through time (BPTT), usually truncated to a fixed window such as 25–100 steps.

Generation runs the model in a loop: sample a character from p_t, feed it back in as the next input, repeat. Temperature T reshapes the distribution. Low T is safe and repetitive. High T is creative, then chaotic.

3. Deep dive

Why prediction forces understanding of structure

To lower its loss on Linux source, the model must know that { needs a matching }, possibly hundreds of characters later. That means storing "how many braces are open" in h_t. No one programs that counter. Gradient descent finds it because it reduces prediction error. This is the key intuition behind self-supervised learning: the data supplies its own labels, and a hard enough prediction task makes useful representations necessary.

What it couldn't do, and why that matters

Long-range dependencies fade. Gradients through many tanh steps shrink (vanishing gradients). LSTMs (Day 2) and later attention (Days 13–15) attack this.
Form ≠ meaning. Small models imitate surface statistics. Scale (Day 25) is what later turned "plausible-looking" into "often correct".
A fixed-size state is a bottleneck. All of the past must squeeze into one vector. Compare Module 05, where Kimi K3's linear-attention layers revisit exactly this trade-off at a million tokens.

Temperature, made concrete

Interactive: softmax temperature

Model logits for the next character after "th". Drag T and watch the distribution sharpen or flatten.

Temperature T:

P(top choice)

–

Entropy (bits)

–

4. Why it matters for robotics

Robot state is a sequence. Joint angles, IMU readings and camera frames arrive over time. Early learned controllers and dynamics models used exactly this recurrent shape: hidden state in, observation in, new hidden state and action out.
Predict-the-next-thing is how robots learn world models. Predicting the next frame or the next state from actions is the robotics version of next-character prediction. It's the self-supervised core of model-based RL.
Robot foundation models tokenise actions. Many VLAs discretise actions into tokens and predict them one at a time, autoregressively, exactly as this post samples characters.
Temperature has a robotics cousin: exploration noise. Too little and the policy repeats itself. Too much and it flails.

5. How to study it

Read Karpathy's post end to end (about 1 h). Note the visualisations of individual neurons.
In the 30u30 Day 1 folder, read implementation.py and run train_minimal.py on a small text. Watch samples improve over training.
Do the exercises in order, then sweep temperature on your trained model.
Stretch: train on a log file from a robot or any machine, and see what structure it picks up.

6. Self-check

What is the RNN's only memory of the past?

The hidden state vector h_t. Its size is fixed regardless of how long the input is.

Why does a character model learn to balance braces without being told?

Predicting a closing brace correctly lowers the loss. Storing the open-brace count in the hidden state is the cheapest way to do that, so gradient descent finds it.

What does lowering temperature toward 0 do to sampling?

It approaches greedy (argmax) decoding: the most likely character every time, which tends to loop and repeat.

Why is BPTT usually truncated?

Memory and compute grow with the unrolled length, and gradients over very long spans vanish or explode anyway.

7. Further reading

The original post and Karpathy's min-char-rnn.py gist (about 100 lines of NumPy)
30u30 Day 1: implementation, notebook, cheatsheet, exercises
Next: Day 02, Understanding LSTM Networks