LLMUnit 5 · Generative models & scaling · Day 25

Scaling Laws for Neural Language Models

Language-model loss falls as a smooth power law in parameters, data and compute. That makes the payoff of a bigger run predictable before you pay for it.

AuthorsJared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei

Year / venue2020 · arXiv preprint (OpenAI)

Core ideaLoss ∝ 1 / scale^α

Study time4–6 h

Original: arXiv:2001.08361 · Code, notebook and exercises: 30u30 Day 25

Start here (no maths needed)

If you make an AI model bigger, give it more reading material and more computer time, it gets better in a smooth, predictable way, so you can guess how good a huge model will be by testing small ones.

Think of it like this

Imagine you're practising free throws in basketball. Your first hour of practice helps a lot. Doubling your practice still helps, but a bit less each time. If you write down your score after 1, 2, 4 and 8 hours, you start to see a pattern. Then you can predict your score after 64 hours without doing all 64 hours first. The researchers did the same thing with AI models: they found the pattern and used it to predict how well much bigger models would do.

How it works

Train lots of small and medium language models of different sizes.
Measure how many mistakes each one makes when guessing the next word.
Draw the results on a special graph where the pattern becomes a straight line.
Extend the line to predict how a much bigger model would do.
Use that prediction to decide how to spend your computer budget: mostly on a bigger model, the paper said.

Why robots care

Robot builders want to know: if we collect ten times more videos of robots picking up cups, will the robot get much better at picking up cups? Nobody knows for sure yet. Robot practice is slow and costly, unlike text from the internet. Testing whether the same kind of pattern holds for robots is a big open question.

Try it

Time yourself doing a puzzle or a times-table quiz once a day for a week. Plot your time each day. Does it keep improving, and by less each day? Can you predict day 10?
In the demo further down, drag the Dataset size D slider to the left. The solid line flattens out: a bigger model stops helping when it has too little to read. Then drag it right again.

Words to know

Parameters: The adjustable numbers inside a model. More parameters means a bigger model.
Loss: A score for how many mistakes the model makes. Lower is better.
Compute: How much computer work the training takes.
Power law: A pattern where each doubling gives roughly the same percentage improvement.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

The authors trained hundreds of Transformer language models on WebText2, from 768 to about 1.5 billion non-embedding parameters, and measured test loss. When only one of model size N, dataset size D or compute C is the bottleneck, loss follows a straight line on a log-log plot. The shape of the network (depth versus width, number of heads) barely matters. Because the lines are so regular, you can fit them on small runs and forecast big ones. The paper's practical advice: given more compute, spend most of it on a bigger model, and stop training well before convergence.

Three power laws: loss versus N, versus D, and versus optimally used compute C_min. Each holds over several orders of magnitude.
Scale beats shape: within a wide range, the number of layers, width and heads change loss only a few percent at fixed N.
Overfitting is predictable: one formula L(N, D) covers both limits. Grow data roughly as N^0.74 to stay out of the overfitting regime.
Big models are more sample-efficient: they reach a given loss with fewer tokens and fewer steps.
Compute-optimal training stops early: with a fixed budget, the best model is large and undertrained. The paper fits N ∝ C_min^0.73.
Later work (Chinchilla, 2022) revised this allocation toward much more data per parameter. The power-law framing survived. The exact exponents did not.

2. The core idea

Each law has one constant and one exponent, fitted from experiments:

L(N)     = (N_c / N)^α_N         α_N ≈ 0.076   N_c ≈ 8.8 × 10^13  params
L(D)     = (D_c / D)^α_D         α_D ≈ 0.095   D_c ≈ 5.4 × 10^13  tokens
L(C_min) = (C_c / C_min)^α_C     α_C ≈ 0.050   C_c ≈ 3.1 × 10^8   PF-days

L(N, D)  = [ (N_c / N)^(α_N / α_D) + D_c / D ]^α_D

C ≈ 6 · N · (tokens processed)

L is cross-entropy test loss in nats per token. Lower is better.
N counts parameters excluding embeddings. Including them makes the trends much less clean.
α is the slope on a log-log plot. α_N ≈ 0.076 means doubling the model multiplies loss by 2^−0.076 ≈ 0.95, a 5% drop.
N_c, D_c, C_c set where the line sits. They depend on the tokenizer and dataset, so the paper says they have no deep meaning.
L(N, D) blends the two limits. With infinite data the second term vanishes and you get L(N). With an infinite model the first vanishes and you get L(D).
6N: about 2N floating-point operations per token for the forward pass and 4N for the backward pass. This is the standard back-of-envelope for training cost.

One PF-day is 10¹⁵ operations per second for a day, about 8.64 × 10¹⁹ operations.

3. Deep dive

Reading the plot: power laws and bottlenecks

A power law looks like a straight line on log-log axes. That's why it's so useful: you fit a line on cheap runs and extend it. But each law only holds when the other resources are plentiful. Fix the dataset and keep growing the model, and loss bends away from the line and flattens at the data floor. The combined formula L(N, D) captures that bend with no extra parameters. Try it below.

Interactive: loss versus model size

Log-log axes. Dashed: L(N) with unlimited data. Solid: L(N, D) with the dataset size you pick. Horizontal line: the data floor L(D). Constants from the paper.

Exponent α_N: Dataset size D: tokens

Loss × per 2× N

–

L(10⁹), no data limit

–

L(10⁹, D)

–

Data floor L(D)

–

Try this: set D to about 10⁸ tokens. Past roughly 10⁷ parameters the solid curve flattens. More model buys almost nothing. Now raise D and watch the line straighten again.

How to spend a compute budget

Given C, you choose N, the batch size and the number of steps. The paper's fits say most extra compute should go to a bigger model (N ∝ C^0.73), a little to bigger batches, and almost none to more serial steps. Data needs grow slowly, about D ∝ C^0.27. The surprising consequence: it's wasteful to train any model to convergence. Better to train a larger one and stop early. This reasoning shaped the very large, relatively undertrained models of 2020–2021.

Hoffmann et al. (2022, "Chinchilla") redid the analysis with learning-rate schedules matched to each run's length, which Kaplan's setup did not do. They found parameters and tokens should grow roughly equally with compute. In practice that means far more data per parameter than Kaplan suggested. Lesson: the existence of a power law is robust. The fitted exponents depend on experimental details, so treat them as estimates.

What else the paper found

Transformers versus LSTMs: LSTMs match Transformers on the first tokens of a context but fall behind later in the context. Compare Day 01 and Day 13.
Transfer: loss on other text distributions tracks WebText2 loss with a roughly constant offset.
Critical batch size grows as loss falls. For the largest models it was about 1–2 million tokens. Splitting such batches across many accelerators is what Day 24 (GPipe) is about.
Learning curves are predictable too: after a warm-up transient, loss versus steps follows its own power law, so early training can forecast the rest.

The "train big, then compress" pattern links to Module 04: once compute-optimal training produces a large model, distillation is how you get something small enough to serve. Module 05 shows the far end of that scaling path.

4. Why it matters for robotics

The big open question is whether robot data scales the same way. Text is nearly free. Robot data costs operator time and hardware hours. Early studies of imitation learning report power-law-like gains as you add more environments and objects, but there is no robotics equivalent of Kaplan's clean, multi-decade curves yet.
Loss is not task success. Kaplan measures next-token loss. A robot policy is judged by whether the grasp works. Small loss gains can mean large or zero changes in success rate, so fit scaling curves on the metric you care about.
Diversity may matter more than volume. Pooling data across many robots and labs (e.g. Open X-Embodiment) is a bet that varied data scales better than more of the same demonstrations.
Simulation turns compute into data. RL in massively parallel simulators (Day 29) is one way to get "more D" for robots. Sim-to-real gaps decide how much of it counts.
The method transfers even if the exponents don't: run a sweep of small models, plot on log-log axes, and extrapolate before you commit a large training budget.

5. How to study it

Read sections 1.1–1.2 of the paper (the summary and the equations), then section 6 on compute allocation. Skim the appendices for the fit details. About 2 h.
In the 30u30 Day 25 folder, read implementation.py: a small Transformer, a power-law fitter and a compute calculator. Run train_minimal.py to train a few tiny models and fit a line through their losses.
Open notebook.ipynb and reproduce a log-log plot. Check the 6N rule by hand for one model.
Do the exercises in exercises/ in order (fitting, compute, bottleneck, prediction, Transformer).
Stretch: read the Chinchilla paper and explain in two sentences why its allocation differs.

6. Self-check

Why does a power law look like a straight line on a log-log plot?

If L = (N_c/N)^α, then log L = α·log N_c − α·log N. That's linear in log N with slope −α.

With α_N ≈ 0.076, how much does a 10× bigger model cut the loss?

By a factor of 10^−0.076 ≈ 0.84, about 16% lower, assuming data isn't the bottleneck.

Why exclude embedding parameters from N?

The authors found the trends were much cleaner without them. Embedding tables add many parameters that contribute little to per-token compute or capability.

Estimate the training compute for a 1B-parameter model on 20B tokens.

C ≈ 6 × 10⁹ × 2 × 10¹⁰ = 1.2 × 10²⁰ FLOPs, about 1.4 PF-days.

What does L(N, D) predict if you keep growing the model on a fixed dataset?

Loss approaches the floor (D_c/D)^α_D. Extra parameters stop helping and the model overfits.

What did Chinchilla change, and what stayed?

It found that parameters and tokens should grow roughly in step with compute, not mostly parameters. The idea that loss follows predictable power laws stayed.

7. Further reading

The original paper (Kaplan et al., 2020)
Training Compute-Optimal Large Language Models (Hoffmann et al., 2022), the Chinchilla revision
30u30 Day 25: implementation, notebook, cheatsheet, exercises
Next: Day 26, Kolmogorov Complexity and Algorithmic Randomness