Scaling laws & data mixing

Before you train, decide three things: how big the model should be, how many tokens to feed it, and what those tokens should be. Scaling laws answer the first two. Deduplication, domain weights and the learning-rate schedule decide how much each token is worth.

Source: Suraj Sharma's roadmap post on X, stage 2 · all 12 stages

Going deeper: everything below is the full version, with some maths and code. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. What this stage covers

• Learn: Chinchilla's compute-optimal model size, near-duplicate removal with MinHash, domain weighting, and learning-rate schedules including cosine with warm restarts.
• Practice: train a ~100M-parameter model three times with different code/maths/web ratios, and plot how the loss curves separate.
• Why (the post's claim): that in 2026 architecture matters less than the exact ratio of the training data. We test this in the reality check.

2. Concepts, explained

Chinchilla-optimal compute

Kaplan et al. (2020) showed that loss falls as a power law in model size, data and compute, and concluded that extra compute should go mostly into a bigger model. Their setup, though, used one learning-rate schedule length for runs of different lengths. Hoffmann et al. (2022) redid the study with over 400 models (70M to 16B parameters, 5B to 500B tokens) and schedules matched to each run's length. Their answer: scale parameters and tokens equally. Double the model, double the data. They tested it by training Chinchilla (70B parameters, 1.4T tokens) on the same compute as Gopher (280B parameters, 300B tokens). Chinchilla beat Gopher on a wide range of tasks.

The paper used three estimation methods. The third fits a parametric loss surface:

L(N, D) = E + A / N^α + B / D^β
published fit: E = 1.69, A = 406.4, B = 410.7, α = 0.34, β = 0.28

C ≈ 6·N·D           (training FLOPs: ~2N forward + ~4N backward per token)
rule of thumb:  D ≈ 20·N   →   N ≈ √(C / 120)

E is the irreducible loss of the data. The A term is the penalty for a model that's too small. The B term is the penalty for too little data. Two caveats matter. First, the methods don't agree exactly: the first two give roughly equal exponents, while the published third fit leans toward smaller models. Besiroglu et al. (2024) tried to replicate the third method and found the published fit inconsistent with the paper's own data. Their refit agrees with the first two methods. Second, the loss constants depend on the dataset and tokenizer, so they don't transfer to your data. Fit your own on small runs (that's CS336 Assignment 3).

"Compute-optimal" means optimal for training cost. If a model will serve billions of queries, a smaller model trained on far more tokens is cheaper overall. Meta's Llama 3 8B, for example, was trained on about 15T tokens, nearly 2,000 tokens per parameter. Read Chinchilla as the starting point for an ablation budget, not a law about production models.

Data deduplication with MinHash

Web crawls are full of exact and near-exact copies: boilerplate, mirrored pages, templated spam. Lee et al. (2021) found a single 61-word sentence repeated over 60,000 times in C4. They showed deduplication cut verbatim memorised output roughly tenfold, needed fewer training steps for the same or better accuracy, and reduced train/test overlap. That overlap had affected over 4% of some validation sets.

Exact duplicates are easy: hash each document. Near-duplicates need MinHash (Broder, 1997). Turn each document into a set of shingles (for example word 5-grams). Similarity is the Jaccard index |A ∩ B| / |A ∪ B|. For a random hash function, the probability that two sets have the same minimum hash value equals their Jaccard similarity. So a signature of k minimum hashes estimates it. To avoid comparing every pair, split the signature into b bands of r rows and only compare documents that match exactly on at least one band (locality-sensitive hashing):

P(candidate pair | Jaccard s) = 1 − (1 − s^r)^b

with b = 16 bands, r = 8 rows (k = 128 hashes):
s = 0.5 → 0.06    s = 0.7 → 0.61    s = 0.8 → 0.95    s = 0.9 → 1.00

That S-shaped curve is the point: pairs above the threshold are almost always caught, pairs well below it almost never. You tune b and r to move the threshold. The FineWeb team (Penedo et al., 2024) ran ablations on MinHash settings and found that deduplicating each Common Crawl snapshot separately worked better than deduplicating across all snapshots at once.

Domain weighting

A pre-training corpus is a mixture of domains: web pages, code, maths, books, papers, other languages. The weights decide what the model is good at, and they interact with how much unique data each domain has. Upweighting a small domain means repeating it. Three ways to set them:

• By hand, with ablations. Train small proxy models on a few candidate mixes and compare on held-out sets and benchmarks. Most published recipes still work this way.
• Temperature sampling. Sample domain i with probability proportional to n_i^1/T, where n_i is its size. T = 1 is proportional to size. Larger T flattens the mix toward uniform. It's common for balancing languages in multilingual models.
• Learned. DoReMi (Xie et al., 2023) trains a small proxy model with Group DRO, which upweights the domains where the proxy lags most behind a reference model. It then uses the resulting weights for the big model. A 280M proxy set weights for an 8B model. On The Pile, it raised average few-shot accuracy by 6.5 points over the default weights and reached the baseline's accuracy in 2.6× fewer steps.

Quality filtering interacts with all of this. FineWeb-Edu, a 1.3T-token subset of FineWeb chosen by an educational-quality classifier, gave much better results on knowledge and reasoning benchmarks like MMLU and ARC. Often "which documents" matters as much as "which domain ratio".

Learning-rate schedules: cosine, warm restarts and WSD

Standard LLM pre-training uses a short linear warmup, then cosine decay to a fraction of the peak. Loshchilov & Hutter (2016) introduced SGDR, cosine annealing with warm restarts. The rate decays along a cosine, jumps back to the peak, and repeats, often with each cycle longer than the last. They showed it on CIFAR image models. In LLM pre-training, restarts are less common than a single decay. Chinchilla noted that letting the cosine cycle run much longer than the actual training run gives a worse model. That's one reason Kaplan's allocation was off: a run cut short in the middle of a long schedule never gets its final decay.

A popular alternative is warmup-stable-decay (WSD), described in the MiniCPM paper (Hu et al., 2024). Hold the rate constant, and decay sharply only at the end. You can branch a decay off any checkpoint, which makes it cheap to measure loss at several data sizes from one run.

def lr(step, peak=6e-4, warmup=500, total=20_000, floor=0.1):
    if step < warmup:
        return peak * step / warmup
    t = (step - warmup) / (total - warmup)
    return peak * (floor + (1 - floor) * 0.5 * (1 + math.cos(math.pi * t)))

def sgdr(step, peak=6e-4, t0=2_000, mult=2):
    t, T = step, t0
    while t >= T:              # find the position inside the current cycle
        t, T = t - T, T * mult
    return peak * 0.5 * (1 + math.cos(math.pi * t / T))

3. The practice project

(a) Colab version: a ~15M-parameter model, three mixes

1. Data. Take three public sources from the Hugging Face Hub: a web sample (for example a FineWeb-Edu sample), a code set (for example a small slice of The Stack) and a maths set (for example OpenWebMath). Tokenize all of them with the same tokenizer (GPT-2's via tiktoken is fine). Loss per token isn't comparable across tokenizers.
2. Clean. Run exact-hash and MinHash deduplication within each source. Then MinHash your validation sets against the training sets and drop overlaps, so you aren't measuring memorisation.
3. Mixes. Build three training streams with the same total tokens, for example web/code/maths at 80/10/10, 40/40/20 and 40/20/40.
4. Train. Use a nanoGPT-style model of about 15M parameters for about 300M tokens each (20 tokens per parameter). That's C ≈ 6 × 1.5×10⁷ × 3×10⁸ ≈ 2.7×10¹⁶ FLOPs per run, small enough for a Colab GPU in roughly an hour or a few per run, depending on how efficient your code is. Use the same seed, schedule and batch size for all three.
5. Measure. Every few hundred steps, log validation loss separately on held-out web, code and maths. Plot nine curves (3 mixes × 3 domains), and the difference between mixes per domain over time.

What to expect: each mix should usually do best on the domain it upweights. The interesting parts are how big the gaps are, when they open, and whether one domain helps another (does code help maths?). Run at least two seeds for one mix. If seed-to-seed differences are as big as mix-to-mix differences, you haven't measured anything yet.

A MinHash sketch to start from, standard library only:

import hashlib

def shingles(text, n=5):
    w = text.lower().split()
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}

def h(s, seed):
    return int.from_bytes(hashlib.blake2b(f"{seed}:{s}".encode(), digest_size=8).digest(), "big")

def minhash(sh, k=128):
    return [min(h(s, i) for s in sh) for i in range(k)]

def est_jaccard(a, b):
    return sum(x == y for x, y in zip(a, b)) / len(a)

def p_candidate(s, bands=16, rows=8):   # LSH: chance two docs share ≥ 1 band
    return 1 - (1 - s ** rows) ** bands

For real volumes, use datatrove, the library behind FineWeb. It has MinHash dedup stages that run in parallel.

(b) The full version from the post: ~100M parameters

• Budget. At 20 tokens per parameter, 100M parameters means ~2B tokens per run and C ≈ 6 × 10⁸ × 2×10⁹ = 1.2×10¹⁸ FLOPs.
• Time. An A100's dense bf16 peak is 312 TFLOP/s. At a realistic 40% utilisation (~125 TFLOP/s), one run takes about 1.2×10¹⁸ / 1.25×10¹⁴ ≈ 9,600 s, under 3 hours. Three mixes plus a repeat seed or two is roughly 8–15 GPU-hours. At typical cloud rental prices that's tens of dollars, not thousands. Small models rarely reach 40% utilisation without care, so budget extra.
• Code. Karpathy's build-nanogpt (GPT-2 124M) is almost exactly the right size and already has warmup plus cosine decay. Swap its data loader for a weighted sampler over your three domains.
• Stretch. Fit your own mini scaling law: train 4–5 sizes at 3 budgets, find each IsoFLOP minimum, and fit N_opt ∝ C^a. Or run a DoReMi-style reweighting with a 15M proxy and test its weights at 100M.

(c) Pitfalls

• Comparing total loss across mixes. A code-heavy mix has lower average loss partly because code is more predictable. Always compare per-domain validation loss.
• Repeating a small domain. If your maths set is small, a 40% maths mix repeats it many times. You're then testing repetition as much as the ratio.
• Schedule mismatch. If runs have different lengths, each needs its own full schedule. Comparing checkpoints from the middle of a cosine schedule biases everything toward longer runs.
• Leakage. Without deduplicating validation against training data, a mix with more duplicated pages looks better than it is.

4. Reality check

The post's claim: in 2026, architecture is secondary, and the real advantage is the exact mathematical ratio of the training data.

• True that data is a huge lever. FineWeb-Edu's quality filtering, deduplication's gains and DoReMi's 2.6× speedup on The Pile are all real, large effects from data alone, with the architecture held fixed.
• Overstated that it's about an exact ratio. Mixture weights are found empirically on proxy models, and loss is often fairly flat around a good mix, much like the flat bottom of the IsoFLOP curve in the demo. Quality filtering, deduplication, and the amount of unique data per domain usually matter more than the second decimal of a ratio.
• Overstated that architecture is secondary. Architecture still moves the scaling curve. Mixture-of-experts, attention variants and long-context designs change how much loss you get per FLOP. Module 05 covers one model credited with about 2.5× better scaling efficiency from architecture. Stage 03 goes further.
• Also worth knowing: "compute-optimal" isn't "best". Production models are usually trained far beyond 20 tokens per parameter because inference cost dominates. So the Chinchilla ratio is a research tool, not a production recipe.

5. In this course

• 30u30 Day 25 · Scaling Laws for Neural Language Models: the Kaplan et al. power laws, with an interactive loss-versus-size demo. Read it before this page's Chinchilla section.
• Module 04 · Evaluation that doesn't fool you: honest train/validation splits and leakage, the same discipline the per-domain validation sets need.
• Module 05 · Mixture of experts: architecture changing the compute-to-loss trade-off, the counterpoint to "data is everything".

6. Why it matters for robotics

7. Self-check

8. Courses & content

Start with the FineWeb write-up: it walks through deduplication, filtering and mixing with real ablation results, so you see which choices actually moved the numbers before you spend GPU time.

Full course map for all 12 stages →

9. Papers & docs

• Kaplan et al., 2020. Scaling Laws for Neural Language Models
• Hoffmann et al., 2022. Training Compute-Optimal Large Language Models (Chinchilla)
• Besiroglu, Erdil et al., 2024. Chinchilla Scaling: A replication attempt
• Lee, Ippolito, Nystrom et al., 2021. Deduplicating Training Data Makes Language Models Better
• Broder, 1997. On the resemblance and containment of documents (the original MinHash paper; not on arXiv)
• Xie, Pham, Dong et al., 2023. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
• Penedo, Kydlíček, Ben Allal et al., 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Loshchilov & Hutter, 2016. SGDR: Stochastic Gradient Descent with Warm Restarts
• Hu, Tu et al., 2024. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (the WSD schedule)
• datatrove documentation and examples