Paper-to-production prototyping

Read a new paper, find the small piece of code that carries its idea, drop it into a strong baseline, and measure honestly whether it helps. The last step is where most reproductions go wrong.

Source: Suraj Sharma's roadmap post on X, stage 11

Going deeper: everything below is the full version. On a first read, skim the headings, try the demo and read the "Why it matters for robotics" box.

1. What this stage covers

• Learn: reading a paper for its mechanism rather than its story; separating the new idea from the scaffolding around it; isolating it as a small, switchable kernel.
• Practice: pick a paper from this week, write the core algorithm in under 500 lines, and benchmark it against the baseline it claims to beat.
• Why (the post's claim): how quickly you can turn academic maths into working code is your main value to an AI lab. Section 4 argues that speed matters, but correctness and honest measurement matter more.

2. Concepts, explained

Rapid arXiv reproduction

Reading for reproduction is different from reading to learn. You're looking for an executable spec. A workable order:

1. Abstract, main figure, main results table. Write down the claim in one sentence, with its number: "X improves Y on benchmark Z by N points at the same compute."
2. The method section and any algorithm box. Translate each equation into tensor shapes. If you can't say the shape of every symbol, you don't understand it yet.
3. The appendix. Hyperparameters, initialisation, warmup, data processing and evaluation details live here. Most reproduction failures trace back to something in the appendix, or missing from it.
4. Code, if released. Diff the code against the paper. Where they disagree, the code is what produced the numbers.

Henderson et al. (2017), Deep Reinforcement Learning that Matters, showed how fragile this is in RL. Random seeds, environment non-determinism and implementation choices moved reported results enough to change conclusions, and the authors called for significance testing and standard reporting.

Stripping academic fluff

Papers are written to be accepted, so the framing often outgrows the mechanism. Stripping means separating three things: the mechanism (the change in the computation), the scaffolding (everything else the authors changed while they were at it) and the story (why they say it works). Only the first is what you reproduce. The ablation table is your best friend: it shows which parts carry the gain.

Be careful, though. The "fluff" is sometimes load-bearing. Engstrom et al. (2020), Implementation Matters in Deep Policy Gradients, found that code-level optimisations, details found only in implementations or mentioned in passing, accounted for most of PPO's reward gain over TRPO. Strip too much and your "faithful" reproduction underperforms for reasons that have nothing to do with the idea. The cure is to keep a list of every deviation you make and test the suspicious ones.

The minimum viable kernel

The post's phrase means the smallest piece of code that, swapped into a strong baseline, carries the claimed effect. For most papers this is a function or a module: a new attention variant, a normalisation, an optimiser update, a loss term, a sampling rule. A good kernel has three properties:

• A switch. One flag turns it on or off, and nothing else differs between the two runs.
• A reduction test. With its new hyperparameter at a neutral value, it should give exactly the baseline's output. That one unit test catches a surprising number of bugs.
• Shape and gradient tests. Check output shapes, and check gradients numerically on a tiny input (torch.autograd.gradcheck in double precision).

class Block(nn.Module):
    def __init__(s, cfg):
        super().__init__()
        s.attn = NewAttn(cfg) if cfg.use_new else Attn(cfg)   # the only change
        ...

def test_reduces_to_baseline():
    torch.manual_seed(0); x = torch.randn(2, 16, 64)
    new = NewAttn(cfg, alpha=0.0); old = Attn(cfg)
    new.load_state_dict(old.state_dict(), strict=False)
    assert torch.allclose(new(x), old(x), atol=1e-6)

Narang et al. (2021), Do Transformer Modifications Transfer Across Implementations and Applications?, is the cautionary tale. They tested many published Transformer modifications in one shared codebase and found most didn't meaningfully improve performance. Many of those that helped were developed in that same codebase or were minor changes. A kernel that wins in its home codebase may not win in yours.

Benchmarking against a baseline honestly

This isn't on the post's Learn list, but its Practice step depends on it. Four rules:

1. Tune the baseline as hard as the new method. Melis, Dyer and Blunsom (2017), On the State of the Art of Evaluation in Neural Language Models, re-tuned standard LSTMs with large-scale hyperparameter search and found they beat several newer architectures. Many "wins" were really tuning-budget gaps.
2. Match the budget. Same tokens, same steps or same FLOPs, stated explicitly. A method that wins only because it ran longer hasn't won.
3. Run several seeds. Bouthillier et al. (2021), Accounting for Variance in Machine Learning Benchmarks, show data sampling, initialisation and hyperparameter choice all move results markedly, and recommend varying them when comparing.
4. Report intervals, not points. Agarwal et al. (2021), Deep RL at the Edge of the Statistical Precipice, found point estimates from a few runs led to conclusions that a proper statistical analysis didn't support. They recommend bootstrap confidence intervals and robust aggregates such as the interquartile mean.

When both models are scored on the same test examples, use a paired comparison. Take the per-example difference, then bootstrap over examples:

d_i  = score_B(x_i) − score_A(x_i)              for each test example i
Δ̂    = mean(d)
repeat R times: resample n indices with replacement → mean of d at those indices
95% CI = 2.5th and 97.5th percentiles of the R resampled means
"B beats A" only if the whole CI is above 0

Pairing removes the shared difficulty of each example: if both models fail the same hard questions, that doesn't add noise to the difference. Two overlapping unpaired intervals don't mean "no difference". A paired interval that crosses zero does mean you can't tell. Module 04 makes the same point about bootstrapping over clips.

3. The practice project

(a) Laptop / Colab version: one kernel, one baseline, honest numbers

1. Baseline. Clone nanoGPT and train its character-level Shakespeare config. The README describes a GPU run of a few minutes and a smaller CPU setting. Record validation loss. Run it with three seeds to see your noise floor before you change anything.
2. Pick a paper. Choose a recent one whose core idea is a local change: an attention or normalisation variant, an optimiser, a loss term, a positional encoding. Write the claim in one sentence, with its number.
3. Write the kernel behind a use_new flag, with the reduction and gradient tests above. Keep a deviations list in the README.
4. Match budgets and tune both. Same iterations, same batch, same data. Give the baseline and the new method the same small learning-rate sweep (for example three values each).
5. Compare. At least three seeds per arm; report mean and range of validation loss, and a paired bootstrap over evaluation batches if you score both on the same batches.
6. Write it up (that's Stage 12): claim, setup, result with intervals, deviations, and what this does not show.

What to expect: no promise. Given Narang et al.'s findings, a common honest outcome at this scale is "within seed noise". That's a valid result if your baseline was tuned and your kernel passed its tests. A tiny character model also may not show effects that only appear at scale, so say so.

import numpy as np

def paired_ci(a, b, R=2000, seed=0):
    d = np.asarray(b, float) - np.asarray(a, float)   # per-example B - A
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, len(d), (R, len(d)))
    means = d[idx].mean(axis=1)
    lo, hi = np.percentile(means, [2.5, 97.5])
    return d.mean(), lo, hi

obs, lo, hi = paired_ci(scores_a, scores_b)
print(f"B-A = {obs:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]",
      "real" if lo > 0 or hi < 0 else "can't tell")

(b) The full version from the post

Take a paper from this week, implement its core algorithm in under 500 lines of PyTorch, and benchmark it against its baseline at the scale the paper used. Cost depends entirely on the paper. An optimiser or attention tweak tested on GPT-2-small-sized models fits on a single rented GPU over days. A claim made at 7B parameters and hundreds of billions of tokens is out of reach for an individual; then reproduce the trend at small scale and say clearly that you didn't test the paper's scale. CS336's assignments are good preparation: little scaffolding, real budgets.

The 500-line target is realistic for the kernel plus a minimal training loop. It isn't realistic for data pipelines, distributed training or evaluation harnesses. Reuse those from a trusted baseline repo rather than rewriting them.

(c) Pitfalls

• Weak baseline. Untuned, under-trained or an older version. The single most common way to "win".
• Unequal budgets. The new method silently gets more steps, more parameters or a better learning rate.
• One seed. A single run per arm says almost nothing about small differences.
• Picking by test score. Choose checkpoints and hyperparameters on a dev split; touch the test set once.
• Unlisted deviations. If you changed the tokenizer, the warmup or the eval script, write it down, or nobody, including you, can interpret the result.
• Too-fresh papers. A paper from this week has no code, no reviews and maybe errors. That's part of the exercise, but budget time for emailing authors and for "the paper is ambiguous here".

4. Reality check

The post's claim is that translation speed, from maths to working code, is your main value to an AI lab.

• True: research engineers spend much of their time implementing ideas (their own team's and others') and turning them into experiments. Being fast and fluent in PyTorch matters, and practice on real papers is the best way to get it.
• Overstated: a fast wrong reproduction has negative value. It can kill a good idea or promote a bad one. Engstrom et al. and Narang et al. are both, in effect, about the cost of fast but uncontrolled comparisons. The scarcer skills are judgement (which ideas are worth trying at all), debugging at scale, and measurement people can trust.
• Missing: in a lab, most new ideas come from inside, not arXiv. Reproducing external papers is practice for implementing your team's ideas, where nobody has a reference number to check against.

5. In this course

• Module 06 · 30u30: thirty from-scratch reproductions with exercises. The best structured practice for this stage. Start with Day 14 (the Annotated Transformer), a line-by-line paper-to-code translation.
• Module 06 · Day 29, PPO: the algorithm at the centre of Engstrom et al.'s implementation-matters study.
• Module 06 · Day 25, Scaling laws: why a small-scale win may not hold at scale, and vice versa.
• Module 04 · Evaluation that doesn't fool you: paired bootstrap, honest splits, never choosing by test score.
• Module 01 · Tuning without fooling yourself: the dev/test discipline behind every comparison.
• Module 02 · TensorTonic: implementation drills for the PyTorch fluency this stage assumes.

6. Why it matters for robotics

7. Self-check

8. Courses & content

Start with nanoGPT: it's the baseline you'll drop kernels into for the laptop project, and it's small enough to read in an afternoon.

Full course map for all 12 stages →

9. Papers & docs

• Henderson, Islam, Bachman, Pineau, Precup, Meger (2017). Deep Reinforcement Learning that Matters
• Melis, Dyer, Blunsom (2017). On the State of the Art of Evaluation in Neural Language Models
• Engstrom, Ilyas, Santurkar, Tsipras, Janoos, Rudolph, Madry (2020). Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• Narang, Chung, Tay, Fedus et al. (2021). Do Transformer Modifications Transfer Across Implementations and Applications?
• Bouthillier, Delaunay, Bronzi et al. (2021). Accounting for Variance in Machine Learning Benchmarks
• Agarwal, Schwarzer, Castro, Courville, Bellemare (2021). Deep Reinforcement Learning at the Edge of the Statistical Precipice
• PyTorch docs: torch.autograd.gradcheck