Synthetic data & curation pipelines

Use a strong model to write training data, use a checker to throw away the bad part, and train a smaller model on what survives. The generator is the easy bit. The checker, the diversity and the bookkeeping decide whether the result is any good.

Source: Suraj Sharma, X post, stage 4

Going deeper: everything below is the full version. On a first read, skim the headings, try the demo, and read the reality check.

1. What this stage covers

• Learn: Self-Instruct, evolutionary prompting (Evol-Instruct), rejection sampling, and using frontier models to generate and filter training data.
• Practice: generate 50,000 complex logic traces with a strong reasoning model, filter them with a verifier, and train a smaller model on the survivors.
• Why (the post's claim): high-quality human text is running out, so curating synthetic data is "the ultimate moat". Section 4 tests that claim.

2. Concepts, explained

Self-Instruct: bootstrapping instructions from a seed set

Self-Instruct (Wang et al., 2022) starts from a small pool of human-written seed tasks (175 in the paper). Each round, it samples a few tasks from the pool as in-context examples and asks the model for new instructions, then for inputs and outputs. Before a new task joins the pool, filters remove invalid outputs and near-duplicates: a new instruction is rejected if its ROUGE-L overlap with any existing one is too high. The pool grows, and later rounds draw on generated tasks too. Fine-tuning vanilla GPT-3 on the result gave a 33% absolute gain on Super-NaturalInstructions, close to InstructGPT-001.

pool = seed_tasks
repeat:
    examples  = sample(pool, k)                       # mix of seed and generated
    new_tasks = LM("write new tasks like these", examples)
    new_tasks = [t for t in new_tasks
                 if valid(t) and max_rougeL(t, pool) < threshold]
    pool += new_tasks

The lesson that carries over: diversity filtering is part of generation. Without the similarity check, the pool fills with paraphrases of the seeds.

Evolutionary prompting: Evol-Instruct

Self-Instruct's tasks tend to be easy. Evol-Instruct (Xu et al., 2023, the WizardLM paper) "evolves" each instruction by asking an LLM to rewrite it. In-depth evolution makes it harder: add a constraint, require more reasoning steps, make the input more complex. In-breadth evolution writes a new instruction on a related but different topic. An elimination step discards evolutions that failed (for example, the rewrite didn't add anything, or the model couldn't answer it). Repeating this for several rounds produces a mix of difficulty levels. The authors fine-tuned LLaMA on the mix and, in their human evaluations, preferred its outputs on the hardest instructions over ChatGPT's.

The general idea is a genetic algorithm over prompts: mutate, test, keep the fit ones. "Fit" here means the instruction is still well-posed and answerable, not just longer.

Rejection sampling: STaR and RFT

For tasks with a checkable answer, sample several solutions per problem and keep only the correct ones.

• STaR (Zelikman et al., 2022): the model writes a rationale and answer for each question. Keep rationales that reach the right answer. For questions it gets wrong, show it the correct answer and ask it to write a rationale that leads there ("rationalisation"). Fine-tune on all kept rationales and repeat.
• Rejection-sampling fine-tuning (RFT) (Yuan et al., 2023): sample many reasoning paths from a supervised model on maths problems, keep the correct ones, deduplicate, and fine-tune on them. They found that more distinct correct reasoning paths helped more than more copies of similar ones.

The arithmetic of a filter. Let p be the generator's pass rate, TPR the chance the verifier accepts a correct trace, and FPR the chance it accepts a wrong one:

yield      = p·TPR + (1 − p)·FPR                  # fraction of generations you keep
precision  = p·TPR / yield                       # fraction of kept data that is correct
generations needed for N survivors  =  N / yield

example: p = 0.10, TPR = 0.90, FPR = 0.05
  yield = 0.09 + 0.045 = 0.135     precision = 0.09 / 0.135 ≈ 0.67   → one kept trace in three is wrong

That's the base-rate trap. A verifier that looks "95% accurate" still lets lots of garbage through when most generations are wrong. Two fixes: make the verifier stricter (exact checks, unit tests), or make the generator better on the problems you pick (don't feed it problems far above its level).

Using frontier models to generate and filter

The most common pipeline in 2026 is distillation through data: a large "teacher" writes answers or reasoning traces, a filter keeps the good ones, a smaller "student" is fine-tuned on them. Three levers matter:

• What you ask for. phi-1 (Gunasekar et al., 2023) trained a 1.3B code model on about 6B tokens of filtered "textbook quality" web data plus about 1B tokens of synthetic textbooks and exercises written by GPT-3.5, and reached 50.6% pass@1 on HumanEval. The point: carefully designed synthetic data can stand in for a lot of raw web data, for a narrow domain.
• How you filter when there's no exact checker. An LLM judge scores helpfulness, correctness or style. Judges have known biases (length, position, self-preference), so calibrate them against a few hundred human labels first. Module 01 covers judge calibration.
• Curation beyond correctness. Near-duplicate removal (MinHash or embeddings), diversity sampling across topics and difficulty, and decontamination: drop anything with heavy n-gram overlap with your evaluation sets. Synthetic data generated by a model that saw the benchmark can leak it.

The risk: model collapse. Shumailov et al. (2023) showed that training each generation of model on the previous generation's output makes the tails of the original distribution disappear: rare facts, unusual styles, minority cases. The practical defences are to keep real data in the mix, filter for diversity, and avoid training repeatedly on your own model's unfiltered output.

3. The practice project

(a) Laptop / Colab version: verified logic traces for a 0.5B student

Pick a task where the verifier is exact and free, so you can study the pipeline without arguing about judges. Programmatically generated puzzles are ideal: you know the answer by construction.

1. Problems: write a generator for multi-step puzzles with a known answer, with a difficulty knob (e.g. chained arithmetic with 3–8 steps, or "who sits where" constraint puzzles solved by brute force). Make 3,000 train and 500 held-out test problems, with different random seeds.
2. Teacher: a 1.5–7B instruct model on Colab (with a fast inference library), or a modest API budget for a few thousand calls to a stronger model. Sample k = 4 traces per problem at temperature ~0.8, asking for working then a final line ANSWER: ….
3. Verify and curate: parse the final answer, check it exactly, drop traces with no parsable answer, deduplicate identical traces, cap at 2 kept traces per problem.
4. Student: fine-tune a ~0.5B model (LoRA is fine) on the survivors. Train a second student on unfiltered traces with the same token budget as the control.
5. Measure: yield per difficulty level; held-out accuracy of the base student, the unfiltered student and the filtered student; accuracy by difficulty level. Report all three numbers, not just the best one.

import random, re

def make_problem(rng, steps):
    x = rng.randint(1, 20); text = f"Start with {x}."
    for _ in range(steps):
        op, n = rng.choice("+-*"), rng.randint(2, 9)
        text += f" Then {'add' if op == '+' else 'subtract' if op == '-' else 'multiply by'} {n}."
        x = x + n if op == "+" else x - n if op == "-" else x * n
    return {"q": text + " What is the result?", "a": x, "steps": steps}

def verify(trace, answer):
    m = re.search(r"ANSWER:\s*(-?\d+)", trace)
    return m is not None and int(m.group(1)) == answer

def curate(problems, sample, k=4, keep=2):
    out, stats = [], {"gen": 0, "kept": 0}
    for p in problems:
        traces = {sample(p["q"]) for _ in range(k)}   # set = exact dedup
        stats["gen"] += k
        good = [t for t in traces if verify(t, p["a"])][:keep]
        stats["kept"] += len(good)
        out += [{"prompt": p["q"], "completion": t, "steps": p["steps"]} for t in good]
    return out, stats

rng = random.Random(0)
train = [make_problem(rng, rng.randint(3, 8)) for _ in range(3000)]

What to expect: the yield will fall as the number of steps grows, so the kept data will be skewed toward easy problems. That skew is the most useful thing to notice. If the filtered student improves mostly on easy problems, try oversampling hard problems (larger k) or the STaR-style "rationalise from the correct answer" trick for the ones that never pass.

(b) The full version from the post

50,000 complex logic traces from a heavy reasoning model, verified, then used to train a smaller model. Points to plan for:

• Generation cost is dominated by trace length. Reasoning traces are often thousands of tokens. At, say, 3,000 output tokens per attempt and a 30% yield, 50,000 survivors needs about 167,000 attempts and about 500M output tokens. Multiply by your provider's output price before you start, or plan for days of a multi-GPU node with an open reasoning model.
• The verifier is the project. "Logic traces" only works at scale if the final answer is machine-checkable (puzzles with unique solutions, maths with numeric answers, code with tests). If you rely on an LLM judge, measure its false-pass rate on a labelled sample, then use the demo's arithmetic.
• Student training: LoRA/QLoRA fine-tuning of a ~7B student on 50k traces fits on one 80 GB GPU. Full fine-tuning needs several. Long traces mean long sequences, so watch memory.
• Terms of use: some model providers restrict using their outputs to train competing models. Check before you build on one.

(c) Pitfalls

• Easy-problem bias. Rejection sampling keeps what the teacher already does well. Track yield per difficulty bucket.
• Right answer, wrong reasoning. An outcome check passes traces that guess or get lucky. Multiple-choice tasks are the worst case. Prefer answers with a large space of possible values.
• Contamination. If your problems resemble a public benchmark, run an n-gram overlap check before reporting results on it.
• Duplicates. Near-identical traces inflate the dataset without adding information. RFT found distinct reasoning paths matter more than count.
• No control. Without the unfiltered-trace baseline and the base model, you can't tell whether filtering helped or the extra training did.

4. Reality check

5. In this course

• Module 04 · teacher → student: distilling a big model into a small one, the same pipeline shape as this stage.
• Module 04 · evaluation that doesn't fool you and its pitfalls: honest splits and leakage checks for your student.
• Module 07 · SFT and RLVR verifiers: where synthetic traces enter post-training, and how verifiers also drive RL.
• Module 01 · LLM evals & calibration: calibrating an LLM judge before you trust it as a filter.
• Module 05 · length: Kimi K3 synthesises long-context tasks that can only be solved by combining scattered information.
• 06 Day 25 · Scaling laws: why data, not just parameters, limits scaling.

6. Why it matters for robotics

7. Self-check

8. Courses & content

Start with the smol-course synthetic data unit: it's short, hands-on and sized for small models, which matches the laptop project. Read Cosmopedia next for how diversity is handled at scale.

Full course map for all 12 stages →

9. Papers & docs

• Wang et al. (2022), Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Xu et al. (2023), WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (Evol-Instruct)
• Zelikman, Wu, Mu & Goodman (2022), STaR: Bootstrapping Reasoning With Reasoning
• Yuan et al. (2023), Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (RFT)
• Gunasekar et al. (2023), Textbooks Are All You Need (phi-1)
• Shumailov et al. (2023), The Curse of Recursion: Training on Generated Data Makes Models Forget
• Villalobos et al. (2022), Will we run out of data? Limits of LLM scaling based on human-generated data
• distilabel documentation