LLM post-training: demonstrations, comparisons and checkers (and the robot equivalents)
Pre-training teaches a model to continue text. Post-training turns it into something that helps. Robot foundation models go through the same two stages.
Start here (no maths needed)
A model that has read the whole internet knows a lot but doesn't know how to help. A second round of training, with three kinds of teacher, fixes that.
Think of it like this
Imagine someone who has read every book in a huge library but has never had a job. Start a sentence and they can finish it. But ask them for help and they might just keep chatting about the topic instead of doing what you asked. To become a good helper, they need training on the job, and there are three kinds of teacher:
- A teacher who shows you: "Here's a request, here's a great answer. Copy that."
- A teacher who compares: "Of these two answers, this one is better." (A food critic can't always cook, but can tell which dish tastes better.)
- A teacher who checks: like a maths answer key. Right or wrong, no arguing.
How it works
- Pre-training: the model reads trillions of words and learns to guess the next word. That's the library-reading part (see Module 06, Day 1).
- Learning from examples: it practises on requests paired with good answers.
- Learning from comparisons: people pick the better of two answers, and the model leans toward what people prefer.
- Learning from a checker: for maths or code, a program marks each attempt right or wrong, and the model does more of what worked.
- Throughout, a "leash" keeps the model close to what it already knew, so it doesn't forget everything while chasing points.
Why robots care
Robot "brains" are now trained the same way. First they learn general knowledge from lots of pictures, videos and robot recordings. Then they're post-trained: people show the robot how to fold a towel (examples), compare two attempts (comparisons), or let a sensor check whether the towel ended up folded (checker).
Try it
- Ask a friend to finish this: "Explain the Moon landing to a 6-year-old…". A "just keep going" brain might add "Explain gravity to a 6-year-old. Explain…", because lists of questions are common online. A helpful brain answers the question.
- In the demo below, drag the Leash strength β slider all the way left. The model piles all its chances onto the answer the judge scores highest, even if it's a trick answer. Move it right and the model stays close to where it started.
Words to know
- Pre-training
- The first, huge training stage: learn to predict the next word (or next action).
- Base model
- A model that has only been pre-trained.
- Post-training
- Everything done afterwards to make the model helpful and safe.
- Reward model
- A "judge" model trained to score answers like people would.
- Verifier
- A program that checks whether an answer is right, such as tests for code.
Going deeper: everything below is the full version. On a first read, skim the headings, the robotics mapping table and the demo.
- Summary
- Why this matters for robotics
- Deep dive: what a base model is missing
- Deep dive: pre-training vs post-training
- Deep dive: instruction tuning (learn from demonstrations)
- Deep dive: preference alignment and the KL leash (interactive)
- Deep dive: RLVR (learn from a checker)
- Playbook
- Self-check
- Glossary & further reading
1. Summary
TL;DR
Pre-training only teaches next-token prediction on web text. The result, a base model, knows a lot, but it doesn't reliably follow instructions, it isn't steered away from harm, and it struggles with long chains of reasoning. Post-training keeps training the same model on small, curated data with a different signal. There are three main families, each with a different "teacher":
- Instruction tuning (SFT): learn from a demonstration.
- Preference alignment: RLHF with a reward model and PPO, or DPO. Learn from a comparison.
- RLVR: learn from a checker.
In the author's phrase: pre-training decides what a model knows, and post-training decides how it uses that knowledge.
- The opening example (InstructGPT, OpenAI, 2022): asked to explain the Moon landing to a 6-year-old, GPT-3 produced more prompts of the same kind ("explain gravity…", "explain relativity…"). That's a reasonable continuation of web text, where question lists are common. The post-trained InstructGPT simply answered.
- Same model, different data and signal: trillions of raw tokens vs. at most a few million curated examples. Most of the compute goes to pre-training.
- Small data can still have large effects: the article cites a study where mixing about 500 safety examples into a large instruction set noticeably reduced harmful answers.
- The KL penalty is a "leash" tying the post-trained model to the original, so it doesn't drift away while chasing reward and forget what pre-training taught it.
- Verifiers can't be sweet-talked, but they're only as good as their rules. A checker that sees only the final answer says nothing about the steps.
2. Why this matters for robotics
This is the training recipe for current robot foundation models (VLAs). The three teachers map almost one-to-one:
| LLM stage | Robot equivalent | Teacher |
|---|---|---|
| Pre-training on web text | A vision-language backbone pre-trained on web images and text, then trained on large multi-robot datasets (many robots, many tasks) | Next token / next action |
| Instruction tuning (SFT) | Behaviour cloning: fine-tune on teleoperated demonstrations for your robot and task. Condition on camera + instruction, and score only the actions. | Demonstration |
| Preference alignment | Humans compare two robot rollouts and a reward model learns from the choices: Module 06, Day 30, the original deep-RL-from-preferences paper, used simulated robots. | Comparison |
| RLVR | RL with a success detector: is the block in the bin? Is the door open? In simulation, the simulator itself is a perfect verifier. | Checker |
| KL leash | Keep the fine-tuned policy close to the demonstrated behaviour (a trust region, as in PPO, Day 29), because unfamiliar actions on real hardware can break things. | Regulariser |
The robotics twist: robot demonstrations are expensive (a person teleoperating for hours) while text is nearly free. So the "small, curated post-training data" point is even stronger for robots. A few hundred good demonstrations of one task is typical, which is why the pre-trained backbone matters so much.
3. Deep dive: what a base model is missing
The article names three gaps, with one root cause: the training objective (predict web text well) doesn't match the goal (be helpful and harmless).
- Instruction following: asked to translate, a base model may just keep writing in the source language, because that's a likely continuation.
- Harm: nothing in next-token prediction penalises dangerous advice, abuse or confidently wrong medical answers.
- Long-chain reasoning: knowing what a maths solution looks like is far from producing a correct one step by step.
This is the same "objective mismatch" theme as Module 01: optimise the wrong metric and you get exactly what you asked for, not what you wanted.
4. Deep dive: pre-training vs post-training
| Pre-training | Post-training | |
|---|---|---|
| Goal | Predict the next token in text | Behave as people want: follow instructions, stay safe, get answers right |
| Data | Raw text, trillions of tokens, mostly web | Small and curated: prompt+answer pairs, A-vs-B comparisons, checkable problems. Large instruction sets reach a few million examples. |
| Signal | Every next token in the text | A demonstration, a "which is better" judgment, or a checker's right/wrong |
| Compute | The vast majority | A small fraction |
| Change to the model | Builds knowledge and language from scratch | Gently steers an already-capable model |
Note "same model": post-training doesn't build a new network. It continues optimising the same weights with new data and a new loss. Module 05 shows a frontier example. Kimi K3's post-training combines SFT, RL across domains, and distilling nine specialist teachers back into one model.
5. Deep dive: instruction tuning (learn from demonstrations)
The data is (instruction, good response) pairs. The training rule barely changes. It's still next-token prediction, with one twist: the loss is only computed on the response tokens. The instruction is context, not something to imitate.
loss = − Σ_{t ∈ response} log p(token_t | instruction, response_<t)- Also called supervised fine-tuning (SFT), because every example has a correct answer to copy.
- Training across many task types improves instruction-following in general, including on tasks never seen in training.
- Limit: the model can only be as good as the demonstrations, and it learns their style and mistakes along with their substance. That's the same caveat as the teacher → student distillation in Module 04.
Robotics: behaviour cloning is SFT with actions as the "response tokens". Condition on images and the instruction, and maximise the likelihood of the demonstrated actions. The classic failure is compounding error: a small mistake leads to a state no demonstration covered, which leads to a bigger mistake. That's why corrective data (e.g. DAgger) and the next two stages matter.
6. Deep dive: preference alignment and the KL leash
Writing a perfect answer is hard, but judging which of two answers is better is easy. Preference data is (prompt, answer A, answer B, which one people prefer). There are two routes:
- RLHF: first train a reward model to score answers as people would. Then optimise the language model against it with RL, classically PPO. This is the InstructGPT recipe.
- DPO (Direct Preference Optimization): skip the separate reward model and learn directly from the pairs.
Both include a KL penalty to the original (reference) model. The objective being maximised is:
E[ reward(x, y) ] − β · KL( π(·|x) ‖ π_ref(·|x) )This has a neat closed-form optimum, the basis of DPO's derivation:
π*(y|x) ∝ π_ref(y|x) · exp( reward(x, y) / β )β is the leash. Large β keeps you near the reference model. Small β lets the policy chase reward, including any flaw in the reward model. That flaw-chasing is called reward hacking: the reward model is a learned classifier with errors (see Module 01), and an optimiser will find them.
Interactive: the KL leash
Five candidate answers to one prompt. Grey bars: the reference (SFT) model's probabilities. Coloured bars: the post-trained optimum π* ∝ π_ref · exp(r/β). Answer E is a reward hack: long and flattering, so the reward model over-scores it, but people actually rate it low.
7. Deep dive: RLVR (learn from a checker)
Some answers can be checked by a program: a maths problem has a final number, and code has tests that pass or fail. Reinforcement Learning with Verifiable Rewards uses such a verifier as the teacher:
- Sample several attempts at the same problem.
- The verifier marks each one right or wrong.
- Raise the probability of the correct attempts and lower the others. Group-relative methods (e.g. GRPO) score each attempt against the group's average, so no separate value model is needed.
- Strengths: no human judge is needed, and a verbose, polished wrong answer still scores zero. That makes RLVR well suited to the long-chain reasoning base models are weak at.
- Caveat: a verifier is only as reliable as its rules. If it checks only the final answer, it can reward lucky guesses or broken reasoning that happens to land right.
Robotics: a success detector is a verifier. In simulation it's exact: the simulator knows if the cube is in the bin. In the real world it's often a learned classifier or a sensor, and the same caveat applies with physical consequences. A detector that only checks "cube near bin" can reward a robot that knocks the bin over onto the cube. Tune its threshold with the methods in Module 01 before trusting it as a reward.
8. Playbook
- Start from the strongest pre-trained model you can afford. Post-training steers capability, it doesn't create it.
- SFT first on high-quality demonstrations of the behaviour you want. For robots, collect clean teleoperation data for your exact setup.
- Where good answers are hard to write but easy to compare, collect preferences. Use DPO for simplicity, or a reward model plus PPO if you need an explicit judge.
- Where success is checkable, use a verifier (RLVR / success detectors). Audit what it actually checks.
- Always keep a leash (a KL penalty or trust region) and watch the true metric, not just the optimised one, for over-optimisation.
- Iterate: each stage can expose gaps that need more data for an earlier one.
9. Self-check
Why did GPT-3 answer "explain the Moon landing to a 6-year-old" with more questions?
It was trained only to continue web text, where lists of similar prompts are common. Continuing the list was a high-probability next step. Following the instruction wasn't part of its objective.
In instruction tuning, which tokens carry the loss?
Only the response tokens. The instruction is context to condition on, not something to imitate.
Why compare two answers instead of writing the ideal one?
Judging is much easier than producing, like a food critic who can't cook. Comparisons are cheaper and more consistent to collect than perfect demonstrations.
What happens with the KL leash removed (β → 0)?
The policy collapses onto whatever the reward model scores highest, including its mistakes (reward hacking), and drifts from what pre-training taught it.
Why is RLVR resistant to "sounds good but wrong"?
The verifier checks correctness (the final number, the tests), not style or length. A polished wrong answer gets zero.
What's the robotics version of each teacher?
Demonstration = behaviour cloning from teleoperation. Comparison = humans ranking robot rollouts (Module 06, Day 30). Checker = a success detector or the simulator's ground truth.
10. Glossary & further reading
| Term | Meaning |
|---|---|
| Base model | A model after pre-training only |
| SFT / instruction tuning | Fine-tuning on (instruction, good response), with loss on the response only |
| Preference alignment | Learning from human A-vs-B judgments |
| Reward model | A model trained on preferences to score responses |
| RLHF | RL against a reward model learned from human feedback |
| PPO | Proximal Policy Optimization, a clipped policy-gradient RL algorithm |
| DPO | Direct Preference Optimization: learns from pairs without a separate reward model |
| KL penalty | Cost for drifting from the reference model's distribution: the leash |
| Reward hacking | Exploiting errors in the reward signal instead of doing the task |
| RLVR | RL with verifiable rewards: a program checks correctness |
| Behaviour cloning | Robotics SFT: imitate demonstrated actions |
| Success detector | A robotics verifier: did the task succeed? |
- Dongxi's original article (Chinese). Later parts of the series will cover each technique's data, losses and failure modes.
- Ouyang et al., “Training language models to follow instructions with human feedback” (InstructGPT), 2022
- Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, 2023
- In this course: PPO (06 · Day 29), RL from human preferences (06 · Day 30), Module 03 (human intent and feedback), Module 04 (learning from a teacher), Module 05 (K3 post-training)