Advanced alignment & preference optimisation

Module 07 showed why preference data works and how DPO falls out of the KL-regularised objective. This stage goes past DPO: KTO learns from plain thumbs-up/thumbs-down labels, ORPO folds alignment into SFT, reward models get trained properly, and reward hacking gets measured and fought instead of just noticed.

Source: Suraj Sharma, X post (stage 5) · roadmap hub: Module 08

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. What this stage covers

• Learn: DPO, KTO, ORPO, reward modelling, and how to detect and reduce reward hacking.
• Practice: take an SFT model, align it with KTO using only good/bad labels, and measure how its behaviour distribution shifts.
• Why (the post's claim): SFT teaches content and alignment teaches behaviour, without huge annotation budgets. Section 4 checks that.
• Builds on: Module 07 for the closed-form optimum π* ∝ π_ref·exp(r/β) and its reward-hacking demo. This page doesn't repeat them.

2. Concepts, explained

DPO, in one line, and what it leaves open

Module 07 derives DPO. The loss you implement is:

h(x, y_w, y_l) = log π_θ(y_w|x)/π_ref(y_w|x)  −  log π_θ(y_l|x)/π_ref(y_l|x)
L_DPO = −log σ( β · h )

The quantity β·log π_θ/π_ref acts as an implicit reward. That's the "your language model is secretly a reward model" in the title of Rafailov, Sharma, Mitchell, Ermon, Manning and Finn (2023). Three practical gaps motivate the rest of this stage:

• It needs pairs for the same prompt. Most real feedback is a single thumbs up or down.
• It needs a reference model in memory, or precomputed reference log-probs.
• It is offline. It learns from a fixed dataset, so it never sees its own new mistakes. Xu et al. (2024) argue that a well-tuned PPO pipeline can beat DPO on several benchmarks, so "DPO replaced RL" is too strong. IPO (Azar et al., 2023) is another variant, aimed at DPO's tendency to overfit deterministic preferences.

KTO: learning from good/bad labels

Ethayarajh, Xu, Muennighoff, Jurafsky and Kiela (2024) borrow from Kahneman and Tversky's prospect theory: people judge outcomes relative to a reference point, and losses hurt more than equal gains please. They show that DPO and similar losses already behave like "human-aware losses". KTO then maximises a prospect-theory-style utility of each output directly, so each training example is just (prompt, completion, good or bad).

r(x, y)  = log π_θ(y|x) / π_ref(y|x)           # implicit reward
z0       = estimate of KL(π_θ ‖ π_ref)          # reference point, no gradient
v(x, y)  = λ_D · σ( β·(r − z0) )   if y is desirable
         = λ_U · σ( β·(z0 − r) )   if y is undesirable
L_KTO    = E[ λ_y − v(x, y) ]

• z0 is the reference point. In practice it's estimated from mismatched (prompt, completion) pairs inside the batch, which is why tiny per-device batches cause trouble.
• λ_D, λ_U weight desirable and undesirable examples. They let you handle imbalanced data (say, 10× more thumbs-up than thumbs-down). The paper suggests choosing them so that λ_D·n_D / (λ_U·n_U) sits roughly between 1 and 4/3.
• Saturation: the sigmoid flattens at both ends. An example the model gets badly wrong produces almost no gradient. That makes KTO more tolerant of noisy labels, but it also means it can ignore genuinely hard examples. The demo below shows this.
• Result claimed: KTO matched or beat preference-based methods at 1B to 30B scale, despite only using a binary signal. Treat this as "competitive", not "always better". The authors themselves argue no single loss wins everywhere.

ORPO: alignment inside SFT

Hong, Lee and Thorne (2024) observe that SFT alone raises the probability of the rejected style too, because it's similar text. Their fix adds a small penalty during SFT itself, with no reference model and no second phase:

P(y|x)    = exp( mean token log-prob of y )      # length-normalised
odds(y|x) = P / (1 − P)
L_OR      = −log σ( log odds(y_w|x) − log odds(y_l|x) )
L_ORPO    = L_SFT(y_w)  +  λ · L_OR

Pros: one training run, one model in memory, roughly SFT cost. Cons: there is no explicit KL leash to the starting model, so the SFT term is what keeps the model grounded. You need paired data, and the chosen answers must be good enough to imitate. The paper reports gains for Phi-2, Llama-2 7B and Mistral 7B fine-tuned on UltraFeedback alone.

Reward modelling

A reward model is usually the SFT model with its language-model head swapped for a scalar head. It's trained on comparisons with the Bradley–Terry loss (the same one in Day 30):

L_RM = −log σ( r_φ(x, y_w) − r_φ(x, y_l) )

• InstructGPT (Ouyang et al., 2022) had labellers rank several outputs per prompt and trained on all pairs from each ranking together, so one prompt's comparisons don't overfit as separate samples.
• Only differences in reward matter, so normalise the scores (for example, mean zero on a reference set) before using them in RL.
• Evaluate it like any classifier: held-out pairwise accuracy, broken down by category and by length difference. A reward model that mostly prefers the longer answer will pass a lazy test. Module 01 covers calibration.
• Uses beyond RLHF: best-of-n sampling at inference, ranking synthetic data (Stage 04), and producing labels for DPO or KTO.

Reward hacking and how to fight it

Module 07's demo shows the shape. The measurements come from Gao, Schulman and Hilton (2022). They used a large "gold" reward model as stand-in humans, trained smaller proxy reward models on its labels, and optimised against the proxy. Gold score rose, peaked and fell as the policy moved away from the start (measured as d = √KL). The curve's shape differed between best-of-n and RL, and its coefficients changed smoothly with reward-model size. Mitigations that have evidence behind them:

• A KL penalty, and early stopping on a held-out true metric, not on the proxy reward.
• Bigger, better-trained reward models over-optimise more slowly in Gao et al.'s setup.
• Reward-model ensembles with conservative objectives (worst-case or uncertainty-weighted) reduced over-optimisation in Coste, Anwar, Kirk and Krueger (2023), including with 25% label noise.
• Control for length. Singhal, Goyal, Xu and Durrett (2023) found much of the RLHF reward gain in their settings came from longer outputs, and traced the bias mainly to reward models. Report length alongside every win rate.
• Refresh the data. Sample from the current policy, relabel, retrain the reward model. Hacks the old model rewarded get caught.
• Use verifiers where you can. For maths or code, a checker can't be flattered (Module 07, RLVR), though it can still be gamed if the test is weak.

3. The practice project

(a) The Colab-sized version

The trick is to pick a behaviour a program can label, so you don't need human raters and you can measure the shift exactly.

1. Model: a small instruction-tuned model (around 0.5B parameters, such as a Qwen2.5 or SmolLM2 instruct checkpoint). Train LoRA adapters only.
2. Behaviour: choose something checkable, for example "when asked for JSON, return valid JSON with the requested keys and nothing else", or "answer in at most 80 words".
3. Data: write or collect 300–1,000 prompts. Hold out 20% for evaluation. Sample 4 completions per training prompt at temperature 0.8–1.0 from the SFT model. Label each with your checker. You now have unpaired good/bad data, naturally imbalanced.
4. Train KTO with Hugging Face TRL's KTOTrainer (dataset columns prompt, completion, label). Start at β = 0.1. Set the desirable and undesirable weights from your class counts. Check the current TRL docs, as argument names move between versions.
5. Measure the shift on the held-out prompts, sampling several completions each: checker pass rate before and after, length distribution, distinct-n diversity, and the mean sampled log-ratio log π_θ/π_ref (a KL estimate). Add a capability canary: 100–200 questions from a general benchmark the behaviour shouldn't touch.
6. Ablate: sweep β, drop the undesirable examples, and compare against DPO trained on pairs built from the same samples (one passing and one failing completion per prompt).

from datasets import Dataset
from trl import KTOConfig, KTOTrainer
from peft import LoraConfig

rows = [{"prompt": p, "completion": c, "label": check(p, c)}
        for p in train_prompts for c in sample(model, p, n=4)]
n_good = sum(r["label"] for r in rows)
n_bad = len(rows) - n_good

cfg = KTOConfig(
    output_dir="kto-out", beta=0.1,
    desirable_weight=1.0,
    undesirable_weight=n_good / n_bad,   # keep the ratio near 1–4/3
    per_device_train_batch_size=8,       # z0 needs mismatched pairs in-batch
    learning_rate=5e-6, num_train_epochs=1,
)
trainer = KTOTrainer(model=model, args=cfg,
                     train_dataset=Dataset.from_list(rows),
                     processing_class=tok,
                     peft_config=LoraConfig(r=16, target_modules="all-linear"))
trainer.train()

What to expect: the pass rate should move if your checker is reasonable and the base model sometimes succeeds. If it starts near 0% or near 100%, there's little signal to learn from, so change the task. Don't trust any single run; use at least two seeds.

(b) The full version from the post

Take a 7B SFT model and align it with KTO on a real binary-feedback dataset (product thumbs data, or a pairwise dataset split into separate good and bad rows). Measure the behaviour shift with a held-out LLM judge plus human spot checks, alongside the measures above.

• Hardware: with LoRA, one 80 GB GPU holds the policy and a reference (or you precompute the reference log-probs once). Full fine-tuning of 7B with a reference model needs several GPUs and sharding (Stage 06).
• Cost: a LoRA run over tens of thousands of examples is GPU-hours, not GPU-weeks. At common cloud rental prices that's tens of dollars. The judge calls and human checks often cost more than the training.

(c) Pitfalls

• Grading with the training signal. If an LLM judge labelled the data, evaluate with a different judge and with humans, or you measure the hack, not the improvement.
• Length drift. Report mean length next to every win rate.
• Chat template mismatches between SFT, sampling and training silently break everything. Print a tokenised example.
• Learning rate too high. Preference methods usually want much smaller rates than SFT. Watch the KL estimate; if it runs away, stop.
• Tiny batches with KTO make z0 noisy.
• Chosen and rejected log-probs both falling under DPO is common. Check absolute likelihoods, not just the margin.

4. Reality check

5. In this course

• Module 07 · Preference alignment and the KL leash: the DPO derivation and the reward-hacking demo this page builds on. Also RLVR verifiers.
• Module 06 · Day 30, RL from human preferences: the original reward model from comparisons, with the Bradley–Terry model.
• Module 06 · Day 29, PPO: the RL step in classic RLHF.
• Module 01 · Calibration: a reward model is a classifier, so evaluate it like one.
• Module 04 · Evaluation that doesn't fool you: for measuring the behaviour shift honestly.

6. Why it matters for robotics

7. Self-check

8. Courses & content

Start with the RLHF Book for the concepts, then the TRL docs, because they're what you'll actually run the practice project with.

Full course map for all 12 stages →

9. Papers & docs

• Direct Preference Optimization (Rafailov et al., 2023)
• KTO: Model Alignment as Prospect Theoretic Optimization (Ethayarajh et al., 2024)
• ORPO: Monolithic Preference Optimization without Reference Model (Hong et al., 2024)
• A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO) (Azar et al., 2023)
• Training language models to follow instructions with human feedback (Ouyang et al., 2022)
• Scaling Laws for Reward Model Overoptimization (Gao et al., 2022)
• Reward Model Ensembles Help Mitigate Overoptimization (Coste et al., 2023)
• A Long Way to Go: Investigating Length Correlations in RLHF (Singhal et al., 2023)
• Is DPO Superior to PPO for LLM Alignment? (Xu et al., 2024)
• Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022)
• Hugging Face TRL documentation