LLM + RoboticsUnit 6 · Deep RL & alignment · Day 29

Proximal Policy Optimization Algorithms

Reuse each batch of experience for several gradient steps, but clip the objective so the new policy can't drift far from the one that collected the data.

AuthorsJohn Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov
Year / venue2017 · arXiv preprint (OpenAI)
Core ideaClipped probability ratio
Study time4–6 h

Original: arXiv:1707.06347 · Code, notebook and exercises: 30u30 Day 29

Start here (no maths needed)

PPO teaches a computer by trial and error, but only lets it change its habits a little at a time, so it doesn't undo what it already learned.

Think of it like this

Think of training a dog with treats. When the dog sits and gets a treat, it starts sitting more often. A good trainer doesn't expect the dog to change everything after one treat. They reward small improvements, again and again. If you got too excited after one good moment and changed your whole training plan, you might confuse the dog. PPO is the careful trainer: learn from what just happened, but only take small steps.

How it works

  1. The program (the "policy") tries the task many times and records what happened.
  2. For each action, it works out: was this better or worse than expected?
  3. It makes actions that went well a bit more likely, and ones that went badly a bit less likely.
  4. It sets a limit on how much any action's chance can change in one round. This is the "clip".
  5. Throw away the old records, try the task again with the improved policy, and repeat.

Why robots care

Many walking robots, like four-legged robot dogs, learn to walk this way. Thousands of copies practise at the same time in a computer simulation. They fall over, get up, and slowly improve. Then the learned skill is moved to the real robot.

Try it

Words to know

Reinforcement learning
Learning by trying things and getting rewards or penalties.
Policy
The robot's "rulebook" for what to do in each situation.
Advantage
How much better (or worse) an action turned out than expected.
Clip
A limit on how big a change is allowed in one step.
Simulation
A computer version of the world where robots can practise safely.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

Plain policy gradient uses each sample once. Optimising on the same data for many steps tends to make destructively large policy updates. TRPO fixed this with a hard KL constraint, but it needs second-order optimisation and is awkward to combine with shared policy/value networks. PPO gets most of TRPO's stability with ordinary first-order SGD. It takes the ratio between the new and old action probabilities and clips it to [1 − ε, 1 + ε], then takes the more pessimistic of the clipped and unclipped objectives. The update then has no incentive to push the ratio past the clip. Several epochs of minibatch Adam on each batch become safe. It's a few lines of change to a policy-gradient implementation, and it became the default on-policy RL algorithm.

2. The core idea

r_t(θ)   = π_θ(a_t | s_t) / π_θold(a_t | s_t)

L^CLIP(θ) = Ê_t[ min( r_t(θ)·Â_t ,  clip(r_t(θ), 1−ε, 1+ε)·Â_t ) ]

L(θ) = Ê_t[ L^CLIP_t(θ) − c1·(V_θ(s_t) − V_t^targ)² + c2·S[π_θ](s_t) ]

Â_t = δ_t + (γλ)·δ_{t+1} + … + (γλ)^(T−t−1)·δ_{T−1}
δ_t = r_t + γ·V(s_{t+1}) − V(s_t)

MuJoCo settings from the paper: horizon T = 2048, Adam step size 3 × 10−4, 10 epochs, minibatch 64, γ = 0.99, λ = 0.95.

3. Deep dive

What the clip does, case by case

Look at one timestep. If the advantage is positive, the update wants to raise the action's probability, so r grows. Once r passes 1 + ε, the objective goes flat and the gradient for that sample is zero. If the advantage is negative, the update lowers the probability. Once r drops below 1 − ε, it's flat again. But if r has moved the wrong way (say the ratio went up on a bad action), the unclipped term is smaller and the min keeps it, so the gradient still pulls it back. Clipping only removes the incentive to overshoot. It's not a hard wall.

Interactive: the clipped objective for one sample

Left: advantage  = +1. Right:  = −1. Dashed: unclipped r·Â. Solid: LCLIP. The marker shows the ratio r you pick. Flat stretches mean zero gradient.

L, Â = +1
–
Gradient, Â = +1
–
L, Â = −1
–
Gradient, Â = −1
–

Try this: at r = 1.3 and ε = 0.2, the positive-advantage sample has zero gradient: the policy already moved enough. The negative-advantage sample still has full gradient, pulling r back down. Now drag r below 0.8.

Where PPO sits on the RL ladder

Module 02's RL ladder climbs Bellman → REINFORCE → A2C → GAE → PPO. Each step fixes the previous one's main weakness. REINFORCE is unbiased but noisy. A baseline and a critic (A2C) cut variance. GAE tunes the bias-variance trade-off. PPO's contribution is sample reuse: K epochs on each batch instead of one step. TRPO had the same goal with a KL constraint solved by conjugate gradient. PPO swaps that for a clip and plain Adam, which also makes it easy to share layers between policy and critic or add auxiliary losses.

It's still on-policy. Once θ has moved, old data no longer represents the current policy, so each batch is discarded after its K epochs. That's why PPO pairs naturally with fast, massively parallel simulators. Off-policy methods (SAC, DQN) reuse data longer and are usually more sample-efficient on real hardware.

From MuJoCo to language models

PPO later became the RL step in RLHF for language models: InstructGPT optimised a learned reward model with PPO plus a KL penalty to the starting model. That's the direct line to Day 30, whose 2017 experiments actually used TRPO and A2C. Newer LLM recipes often replace PPO's critic with group-relative baselines (GRPO) or skip RL for DPO. The clipped-ratio idea survives in most of them. Compare the multi-teacher on-policy distillation in Module 05, another way to train a model on its own samples.

4. Why it matters for robotics

5. How to study it

  1. Read the paper (it's 12 pages). Focus on sections 3 (clipping), 5 (algorithm) and 6.1 (the ablation table). About 1.5 h.
  2. Re-derive the four cases of the clip (positive or negative advantage, ratio above or below the clip) with the demo above.
  3. In the 30u30 Day 29 folder, read implementation.py: actor-critic network, rollout buffer with GAE, the PPO loss and the training loop. Run train_minimal.py on a small gym task.
  4. Work through notebook.ipynb, then the exercises in order: clipped loss, GAE, actor-critic, ε ablation, PPO versus vanilla policy gradient.
  5. Stretch: train a MuJoCo or Isaac locomotion task with an existing PPO implementation and watch how the clip fraction and KL behave during training.

6. Self-check

Why not just run many gradient steps on the plain policy-gradient loss?

The loss is only valid near the policy that collected the data. Repeated steps on it push the policy far away, often making destructively large updates. The paper's no-clipping baseline scored below a random policy on average.

With  > 0 and r = 1.5, ε = 0.2, what's the gradient of LCLIP for that sample?

Zero. The clipped term 1.2·Â is smaller than 1.5·Â, so the min selects the clipped term, which doesn't depend on θ.

With  < 0 and r = 1.5, is the gradient zero?

No. Because  is negative, r·Â = 1.5 is lower than the clipped 1.2Â, so min picks the unclipped term and the gradient pushes r back down. Clipping only blocks overshooting in the helpful direction.

What does λ in GAE control?

The bias-variance trade-off of the advantage estimate. λ = 0 uses only the one-step TD error (low variance, more bias). λ = 1 uses the full return (unbiased given the baseline, high variance).

Why is PPO a good match for GPU physics simulators?

It's on-policy and stable, so it benefits from huge, fresh batches of parallel experience and doesn't need a replay buffer. Simulators can supply that data cheaply.

7. Further reading