From REINFORCE to PPO to GRPO: the RL behind reasoning models

Reasoning models are trained by letting them attempt problems and reinforcing the attempts that work. The algorithms behind that step form a family tree: REINFORCE, then baselines and value functions, actor-critic and GAE, TRPO and PPO, and finally GRPO, which drops the value network and compares each answer with the others in its group. This guide follows the video's chapters, points to the course's in-depth PPO material, and spends its own space on GRPO and the Dr. GRPO correction.

Video by Jia-Bin Huang · Watch on YouTube

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Watch

Chapter guide

The video's description also recommends three longer resources: Pieter Abbeel's Foundations of Deep RL lectures, David Silver's UCL course and Sutton and Barto's book. They're listed in the references.

2. The ideas, explained

Policy gradient (REINFORCE)

Williams (1992) showed how to follow the gradient of expected reward using only samples. Because the environment's transition probabilities don't depend on θ, they vanish from the gradient of the log-probability of a trajectory, and only the policy's own terms remain:

J(θ) = E_{τ~π_θ}[ R(τ) ]
∇J(θ) = E_τ[ Σ_t ∇_θ log π_θ(a_t | s_t) · G_t ]      G_t = Σ_{k≥t} γ^{k−t} r_k  (reward-to-go)
with a baseline:  replace G_t by G_t − b(s_t)          (still unbiased, lower variance)

The estimate is unbiased but very noisy. The rest of the family tree is mostly about reducing that noise and keeping updates safe. The course derives and codes these steps on Module 02's RL ladder.

Value functions, advantages and GAE

The best simple baseline is the state value V(s), the expected return from s. Then G_t − V(s_t) estimates the advantage A(s, a) = Q(s, a) − V(s): how much better this action was than the policy's average. Actor-critic methods learn V with a second network and use it in the update; Mnih et al.'s A3C (2016) is the reference the video cites. Generalized Advantage Estimation (Schulman, Moritz, Levine, Jordan and Abbeel, 2015) blends short and long look-aheads:

δ_t = r_t + γ V(s_{t+1}) − V(s_t)                  (TD residual)
Â_t^GAE(γ,λ) = Σ_{l≥0} (γλ)^l δ_{t+l}
λ = 0: one-step TD (low variance, biased by V)   λ = 1: Monte Carlo return − V (unbiased, noisy)

TRPO and PPO

Policy gradients can take a step that's too large and collapse the policy. TRPO (Schulman et al., 2015) maximises an importance-weighted surrogate subject to a limit on the average KL divergence from the old policy, solved with conjugate gradients and a line search. PPO (Schulman et al., 2017) gets most of the benefit with a first-order trick: clip the probability ratio so there's no incentive to move it outside [1 − ε, 1 + ε].

ρ_t(θ) = π_θ(a_t | s_t) / π_θ_old(a_t | s_t)
TRPO:  max E[ ρ_t Â_t ]   s.t.  E[ KL(π_θ_old ‖ π_θ) ] ≤ δ
PPO:   L^CLIP = E[ min( ρ_t Â_t, clip(ρ_t, 1−ε, 1+ε) Â_t ) ]

In LLM fine-tuning (RLHF), PPO usually runs with four models in memory: the policy, a critic of similar size, a frozen reference model for the KL penalty, and a reward model. The PPO mechanics, ε ablations and a full implementation are on 30u30 Day 29; this page doesn't repeat them.

Group Relative Policy Optimization (GRPO)

GRPO was introduced by Shao et al. in the DeepSeekMath paper (2024) as a PPO variant that improves maths reasoning while reducing PPO's memory use. Its key move is to remove the critic. For each question q, sample a group of G outputs o₁…o_G from the old policy, score them, and use the group's statistics as the baseline:

Â_{i,t} = ( r_i − mean(r_1..r_G) ) / std(r_1..r_G)       (outcome reward: same for every token of o_i)

J_GRPO(θ) = E[ (1/G) Σ_i (1/|o_i|) Σ_t {
              min( ρ_{i,t} Â_{i,t}, clip(ρ_{i,t}, 1−ε, 1+ε) Â_{i,t} )
              − β · D_KL[ π_θ ‖ π_ref ] } ]

D_KL estimate per token:  π_ref/π_θ − log(π_ref/π_θ) − 1      (non-negative, unbiased)

Three differences from PPO as used in RLHF:

• No value network. The group mean plays the role of V(s), which suits LLMs because training a token-level critic for long reasoning traces is hard and doubles memory.
• KL in the loss, not the reward. PPO-for-RLHF usually subtracts a KL penalty from the reward; GRPO adds it directly to the objective, so it doesn't complicate the advantage.
• Group comparison matches how rewards work. Reward models and verifiers are often better at comparing answers to the same question than at giving absolute scores. The paper also describes a process-supervision variant, where step-level rewards are normalised the same way.

DeepSeek-R1 (DeepSeek-AI, 2025) made GRPO famous: its R1-Zero model was trained with large-scale RL using GRPO and rule-based rewards directly on a base model, without supervised fine-tuning first, and long reasoning behaviour emerged. Module 07 covers this RL-with-verifiable-rewards recipe in context (RLVR section).

Dr. GRPO

Liu et al. (2025), in "Understanding R1-Zero-Like Training: A Critical Perspective", identify an optimisation bias in GRPO that artificially increases response length, especially for incorrect outputs, and propose Dr. GRPO ("GRPO Done Right") as an unbiased fix. It removes two normalisations:

• The 1/|o_i| length term. Each answer's token losses are averaged over its own length. For a wrong answer (negative advantage), a longer answer spreads the same penalty over more tokens, so each token is punished less. That quietly rewards long wrong answers. For correct answers it favours short ones. Dr. GRPO sums over tokens with a constant normaliser instead.
• The std term in the advantage. Dividing by the group's std gives prompts whose groups are nearly all right or nearly all wrong (small std) much larger weight, a question-level difficulty bias. Dr. GRPO uses r − mean only. The demo above shows the size of this effect.

Dr. GRPO:  Â_{i,t} = r_i − mean(r_1..r_G)
           J = E[ (1/G) Σ_i Σ_t min( ρ_{i,t} Â_{i,t}, clip(ρ_{i,t}, 1−ε, 1+ε) Â_{i,t} ) ] / C   (C constant)

The paper reports better token efficiency with similar reasoning performance, and a minimal R1-Zero-style recipe reaching 43.3% on AIME 2024 with a 7B base model. With r − mean as the advantage, the group mean is an ordinary REINFORCE baseline again (close to the leave-one-out style), which is a nice way to see that GRPO is still a policy gradient method underneath.

The family tree in one table

3. Where the course goes deeper

Why it matters for robotics

Self-check

References

From the video's description (resources):

• Foundations of Deep RL, 6-lecture series by Pieter Abbeel
• DeepMind x UCL · Introduction to Reinforcement Learning by David Silver
• Reinforcement Learning: An Introduction (Sutton, Barto, 2nd edition)

From the video's description (references):

• Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (REINFORCE) (Williams, 1992)
• Asynchronous Methods for Deep Reinforcement Learning (A3C) (Mnih et al., 2016)
• High-Dimensional Continuous Control Using Generalized Advantage Estimation (Schulman, Moritz, Levine, Jordan, Abbeel, 2015)
• Trust Region Policy Optimization (Schulman, Levine, Moritz, Jordan, Abbeel, 2015)
• Proximal Policy Optimization Algorithms (Schulman, Wolski, Dhariwal, Radford, Klimov, 2017)
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO) (Shao et al., 2024)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI, 2025)
• Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO) (Liu et al., 2025)
• Manim Community, the animation library used to make the video