Clip, Compare, or Skip: How PPO, GRPO, and DPO Align Language Models

Why your chatbot loves saying “Great question!”

Pretraining gives a language model broad knowledge, but not preferences. It doesn’t know which of two plausible answers a person would actually want. Alignment closes that gap: you collect human preference data, then use it to push the model toward better responses without letting it drift into nonsense or exploit blind spots in how “better” is measured. That last failure mode is reward hacking, which I covered in Gaming the System.

This post walks through the three algorithms you’ll most often see for this step: PPO, GRPO, and DPO. For each one, I’ll cover the problem it solves, how it works, what it costs, and how it relates to reward hacking. There’s a side-by-side comparison at the end.

To keep things concrete, I’ll use one toy prompt throughout: “What is 17 × 24?” The right answer is 408.

Notation:

\(x\): prompt
\(y\): response
\(\pi_\theta\): the policy (the model being trained)
\(\pi_{\text{ref}}\): frozen reference model, usually the SFT model
\(r_\phi(x, y)\): learned reward model
\(\beta\): strength of the KL penalty

1. Proximal Policy Optimization (PPO): small steps toward the reward

The problem. You’ve trained a reward model \(r_\phi(x, y)\) on human preferences, and now you want the language model to produce answers it scores highly. Taking big gradient steps toward higher reward is unstable, and it quickly finds outputs the reward model happens to love but humans wouldn’t.

The idea. Sample answers from the current model, score them, and nudge the model toward the answers that did better than expected, but only a little per update. A KL penalty acts as a leash back to the reference model.

Example. The model answers “17 × 24 = 340 + 68 = 408.” The reward model scores it 0.9. A separate value model, which predicts how well answers to a prompt like this usually score, expected about 0.6. So the answer did better than expected (an advantage of roughly +0.3), and its tokens get pushed up. With a typical clip of \(\epsilon = 0.2\), once an update has made those tokens about 20% more likely than before, PPO stops pushing for this batch.

Animation of one PPO update: the policy samples an answer, the reward model scores it 0.9, the value model expected 0.6 so the advantage is +0.3, and the token probability ratio rises from 1.0 until it hits the clip at 1.2.

The math. The overall objective is to maximize reward while staying close to the reference model:

\[J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)}\big[r_\phi(x, y)\big] \;-\; \beta\, \mathrm{KL}\big(\pi_\theta(\cdot \mid x)\,\|\,\pi_{\text{ref}}(\cdot \mid x)\big)\]

PPO optimizes this one token at a time. The state is the prompt plus everything generated so far, \(s_t = (x, y_{<t})\), and the action is the next token \(y_t\). For each token, PPO compares the new policy to the one that generated the data:

\[\rho_t(\theta) = \frac{\pi_\theta(y_t \mid s_t)}{\pi_{\theta_{\text{old}}}(y_t \mid s_t)}\]

and maximizes the clipped objective:

\[L^{\text{CLIP}}(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t(\theta)\,\hat{A}_t,\; \mathrm{clip}(\rho_t(\theta),\, 1-\epsilon,\, 1+\epsilon)\,\hat{A}_t\big)\Big]\]

In words: push up tokens with positive advantage and push down tokens with negative advantage, but once \(\rho_t\) leaves \([1-\epsilon, 1+\epsilon]\), the gradient for that token goes to zero. This keeps each update close to the policy that generated the data, approximating a trust region without the second-order machinery of TRPO.

For the curious: where the advantage comes from

In InstructGPT-style setups, the KL penalty is folded into a per-token reward, with the reward model’s score added at the final token \(T\):

\[r_t = -\beta \log \frac{\pi_\theta(y_t \mid s_t)}{\pi_{\text{ref}}(y_t \mid s_t)} \;+\; \mathbb{1}[t = T]\, r_\phi(x, y)\]

A learned value model \(V_\psi(s_t)\) estimates how good each partial response is, and the advantage is typically computed with Generalized Advantage Estimation (GAE):

\[\delta_t = r_t + \gamma V_\psi(s_{t+1}) - V_\psi(s_t), \qquad \hat{A}_t = \sum_{l=0}^{T-t} (\gamma\lambda)^l\, \delta_{t+l}\]

The advantage answers one question: was this token better or worse than the value model expected at this point?

Where reward hacking hides. PPO has two brakes, and they do different jobs. The clip limits how fast the policy moves; the KL penalty limits how far. Clipping alone can’t stop reward hacking, because a policy can drift toward an exploit one small, clipped step at a time. Suppose opening with “Great question!” happens to raise the reward model’s score. PPO will learn to say it, just slowly. The KL penalty pushes back against the total drift from \(\pi_{\text{ref}}\), which makes gibberish-but-high-reward outputs expensive. Neither brake fixes the reward model itself: if \(r_\phi\) has a blind spot, PPO will still find it.

Animation comparing PPO with and without a KL penalty: with clipping alone, the policy takes small steps that eventually reach a reward hack; with a KL penalty, the policy is pulled back and settles where extra reward no longer outweighs the penalty. — *Illustrative numbers. The clip limits how fast the policy moves; the KL penalty limits how far.*

The cost. PPO keeps four models in play: the policy, the reference model, the reward model, and the value model. The value model is as large as the policy and needs its own training. That overhead is exactly what GRPO was designed to cut.

2. Group Relative Policy Optimization (GRPO): grade answers against each other

The problem. PPO needs a value model to judge whether each answer did better or worse than expected. That’s a whole extra model to train and keep in memory, and a poorly trained value model gives noisy advantages.

The idea. Skip the value model. Ask the same question several times, and judge each answer against its siblings. Answers that beat the group average get pushed up; answers that fall below it get pushed down. GRPO was introduced in DeepSeekMath and popularized by DeepSeek-R1.

Example. The model samples 4 answers to “What is 17 × 24?” Three say 408 (correct, reward 1) and one says 418 (wrong, reward 0). The group’s average reward is 0.75 and its standard deviation is about 0.43.

Each correct answer is slightly above average. Its advantage is (1 − 0.75) / 0.43 ≈ +0.58, so it gets a gentle push up.
The wrong answer is well below average. Its advantage is (0 − 0.75) / 0.43 ≈ −1.73, so it gets a strong push down.

Animation of one GRPO update: four answers are sampled and scored 1, 0, 1, 1; with a group mean of 0.75 and standard deviation of 0.43, the correct answers get an advantage of +0.58 and are pushed up, while the wrong answer gets −1.73 and is pushed down.

If all four answers had been correct, every advantage would be zero, and the model would learn nothing from that prompt.

Animation showing that when all four answers get the same reward, whether all correct or all wrong, the standard deviation is zero, every advantage is zero, and the prompt produces no learning signal.

The math. For each prompt \(x\), sample a group of \(G\) responses \(\{o_1, \dots, o_G\}\) from \(\pi_{\theta_{\text{old}}}\) and score them to get rewards \(r_1, \dots, r_G\). Each response’s advantage is its reward normalized within the group:

\[\hat{A}_i = \frac{r_i - \mathrm{mean}(r_1, \dots, r_G)}{\mathrm{std}(r_1, \dots, r_G)}\]

Every token in \(o_i\) gets the same advantage \(\hat{A}_i\). The policy is then updated with PPO’s clipped objective, averaged over the group, with the KL penalty applied directly in the loss.

For the curious: the full GRPO objective

\[J_{\text{GRPO}}(\theta) = \mathbb{E}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\Big(\min\big(\rho_{i,t}\,\hat{A}_i,\; \mathrm{clip}(\rho_{i,t},\, 1-\epsilon,\, 1+\epsilon)\,\hat{A}_i\big) - \beta\, \mathbb{D}_{\text{KL}}\big[\pi_\theta \,\|\, \pi_{\text{ref}}\big]\Big)\Bigg]\]

where

\[\rho_{i,t} = \frac{\pi_\theta(o_{i,t} \mid x,\, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid x,\, o_{i,<t})}\]

and the KL term is estimated per token as

\[\mathbb{D}_{\text{KL}} = \frac{\pi_{\text{ref}}(o_{i,t} \mid \cdot)}{\pi_\theta(o_{i,t} \mid \cdot)} - \log\frac{\pi_{\text{ref}}(o_{i,t} \mid \cdot)}{\pi_\theta(o_{i,t} \mid \cdot)} - 1\]

In practice, a small constant is added to the standard deviation so the advantage stays defined when every reward in a group is the same.

Where reward hacking hides. GRPO doesn’t care where the reward comes from. DeepSeekMath experimented with both outcome rewards (one score per response) and process rewards (scores for each reasoning step). DeepSeek-R1-Zero used simple rule-based rewards for answer accuracy and output format, partly because learned reward models are vulnerable to hacking at scale. So GRPO is exactly as hackable as its reward. With verifiable rewards (does the code pass the tests? is the answer 408?), there’s far less to exploit than with a learned reward model, though format rewards can still be gamed.

The cost. GRPO drops the value model, leaving the policy, the reference model, and a reward model or rule-based scorer. In exchange, it generates \(G\) responses for every prompt, so it spends more compute on sampling.

3. Direct Preference Optimization (DPO): skip reinforcement learning entirely

The problem. PPO and GRPO both need a reward signal and an online loop that samples from the model during training. If what you have is a dataset of preference pairs, do you need either?

The idea. No. Direct Preference Optimization (Rafailov et al., 2023) shows that the KL-regularized objective PPO optimizes can be rewritten as a simple classification-style loss on preference pairs. You train directly on “this answer was preferred over that one,” with no reward model and no sampling. On sentiment control, summarization, and single-turn dialogue, the paper shows DPO matching or beating PPO-based RLHF.

Example. A preference pair for our prompt:

Preferred (\(y_w\)): “17 × 24 = 340 + 68 = 408.”
Rejected (\(y_l\)): “418.”

At the start of training, the policy equals the reference model, so the model has no preference between the two. The loss is \(\log 2 \approx 0.69\). As training makes the preferred answer more likely and the rejected one less likely, both relative to the reference model, the loss falls. With \(\beta = 0.1\), if the preferred answer’s log-probability rises by 2 and the rejected one’s falls by 2, the loss drops to about 0.51. The parameter \(\beta\) controls how hard the model is willing to pull away from the reference model to separate the two.

Animation of one DPO update: a preferred answer and a rejected answer start with log-probability ratios of 0 against the reference model, giving a loss of 0.69; training raises the preferred answer to +2 and lowers the rejected one to −2, and the loss falls to 0.51.

The math.

\[\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x,\, y_w,\, y_l) \sim \mathcal{D}}\Big[\log \sigma\Big(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\Big)\Big]\]

where \(\sigma(z) = 1 / (1 + e^{-z})\). Two things are worth noticing:

This is the same \(\beta\) as PPO’s KL penalty. DPO doesn’t drop the KL constraint; it builds it into the loss.
The quantity \(\beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\) acts as an implicit reward. That’s why the paper’s subtitle is “Your Language Model Is Secretly a Reward Model.”

For the curious: how the loss is derived

DPO starts from the same KL-regularized objective PPO optimizes. That objective has a closed-form optimal policy:

\[\pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\, \exp\!\Big(\frac{1}{\beta}\, r(x, y)\Big)\]

Rearranging expresses the reward in terms of the policy:

\[r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)\]

The Bradley–Terry model says the probability that \(y_w\) is preferred over \(y_l\) is \(\sigma\big(r(x, y_w) - r(x, y_l)\big)\). Substituting the expression above, the intractable \(Z(x)\) cancels, and maximizing the likelihood of the observed preferences gives the DPO loss.

Where reward hacking hides. Because DPO trains offline on a fixed dataset, there’s no reward model for it to exploit during training. But it absorbs every bias in the preference data. Length is the classic one: if annotators tended to prefer longer answers, DPO learns that longer is better. And it can still over-optimize. Rafailov et al. (2024) found that direct alignment methods show the same over-optimization pattern as classic RLHF, with quality degrading as the model moves further from the reference, often before a single epoch is complete.

Animation showing a dataset where the preferred answer is always longer, followed by a model that pads a simple answer to 17 × 24 with extra sentences as its implicit reward rises. — *Illustrative. DPO can't tell length from quality if the data never separates them.*

The cost. DPO is the cheapest of the three: the policy and the reference model, and the reference model’s log-probabilities can be precomputed. There’s no sampling loop and one main hyperparameter, \(\beta\).

Side by side

Diagram comparing what PPO, GRPO, and DPO need during training: PPO uses a policy, reference model, reward model, value model, and online samples; GRPO drops the value model; DPO also drops the reward model and trains on fixed preference pairs. — *Each method removes a piece of the one before it.*

	PPO	GRPO	DPO
Models during training	Policy, reference, reward, value	Policy, reference, reward (or a rule-based scorer)	Policy, reference (reference log-probs can be precomputed)
Online sampling?	Yes	Yes, \(G\) responses per prompt	No, uses fixed preference pairs
Baseline	Learned value model + GAE	Group mean and standard deviation	None (implicit reward)
Main knobs	\(\epsilon\), \(\beta\), GAE \(\lambda\), value model	\(G\), \(\epsilon\), \(\beta\)	\(\beta\)
Reward hacking	Clip slows it, KL bounds it; reward-model blind spots remain	As hackable as the reward; verifiable rewards help a lot	No online exploitation, but inherits dataset biases and can still over-optimize
Reach for it when	You have a learned reward model and the compute for it	Rewards are verifiable (math, code) or memory is tight	You have good preference pairs and want simplicity

Choosing between them

In practice, I’ve found the choice of algorithm matters less than the signal you feed it: a clean, verifiable reward or a well-curated preference set will beat a clever optimizer on noisy data. If I could only get one thing right, it would be the signal, not the method.

When you do choose, it’s less about which one is “best” and more about where you want the risk to sit. PPO gives you the most control and the most moving parts. GRPO trades the critic for more samples, and it shines when the reward is something you can verify. DPO removes the RL loop entirely, moving the risk out of training and into your data. None of them makes reward hacking go away; they decide where you have to watch for it.

For what reward hacking looks like in practice, see Gaming the System. For the broader pattern of optimizing a proxy until it stops meaning anything, see Winning the Metric, Losing the Point.

1. Proximal Policy Optimization (PPO): small steps toward the reward

2. Group Relative Policy Optimization (GRPO): grade answers against each other

3. Direct Preference Optimization (DPO): skip reinforcement learning entirely

Side by side

Choosing between them

Further reading