Turn 'A beats B' straight into a better model — no reward model, no RL. · Source

Teach it what we prefer — no referee, no trial and error.

To bend a model toward what people want, the usual recipe is a Rube-Goldberg machine. Direct Preference Optimization throws the machine out. Hand it pairs — this answer beats that one — and one clean loss turns the preference straight into a better model. No separate judge to train. No reinforcement learning to babysit. Straight from the source.

The usual way takes the long road — and you can game it.

\max_{\pi_\theta}\ \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta}\big[r(x,y)\big]-\beta\,D_{\mathrm{KL}}\big(\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\big)

RLHF takes the scenic route: train a separate judge to score answers, then chase its score with reinforcement learning — all without drifting far from the model you began with. In plain words: win the most reward you can without wandering from home. Two fragile stages — and push too hard and it learns to flatter the judge, not actually improve. Like a switchback road: the long climb, with room to wander off.

That long goal has one exact best answer — already written.

\pi^{*}(y\mid x)=\frac{1}{Z(x)}\,\pi_{\mathrm{ref}}(y\mid x)\,\exp\!\Big(\tfrac{1}{\beta}\,r(x,y)\Big),\quad Z(x)=\sum_{y}\pi_{\mathrm{ref}}(y\mid x)\,\exp\!\Big(\tfrac{1}{\beta}\,r(x,y)\Big)

Here's the secret folded into that goal: the very best policy isn't hunted down by trial and error — it has a closed form. Take the model you started with and gently re-weight it, lifting the answers the reward favors. In plain words: the winner is just the base model, leaned toward reward. Like a trellis on a vine: the plant still grows itself — the frame only leans it toward the light.

Read it backwards — the reward was inside the model all along.

r(x,y)=\beta\,\log\frac{\pi^{*}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\,\log Z(x)

Now flip that equation around and solve for the reward. Out it pops — written entirely in terms of the policy itself. The thing we were going to train a whole separate network to learn was already there, folded inside the model. In plain words: your model is secretly its own reward judge. Like a watermark in paper: invisible flat on the desk — tilt it to the light and the hidden mark is right there.

Drop it into the preference — and the impossible term cancels.

\mathcal{L}_{\mathrm{DPO}}=-\,\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\Big[\log\sigma\!\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\Big)\Big]

That reward still hides one monster: Z, the sum over every answer the model could ever give — impossible to compute. But a preference only ever compares two answers to the same question. Subtract one reward from the other and Z — identical on both sides — simply vanishes. What's left is a single loss, in the policy alone. In plain words: comparison kills the impossible part. Like a balance scale: equal weights on both pans cancel; only the extra pebble tips it.

One tug: lift the winner, drop the loser, hardest where it's wrong.

\hat r_\theta(x,y)=\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)},\quad \nabla_\theta\mathcal{L}_{\mathrm{DPO}}=-\beta\,\mathbb{E}\Big[\sigma\big(\hat r_\theta(x,y_l)-\hat r_\theta(x,y_w)\big)\big(\nabla_\theta\log\pi_\theta(y_w\mid x)-\nabla_\theta\log\pi_\theta(y_l\mid x)\big)\Big]

Strip the loss to its pull and it's a tug-of-war on probability: raise the answer people preferred, lower the one they rejected. And the pull isn't even — the term out front swells exactly when the model has the pair backwards, rating the loser above the winner. In plain words: it corrects itself hardest where it is most confidently wrong. Like a tug-of-war: you heave hardest the instant the rope starts slipping the wrong way.

A pile of 'this beats that' — into a better model, in one step.

So the whole machine collapses into a single line. No judge to train. No reinforcement rollouts to stabilize. Just a fixed pile of this beats that and one gradient step — yet it lands at the exact same goal RLHF was reaching for, reached directly instead of circled. Like a tunnel through the mountain: the same valley waits on the far side — you simply skipped the climb.

🌱 It learned what we chose — never why we chose it.

DPO learns our preferences perfectly — which answer we reached for, every time. But a preference is only the shadow a reason casts. We handed it the choice and kept the reason for ourselves. So when it pleases us, does it grasp why the better answer is better — or has it only learned the shape of our wanting?