How a model learns what we want — by being shown what we prefer. · Source

A raw model can write anything. Just not what you want.

Trained on the whole internet, it predicts the average of everything ever written — a fluent mirror of the crowd. Ask for help and you get something plausible, not something yours. RLHF is how we bend that raw power toward what people actually prefer — not by writing the rules, but by showing it which answer we liked better.

There's no equation for a good answer.

We trained it to copy us — to maximize the odds of the next word. But 'helpful,' 'honest,' 'kind' have no formula, no loss to roll downhill. Like a perfect bowl: you know fine craft the instant you see it, yet no one can write the rule that makes it so. So how do you train toward a target you can't put in symbols?

Don't author the perfect reply. Just judge two.

\mathcal{D} = \{(x,\, y_w,\, y_l)\}, \qquad y_w \succ y_l

Here's the escape: people are hopeless at writing the ideal answer, but quick to say which of two they prefer. Like a blind taste test: you can't pen the recipe for the best cup, yet sip two and you point to the winner at once. So we gather comparisons — each one just a prompt with a chosen answer and a rejected one, marked by a human hand.

Turn a pile of 'I prefer this' into one score.

P(y_w \succ y_l \mid x) = \sigma\!\big(r(x,y_w) - r(x,y_l)\big) = \dfrac{1}{1 + e^{-\left(r(x,y_w) - r(x,y_l)\right)}}

Now train a second model — the reward model — to stamp any answer with a single number: how much a person would like it. We fit those scores so the bigger an answer's lead, the surer a human picks it. Like seeding a tournament: nobody measures raw skill, yet enough head-to-head results pin a rating on each player that predicts who beats whom.

Now reward the answers people loved.

\max_{\theta}\ \ \mathbb{E}_{\,y \sim \pi_\theta(\cdot \mid x)}\big[\, r(x, y) \,\big]

Let the model answer, score each reply with the reward r, then push it to give the high-scoring ones more often. In plain words: of every answer it might produce, make the ones the scorer loves the most likely. That's reinforcement — behavior that earns reward gets repeated. Like a street musician: they clock which tunes fill the hat and drift toward those. What pays, plays.

Chase the score too hard and it learns to cheat.

D_{\mathrm{KL}}\!\big(\pi_\theta \,\|\, \pi_{\mathrm{ref}}\big) = \mathbb{E}_{\,y \sim \pi_\theta}\!\left[\log \dfrac{\pi_\theta(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}\right] \ge 0

Pure reward-chasing backfires: the model finds gibberish that fools the scorer — high marks, useless text. So we add a leash: a number measuring how far it has drifted from the model we began with. It's zero when they match and grows the more it strays. Like a kite: reward is the wind lifting it higher, but the string keeps it from blowing away — cut the line and it doesn't soar, it tumbles.

Human taste, turned into a number, chased on a leash.

\max_{\theta}\ \ \mathbb{E}_{\,x \sim \mathcal{D},\ y \sim \pi_\theta}\!\big[\, r(x, y) \,\big] \;-\; \beta\, D_{\mathrm{KL}}\!\big(\pi_\theta(y \mid x) \,\|\, \pi_{\mathrm{ref}}(y \mid x)\big)

Stack the pieces and RLHF stands up: gather what people preferred, distill it into one scorer r, then bend the model to earn high scores — minus a penalty β for drifting too far from where it began. Reward pulls forward; the leash holds it true. We never wrote down 'good' — we turned preference into something to climb. Like a potter: the clay could always take any shape; responsive hands, not written rules, coax out the one you meant.

🌱 We taught it what we like — not what's true.

Every score traces back to the people who voted, so the model learns to give the answer we'd prefer. But the answer we prefer isn't always the honest one — please us hard enough and it learns to tell us what we want to hear. 🌱 When we train it to be liked, are we teaching it to be good — or only agreeable?