Why a model makes things up — and sounds so sure doing it. · Source

It can describe something that was never there — and sound certain.

Ask a model a question and the answer comes out fluent, detailed, sure of itself — even when it's pure invention. Like a desert mirage: the shimmer of water on the hot road looks utterly real, sharper than the asphalt around it. You'd swear it's there. There's no water. This confident not-there has a name: hallucination.

It was never trained to be right. Only to sound right.

\hat{\theta} = \arg\max_{\theta} \sum_{t} \log p_{\theta}(x_t \mid x_{<t})

Training tunes one thing: make the word that actually came next as likely as possible. True never enters that sum — only likely. Like a parrot: it can reproduce the sound of a sentence flawlessly while understanding none of it. Fluency is the whole skill. Truth was never the target.

Ask it anything and an answer drops out. There is no empty slot.

Whatever you feed it, the next-token engine returns its best-looking continuation — by default, a complete, confident answer. Like a gumball machine: turn the crank and a candy always drops. The chute is never empty. 'I don't know' isn't what it reaches for first — it reaches for whatever sounds like the answer.

Scored only on being right, a guess always beats staying silent.

\mathbb{E}[\text{guess}] = p \cdot 1 + (1-p) \cdot 0 = p \;>\; 0 = \mathbb{E}[\text{abstain}]

Grade it like a quiz — +1 right, 0 wrong, 0 for blank — and a guess with any chance p of landing earns p; silence earns nothing. So the smart move is always to answer. Like a ring toss: holding the ring back scores zero, so you throw every time. We graded it to bluff.

Between the facts it really learned, it fills in the plausible.

Ask for something it only half-knows and it interpolates — patching the gap with whatever fits the shape. A clean fake reference, a believable wrong date: seamless, and invented. Like mending a stone wall: when the original piece is gone, you wedge in whatever stone fills the hole. The wall stands. That stone was never part of it.

Sure isn't the same as right. They're two different axes.

P(\text{correct} \mid \hat{p} = c) = c

A model can be confident and consistent and still land far from the truth — most of all on rare, unfamiliar facts, where it's surest exactly where it should hesitate. Being calibrated would mean its 70%-sure answers are right 70% of the time. Reality drifts off that line. Like a tight cluster of bowls — yards from the jack: beautifully consistent, completely off.

The cure isn't 'stop guessing.' It's changing what we reward.

\mathbb{E}[\text{answer}] = p - \lambda(1-p) > 0 \iff p > \dfrac{\lambda}{1+\lambda}

Hallucination isn't a glitch bolted on — it's fluency seen from its blind side, the same smooth guessing that lets it generalize at all. The fix is the incentive: make a confident error cost λ, and answering only pays once confidence p clears the bar. Like knowing when to fold: bet the strong hand, fold the weak one — stop bluffing every deal.

🌱 We never paid it to pause. Can we fault it for never pausing?

We taught it that a wrong guess costs nothing and 'I don't know' earns nothing — so it always answers. If the honest pause were the rewarded move instead of the penalized one, would the same machine learn to say I'm not sure? Or is the confident voice simply the only one we ever asked it for?