Why we teach it to never be perfectly sure. · Source

We train it to leave a sliver of doubt — on purpose.

Show a model an answer key marked 100% right, 0% everything else, and it learns to be a know-it-all — certain of every answer, even the wrong ones. So we do something odd: we blur the key. We tell it the right answer is almost certain, never flat-out certain. The trick is label smoothing, and that pinch of doubt makes the model both humbler and stronger.

The usual answer key is brutally all-or-nothing.

y_i = \begin{cases} 1 & i = c \\ 0 & i \neq c \end{cases} \qquad p_c \to 1 \ \text{only as}\ z_c - z_j \to \infty

The training target is one-hot: a single 1 on the right answer, a flat 0 on every other. Like a target with only a bullseye: the gold counts, and a ring a hair outside scores the same as a wild miss. To make softmax output a true 1, the winning score has to climb forever — an order the model can never finish obeying.

Chasing total certainty never ends.

\frac{\partial \mathcal{L}}{\partial z_c} = p_c - 1 < 0 \quad \text{for all } p_c < 1

Push for that perfect 1 and the math never lets go. The slope on the right answer's score stays negative until it reaches 1 — so the score just keeps rising, its lead over the rivals widening with no ceiling. Like climbing a ladder with no top rung: you're already high enough to see everything, yet 'be certain' keeps you climbing into thin air, for nothing.

The fix: share a crumb of the answer with everyone.

y^{\text{LS}}_i = (1-\alpha)\, y_i + \frac{\alpha}{K}

So soften the key. Keep most of the weight on the right answer, then spread a thin, equal share — α/K apiece — across all K answers. Like cutting a cake: the guest of honor gets the big slice, but everyone at the table still gets a sliver, so no answer sits at flat zero. The dial α sets how much you give away.

Now the loss pulls two ways at once.

\mathcal{L}_{\text{LS}} = (1-\alpha)\,(-\log p_c) \;+\; \alpha\left(-\frac{1}{K}\sum_{i} \log p_i\right)

The smoothed target splits the loss into two pulls. One still drags the model toward the right answer; the other gently tugs it toward even — a flat, know-nothing guess. Like a kite on a string: the wind hauls it up and out, your hand holds a steady tension back, and only both together keep it flying without tearing away.

Certainty finally has a resting place.

\Delta^{\star} = \ln\!\frac{(1-\alpha)K + \alpha}{\alpha} \quad (\text{finite, not } \infty)

Now the best the model can do isn't infinity — it's a fixed gap. The smoothed target is happiest when the right answer's confidence settles at one finite height, comfortably below 1. Like a ball rolling to rest in a bowl: the old target was an endless downhill with no bottom; this one finally gives it a low point to settle into. For K=1000 answers and α=0.1, that best gap is about 9.1 — and never more.

But the gift of doubt costs some nuance.

Here's the catch. That equal sliver treats every wrong answer the same — cat mistaken for dog scores no closer than cat mistaken for truck. Like fresh snow over rough ground: it calms the harsh, jagged certainty into something even and gentle, but it also buries the small ridges and ruts that told one wrong answer from another. You gain calibration; you trade away a little of that fine structure.

🌱 We forbade certainty. Is that wisdom, or manners?

We never showed it which things to doubt. We just rewrote the answer key so it can never be wholly sure of anything — a sliver always held back by hand, like a moon kept from ever going full. So when it hedges, is that the dawn of real humility — a mind feeling the edge of what it knows — or only a courtesy we stitched in from outside?