How a row of scores becomes a confident choice. · Source

The last step: turn a row of scores into one honest answer.

Every model's final layer puts out a raw score for each option — bigger means 'more likely,' but that's all it means. Before it can answer, it has to turn that messy row into a real probability: which one, and exactly how sure. That single move is the softmax.

Raw scores aren't probabilities. Two rules say so.

p_i \ge 0 \qquad \sum_i p_i = 1

Like a harvest with no scale: each score has real heft, but there's no shared measure — nothing says what fraction of the whole any of them is. A probability must obey two rules: never negative, and all of them add to one. Raw scores honor neither. We need one move that fixes both.

First move: raise <em>e</em> to every score.

First move: raise e to every score.

\dfrac{e^{z_i}}{e^{z_j}} = e^{\,z_i - z_j}

Exponentiating does two jobs at once. It makes every score positive — even the negatives — and it's greedy: it stretches the gaps. Like a snowball down a slope, a small lead turns into a runaway one. The ratio between any two depends only on the gap between them — and every extra point multiplies a score's weight by about 2.7.

Second move: divide each by the total.

\operatorname{softmax}(\mathbf{z})_i = \dfrac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}

Now add up all the exponentials and divide each one by that sum. Suddenly they're positive and they add to exactly one — a real probability distribution. Like one pie carved by appetite: the whole is fixed, and the biggest score simply claims the biggest slice. Exponentiate, then normalize — that's the softmax.

Why <em>soft</em>? Because it never silences the losers.

Why soft? Because it never silences the losers.

\dfrac{\partial p_i}{\partial z_j} = p_i\,(\delta_{ij} - p_j)

A hard 'pick the biggest' gives the winner everything and the rest nothing — and it leaves no slope to learn from. Softmax hands the winner the largest share yet still lets the runners-up speak, and it bends smoothly everywhere. Like a photo finish: second by a nose is still on the record. That smooth slope (here p is its own output) is why blame can flow back through it while the model trains.

It reads the gaps, not the heights.

\operatorname{softmax}(\mathbf{z} + c\mathbf{1})_i = \operatorname{softmax}(\mathbf{z})_i

Add the same number to every score and the probabilities don't move at all. Softmax only cares about the differences between scores, never their absolute size. Like peaks above the clouds: what counts is how far one rises above the rest, not its height over the distant sea. It's also the quiet trick that stops the math from overflowing — just subtract the largest score first.

Two moves: make it positive, make it sum to one.

That's the whole engine. Exponentiate to make every score positive and widen the lead; normalize to turn the row into shares of one. Like a string of bulbs on one circuit: every option draws a little current, the favorite burns brightest, and together they light up exactly one whole. It's the model's final step — turning a private hunch into a public, comparable probability.

🌱 It can always answer. It can never say 'I don't know.'

Softmax always returns a full distribution — every option gets a slice, and the slices always sum to one. Even total ignorance comes out as a tidy, confident spread. There's no way for it to hand you an empty answer, no slot for 'no idea.' 🌱 When a system is built so it must always answer, is its confidence the same as knowing — or just the shape we forced onto it?