The one dial between safe and surprising. · Source

The model doesn't pick the next word — it rolls loaded dice.

Out of softmax comes a full table of odds for the next word. The model doesn't just grab the top one — it rolls, drawing a word in proportion to its probability. Temperature is the single knob that sets how loaded those dice are: turn it down and it plays safe, turn it up and it gets adventurous.

Always safe is robotic. Always random is nonsense.

Always take the likeliest word and the model turns stiff — the same reply every time, sentences trapped in loops. Sample straight from the raw odds and it rambles, now and then blurting a word from nowhere. Like a stuck assembly line: one setting stamps out identical parts forever. We want a knob between dead and unhinged.

The dial: divide every score by <em>T</em> before softmax.

The dial: divide every score by T before softmax.

p_i = \dfrac{e^{z_i/T}}{\sum_j e^{z_j/T}}

Before the scores turn into odds, divide each one by a number T. Small T (below 1) stretches the gaps — the favorite runs away with it. Large T flattens them — underdogs get a real shot. At T = 1, nothing changes. Like a pot on the stove: a low simmer barely stirs and the heaviest bits settle; a rolling boil sends everything churning at once.

Turn it to zero and it freezes; to infinity, pure chaos.

\lim_{T\to 0^{+}} p_i = \mathbb{1}\!\left[\,i=\operatorname*{argmax}_j z_j\,\right] \qquad \lim_{T\to\infty} p_i = \frac{1}{n}

Push the dial to its stops. As T→0, the top score swallows all the odds — always the single likeliest word, the same fixed answer every time (this is what 'temperature 0' means). As T→∞, every word flattens to the same odds — a blind coin-flip across the whole vocabulary. Like a marble in a bowl: left alone it always rolls to the one lowest point; shake hard enough and it could end up anywhere.

Even tamed, a long tail of bad words still lurks.

Pick a sane temperature and a problem remains: a vocabulary holds tens of thousands of words, and the thousands of near-impossible ones each keep a crumb of probability. Crumbs add up. Sample long enough and you'll eventually draw one that derails the whole sentence. Like a cloud of midges: no single one matters, but the swarm is so vast that sooner or later something bites.

So cut the tail: keep the few that cover the mass.

S=\operatorname*{arg\,min}_{S'}\Big\{\,|S'| : \textstyle\sum_{i\in S'} p_i \ge p\,\Big\} \qquad \hat p_i=\frac{p_i}{\sum_{j\in S} p_j}

Sort the words from likeliest down, then keep only the smallest group whose odds already add up to, say, 0.9 — and throw the rest away before rolling. Re-weight the survivors so they sum to one again. (Keep a fixed number instead, and that's top-k.) Like panning for gold: swirl the worthless sand over the rim, keep the few heavy flecks, and work only with those.

One model, one dial — careful clerk or wild poet.

That's the whole recipe. Temperature warps the odds — sharp and safe, or flat and daring. The cutoff trims the nonsense off the tail. Then the model rolls. Nothing inside the network changed; you only turned a knob. Like a glassblower's furnace: the same glowing gob, and the heat alone decides whether it sets into something rigid and exact or flows into something wild.

🌱 Is there a right next word — or only a temperature you chose?

🌱 If every word is just a weighted roll of the dice, here's the unsettling part: there may be no single right next word waiting to be found — only the one your chosen temperature happened to favor. Turn the dial and a different sentence, just as fluent, falls out. So when the model sounds sure of itself, is it telling you what's true — or only what was likely at the heat you picked?