How a small model inherits a giant's skill. · Source

A small model can do a giant's job — if it learns from the giant.

The best model is often too big to run anywhere useful — too slow, too costly. So we shrink it. But here's the trick: the small one doesn't learn from the data. It learns from the giant itself. How the master weighs a choice teaches more than any answer key could.

The answer key is a blunt teacher.

\mathbf{y} = \mathbf{e}_c,\qquad y_i = \mathbb{1}[\,i = c\,] \in \{0,1\}

A plain label says one word — 'dog' — and marks everything else equally wrong: wolf and wristwatch get the same zero. In plain words, the target is 1 at the true answer and 0 at every other — no room to say 'close.' Like a coin sorter: it keeps one and sweeps the rest into a single reject cup, the near-match and the foreign piece dumped together.

The giant doesn't say 'dog.' It says how sure.

p_i = \dfrac{e^{z_i}}{\sum_j e^{z_j}}

Ask the giant and you get a spread: 92% dog, 7% wolf, 1% car. That shape is a hidden map — it shows dog sits next to wolf and far from car. The wrong answers carry the lesson. Like a horseshoe pitch: the ringer counts, but you also see which throws leaned close and which sailed wide. In plain words, the raw scores z become a full set of odds, not one winner.

Turn up the heat to hear the whispers.

p_i^{(T)} = \dfrac{e^{z_i/T}}{\sum_j e^{z_j/T}}

That 1% clue is too quiet to learn from. So soften the spread: divide every score by a temperature T before the odds are formed. Crank T above 1 and the faint guesses rise into earshot. Like warming a glass of brandy: a little heat lifts the delicate aromas you'd never catch cold. In plain words, a bigger T flattens the curve, letting the small probabilities matter.

Now the small one copies the whole curve.

\mathcal{L}_{\text{soft}} = -\sum_i p_i^{(T)} \log q_i^{(T)} = D_{\mathrm{KL}}\!\left(p^{(T)} \,\|\, q^{(T)}\right) + H\!\left(p^{(T)}\right)

Train the student to match the teacher's softened spread — not the bare answer, the whole shape, point for point. Like cutting a duplicate key: a blank is ground to follow every ridge of the original, notch for notch. In plain words, the student's softened guess q is pulled onto the teacher's p; since the teacher's part is fixed, closing the gap just means shrinking the divergence between the two curves.

Trust the master — but keep one foot on the truth.

\mathcal{L} = \alpha\,T^2\,\mathcal{L}_{\text{soft}} + (1-\alpha)\,\mathcal{L}_{\text{hard}}

Copy the giant too slavishly and you inherit its mistakes too. So blend: weight α toward the soft lesson, (1−α) toward the real, hard answer. One subtlety — multiply the soft part by T², because softening shrank its pull by exactly that much. Like climbing roped to a leader and an anchor: reach up toward the one ahead, but stay clipped to solid rock below.

$A fraction of the size. Nearly all of the skill.$

A fraction of the size. Nearly all of the skill.

C_{\text{token}} \approx 2P \;\Rightarrow\; \dfrac{C_{\text{student}}}{C_{\text{teacher}}} \approx \dfrac{P_{\text{student}}}{P_{\text{teacher}}} \ll 1

The result is a model you can actually run — a tenth the weight, a tenth the cost per word — carrying nearly the giant's skill. Not because it saw more data, but because it inherited how the giant weighs the world. Like a still: a whole vat of mash boiled down to one small bottle — far less volume, the same character, concentrated. In plain words, cost per word tracks the parameter count P, so a tenth the parameters runs about a tenth as dear.

It learned the doubts, not just the answers.

🌱 The small one learned best from the giant's hesitations — the 7% wolf, the 1% car — everything a single confident answer leaves out. So what does any master truly know that never fits into one reply? And if the lesson was the doubt all along, what is a clean, certain answer quietly hiding?