How a model keeps every layer's signal from exploding or fading. · Source

Go deep enough and the signal quietly blows up — or fades.

A deep network passes its signal through layer after layer. Each one multiplies and adds, and tiny imbalances in scale compound: by the tenth layer some numbers have ballooned, others shrunk to almost nothing. Fed that lopsided mess, the next layer can't settle, and training stalls. The fix is almost janitorial: before handing the signal on, rinse it back to a sane, standard scale — every layer, every step.

Layer after layer, the scale drifts — and never sits still.

Why does depth do this? Every layer reshapes the numbers, and small biases in scale don't cancel — they accumulate. One feature creeps louder each layer; another fades. So the layers downstream are forever re-aiming at a target that won't hold still. Like a photocopy of a photocopy: each copy drifts a shade darker or lighter, and a hundred copies down it's unreadable. In plain words: nothing is broken — the drift is just what stacking does.

The fix: re-center to zero, rescale to a standard spread.

\hat{x}_i = \dfrac{x_i - \mu}{\sqrt{\sigma^2 + \varepsilon}}

So reset the signal at every layer. Two moves: slide all its numbers so their average sits at zero, then squeeze or stretch them so their spread is one. Whatever came in — tiny, huge, lopsided — leaves in the same tidy shape. Like centering a photo in a mat: nudge the picture until it sits dead center, then size it to fill the same frame every time. In plain words: subtract the average, then divide by the spread.

And it measures each token by its own numbers alone.

\mu = \dfrac{1}{d}\sum_{i=1}^{d} x_i, \qquad \sigma^2 = \dfrac{1}{d}\sum_{i=1}^{d}(x_i - \mu)^2

But the average and spread of what? Here's the quiet, crucial choice. It takes them across that one token's own features — its own private list of numbers — and nothing else. Not the other examples flowing through alongside it. Like a cook balancing a single plate: you taste that dish and adjust it on its own, never by averaging the whole kitchen's output. Each token is made well-behaved using only what's already inside it.

Then two learned dials can stretch and slide it back.

y_i = \gamma_i\,\hat{x}_i + \beta_i

A rigid zero-and-one shape would be a straitjacket — sometimes a feature should be loud, or off-center. So give the layer two trainable controls: one to rescale, one to reshift. It can fine-tune the standard shape — or, if it pays to, undo the reset entirely and recover the original. Like a sail trimmed by two lines: one sets how full it bellies, the other its angle — together they catch any wind, or spill it flat. The reset is a starting point, not a cage.

Because it never looks sideways, it never wobbles.

This 'each token on its own' choice quietly pays off. Since a token's reset never depends on its neighbors, it behaves identically whether you run one example or a thousand, a short prompt or a long one, in training or in live use. Like a plumb line: hang it by a crooked wall or a straight one, in a gale or dead calm, and it still points to the very same true vertical. Nothing around it can pull it off-true.

Now every layer is handed a clean, climbable signal.

Add it all up. Each layer now receives input in the same well-behaved range, so its gradients stay healthy and you can stack a hundred layers and train them fast — the optimization landscape itself turns smoother. Like a canal lock: a boat can't ride a hillside of water in one surge, but step it through chamber after chamber, each one leveling the water anew, and it climbs an impossible height. Paired with a skip wire that lets the signal flow straight through, this is why depth finally became a setting you can turn up.

We told it size means nothing. Only shape. Is that true?

Step back at what we just did. To tame the deep stack, we declared the sheer magnitude of a signal meaningless — only its pattern, its relative shape, is allowed to carry meaning. Every layer is forced to speak at the same volume; the loudness is rinsed away, handed back only if the model earns it. 🌱 But is intensity really empty? When something is felt overwhelmingly, or barely at all, we tend to think that matters. What conviction did we wash out when we made everything neutral by default?