How a deep net trains fast — by judging each value against its batch. · Source

One trick let deep nets train far faster: judge each value by its batch.

Train a deep network and the numbers racing through it swing wildly — so you had to crawl forward in tiny, careful steps. Batch normalization fixed it with a strange move: judge each value not on its own, but against the others in its batch. The same network suddenly trained several times faster, and far deeper towers became possible.

Every layer learns from a target that won't hold still.

A deep net is a stack: each layer reads what the layer below sends up. But training nudges those lower layers every step — so the distribution of numbers each layer sees keeps drifting. Like laying bricks on wet concrete: every course you set, the foundation underneath shifts again, and you spend your time re-leveling instead of building up.

The fix: score each value against its whole batch.

\mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i, \qquad \sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2, \qquad \hat{x}_i = \frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\varepsilon}}

So pin the numbers down. For each feature, gather its values across the whole mini-batch, subtract the batch average, and divide by the spread. Now every feature arrives centered at zero with a standard width — no more wild swings. Like grading on a curve: a raw score means little until you see it against everyone who sat the same exam.

But forcing zero and one throws away real signal.

y_i = \gamma\,\hat{x}_i + \beta

Flattening every feature to the same shape can erase information the network needs. So hand it two learned dials per feature: one to stretch the width back out, one to slide the center — it can even cancel the whole step if that serves the task. Like a glassblower: melt everything down to one even, glowing blob first, then shape that clean baseline into exactly the form you want.

Now each value's fate depends on its batch-mates.

\hat{\mu} \leftarrow (1-\rho)\,\hat{\mu} + \rho\,\mu_B, \qquad \hat{\sigma}^2 \leftarrow (1-\rho)\,\hat{\sigma}^2 + \rho\,\sigma_B^2, \qquad y = \gamma\,\frac{x-\hat{\mu}}{\sqrt{\hat{\sigma}^2+\varepsilon}} + \beta

There's a price: because the average comes from the batch, one example's output now depends on whoever else rode along. And at test time you may have just one input — no batch to average. The fix: while training, quietly keep a running average of the mean and spread, then use those frozen numbers when alone. Like a seasoned baker: over a whole season of batches you learn the oven's quirks, so you can bake a single loaf by memory.

Why it works isn't the story we first told.

The original pitch was that it stops those drifting distributions — the shifting concrete from before. Later work pushed back: the deeper reason is that normalizing smooths the loss landscape, so the gradients stay tame and big, bold steps don't blow up. Like planing a warped plank: shave off the ridges that forced tiny, cautious passes, and every stroke can run long and confident.

One idea, two twins — and why transformers chose the other.

So the recipe is simple: flatten to a clean baseline, then let the network reshape it. That alone made very deep vision networks trainable. But leaning on the batch is the weak spot — tiny or lopsided batches poison the average. Drop the batch and normalize each example across its own features instead, and you get its twin, layer normalization — the one transformers run on.

🌱 A value judged by its company — is that about the value, or the room?

During training, the very same value is normalized differently depending on which examples happen to share its batch; its meaning bends with the company it keeps. We praise a model for weighing each input on its own merits — yet here, for a while, an input's fate depended on the crowd it arrived with. When does context sharpen a judgment, and when does it quietly replace it?