How one tiny wire lets a network go truly deep. · Source

More layers should help. Past a point, they hurt.

Stack a few layers, the network gets better. Stack a hundred, and something strange happens: it gets worse — and not from memorizing. It can't even fit what it's already seen. The depth itself became the obstacle. The fix is one wire, almost too simple to believe: let each layer's input skip ahead and add itself to the output.

A deep net should copy a shallow one. It couldn't.

Here's the puzzle. A deep network could always match a shallow one — just let the extra layers do nothing, pass the signal along. Yet plain deep stacks couldn't even learn that. Like a bucket brigade: pass a full pail down a long line of hands and it arrives half-empty — every hand-off spills a little. Learning to merely preserve a signal, across dozens of layers, turns out to be hard.

Don't redo the whole thing. Just add the change.

\mathcal{H}(x) = \mathcal{F}(x) + x

So flip the job. Don't ask the layer to rebuild the answer from scratch — ask it only for the correction to add. The input rides through untouched; the layer supplies just the difference. Like a proofreader's red pen: you don't recopy the page, you mark what changes. In plain words: the output is the input plus whatever small edit this layer learned.

Now doing nothing is the easy default.

\mathcal{F}(x)\to 0 \;\Rightarrow\; \mathcal{H}(x) = x

Watch what this buys. To leave the signal alone, the layer just has to make its correction zero — and the input glides straight through. Like a curling stone on ice: release it with no sweep and it carries on exactly as it arrived. Identity is no longer a hard thing to learn — it's the resting state, free. The layer only has to learn how far to nudge off it.

The same wire is a highway for learning.

\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y}\left(1 + \frac{\partial \mathcal{F}}{\partial x}\right)

There's a deeper gift, hidden in the math of backprop. Differentiate the skip and you get a 1 sitting beside the layer's own term. That 1 hands the learning signal straight back, undiluted — so even the earliest layers feel it clearly. Like a fireman's pole: drop onto it at the top and you reach the bottom in one straight slide, skipping every floor. This is how the vanishing gradient dies.

So the whole network is one running total.

x_L = x_\ell + \sum_{i=\ell}^{L-1}\mathcal{F}(x_i)

Zoom out and the tower dissolves. Each block doesn't replace the signal — it adds a small update to a stream that runs through them all. The deep feature is just the first feature plus every block's little contribution. Like coins dropped in a jar: the level only ever rises by small additions, and the total is the whole history. The layers don't transform the signal so much as keep editing a shared one.

This one wire is why networks got deep.

Before the skip, useful networks ran tens of layers. After it, hundreds — even a thousand — trained without falling apart. Like a suspension bridge's main cable: one continuous line, end to end, lets the span run far longer than any single beam could hold. That same wire now threads every transformer block you've heard of. Depth stopped being a wall and became a dial.

If each layer only nudges, where does it think?

Here's the unease. If the signal mostly flows straight through, and each layer only adds a faint correction to a stream that's already nearly the answer — then where does the network actually live? In the busy layers doing the nudging, or in the quiet stream they never stop editing? 🌱 Maybe depth was never a stack of thinkers. Maybe it's one long, slow reconsideration.