How a tiny winner hides inside a giant from the start. · Source

A trained net is mostly padding. The winner was there at birth.

Train a giant network and you can throw away most of it. Buried in its random starting numbers was a tiny subnetwork that, alone, could have learned the whole task. Like a lottery drum: thousands of balls tumble, and the rare gold one was gold from the start — training doesn't build the winner, it finds the ticket that was already lucky.

Delete the smallest 90% of the weights. It barely flinches.

m \in \{0,1\}^{|\theta|}, \qquad s = 1 - \frac{\lVert m\rVert_0}{|\theta|}

Take a finished network and cut the connections whose weights sit nearest zero — often 90% of them — and accuracy holds. A mask tags every weight keep-or-cut, a 1 or a 0; the sparsity is just the fraction you cut. Like a packed moving truck: most of what's inside is padding — pull it out and the few real pieces of furniture are all that was ever being carried.

Why call it a lottery? Count the possible subnetworks.

\#\{\text{subnetworks}\} = 2^{N}, \qquad N = |\theta|

Every weight is either in or out — so a network of N weights hides 2^N possible subnetworks, more than there are atoms in the sky. Almost all are useless; a vanishing few are winners. Like a tray of tossed coins: with enough coins the patterns of heads and tails are astronomical — nearly every one is nothing, a rare few spell a winner, and the toss was decided the instant they landed.

The recipe: train, cut the weak, then rewind the rest.

m_i = \mathbb{1}\!\left[\,|\theta_i| \ge \tau\,\right], \qquad \theta_{\text{ticket}} = m \odot \theta_0

Here's how you find a winner. Train a while, then cut every weight that stayed small (below a cutoff τ) and keep the ones that grew strong. Now the twist: don't fine-tune the survivors — snap them back to the exact values they were born with, θ₀. Like a ship's rigging: drop the slack lines, keep the few under real strain, and re-tension each to precisely the setting it was first rigged at — not a fresh guess.

The proof: keep the wiring, reshuffle the start — it fails.

a(m \odot \theta_0) \approx a_{\text{dense}} \;>\; a(m \odot \theta_0'), \qquad \theta_0' \sim \mathcal{D}

Why rewind, instead of starting those survivors fresh? Because the luck is in the birth values. Keep the exact same wiring but re-roll its starting numbers at random, and the magic is gone — it trains slower and lands lower. Like a music box: keep the very same comb and drum, but knock its pins to random spots and only noise comes out. The melody lived in the original placement, not the mechanism alone.

Peel a little, rewind, repeat — down to a tenth that still wins.

\frac{\lVert m\rVert_0}{|\theta|} = (1-p)^{r}

One pass prunes gently, so repeat: cut a slice, rewind, retrain, cut another. Shave a fraction p each round and after r rounds only (1−p)^r of the weights remain — peel five times and barely a third is left. Yet at 10–20% of its weights the network still matches the full thing. Like whittling a flute: shave a curl, test the note, shave again — the thick block becomes a slender instrument that plays every tune the whole block could.

The honest catch: big nets rewind to an early step, not step zero.

\theta_{\text{ticket}} = m \odot \theta_k, \qquad 0 < k \ll T

The first story said step zero — the very first instant. At real scale that's too fragile. The fix is gentler: let the net train a short while first, then rewind the survivors to that early checkpoint θ_k, not to birth. Like curing concrete: you can't trust the joint the instant it's poured — give it a few hours to set, and then it holds. A winning ticket needs a moment to firm up before it's stable.

🌱 We trained a mansion to keep one room. Why build the mansion?

If a sliver could have done it all, why train the whole giant? Maybe the extra weights weren't waste — maybe they were the many paths that let training stumble onto the lucky one. Take them away first and the search comes up empty. So which is it: is all that capacity the cost of finding the answer — or did the answer need room to be found?