Why the model we keep is one we never trained. · Source

The model we keep is the average of the journey, not the last step.

Stochastic weight averaging keeps a quiet running average of every model the training run passed through — and that phantom usually beats the one you actually stopped on. Like an anvil's polished face: no single hammer-blow shaped that mirror-smooth steel — the average of ten thousand did, truer and calmer than any one strike. We keep the average, almost for free.

Near the bottom, training never sits still.

\theta_{t+1} = \theta_t - \eta\,\tilde{g}_t, \qquad \tilde{g}_t = \nabla L(\theta_t) + \xi_t, \quad \mathbb{E}[\xi_t]=0

Gradient descent doesn't settle at the minimum — it orbits it. Like a buoy at anchor: moored over one spot yet forever bobbing with every wave, never resting on the exact point. In plain words: at the bottom the true slope ∇L is near zero, but each mini-batch still shoves the weights by a little random ξ — so they random-walk in place. The model you stop on is just wherever the bob happened to be.

The fix: keep a shadow copy, blend in a sliver each step.

\bar{\theta}_t = \beta\,\bar{\theta}_{t-1} + (1-\beta)\,\theta_t

So don't trust the last twitch. Keep a second set of weights — a running average — and after every step stir the live model into it by a hair: mostly the kept blend, a thin slice of the new. Like a tub of paint you nudge daily: one fresh dab stirred in each day, the color holding steady, drifting only a hair at a time. In plain words: β near 1 keeps almost all of the old average and lets in just 1−β of the newest weights.

One knob sets how far back it remembers.

\bar{\theta}_t = (1-\beta)\sum_{k=0}^{t-1}\beta^{k}\,\theta_{t-k}, \qquad \sum_{k=0}^{\infty}(1-\beta)\beta^{k}=1, \qquad \tau=\frac{\beta}{1-\beta}

Unroll that blend and each past model still counts — but less the older it is, fading geometrically: weight (1−β)β^k on the step k ago. Those weights sum to exactly 1 (a clean average), and the memory reaches back about 1/(1−β) steps. Like footprints fading in fresh snow: the print you just made is crisp, each older one softer as the drift fills it — at β=0.999, prints stay legible about a thousand steps back.

Why it helps: averaging cancels the jitter.

\operatorname{Var}(\bar{\theta}) = \frac{1-\beta}{1+\beta}\,\sigma^{2}, \qquad N_{\text{eff}} = \frac{1+\beta}{1-\beta}

The snapshots scatter around the true bottom; averaging them cancels the random part. Like pebbles tossed at a mark: no single throw lands on it, but the center of all the landings sits right on target — and the more throws, the surer that center. In plain words: if each wobble were independent, the kept average has variance (1−β)/(1+β) of one snapshot's — at β=0.999, like averaging ≈2000 of them.

And it lands in a wider, calmer spot.

L\!\left(\frac{1}{T}\sum_{t=1}^{T}\theta_t\right) \le \frac{1}{T}\sum_{t=1}^{T} L(\theta_t)

Near the bottom the basin is bowl-shaped, so by convexity the averaged weights score no worse than the typical snapshot — and usually better, nearer the broad flat center. Like the low center of a hammock: sit anywhere and you slide to the wide middle, a forgiving seat that barely shifts when nudged. That width is the point: a flat minimum tolerates new data, so the averaged model generalizes.

But only average selves standing near each other.

Averaging is safe inside one basin. Blend two genuinely different solutions and you get the worst of both. Like splitting two roads around a lake: aim for the average of a route up each shore and you drive into the water — the midpoint of two valleys is the ridge between them. So it only pays off late, once the run has settled into one bowl. And the kept model is a phantom you never ran: its running statistics need one fresh pass before you trust it.

🌱 We keep a self that no single step ever was.

Stack every snapshot and the model we keep is one no single training step ever produced — a quiet consensus of selves. Like a nautilus shell: its smooth spiral is every chamber it ever grew, all earlier selves held at once; no one chamber is the shell. So which is the truest version — the last lesson it learned, or the calm average of everywhere it has stood? 🌱