How a model simplifies itself — without warping how it learns. · Source

Two ways to keep a model humble — once identical, now not.

A model that leans too hard on a few quirks won't travel: razor-sharp on what it saw, lost on what it hasn't. The cure is to keep nudging every weight gently toward zero. For years, two recipes for that nudge were proven identical — so nobody minded which they used. Then the optimizer changed, and the two quietly split apart. Picking the right one buys real accuracy. This is decoupled weight decay.

Big weights make a brittle model. So tether each to zero.

\mathcal{L}_{\text{reg}} = \mathcal{L} + \tfrac{\lambda}{2}\lVert w \rVert^{2}

Let a weight grow huge and the model stakes everything on one loud quirk. So we add a gentle pull toward zero: the larger a weight, the harder it is tugged back. Like a spring: every weight hangs on a faint elastic tether to zero, and the further it strays, the stronger the snap back. In plain words: the loss gets a surcharge of half of λ times the weight's length squared — exactly a spring's stored energy.

For the plain optimizer, the spring is just a steady shrink.

w_{t+1} = w_t - \eta\nabla\mathcal{L} - \eta\lambda w_t = (1-\eta\lambda)\,w_t - \eta\nabla\mathcal{L}

Take the spring's pull and the math is tame: its slope is simply λ times the weight, so each step first scales the weight down a hair, then follows the data. Penalty and shrink are the same move. Like a tailor: redraw the pattern smaller, or take in every seam a notch — the finished garment comes out the same size. In plain words: adding the spring to the loss is identical to multiplying each weight by (1−ηλ) every step.

But the modern optimizer doesn't step evenly.

w_{t+1} = w_t - \eta\,\dfrac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}

The optimizer almost everyone now uses gives each weight its own step size — dividing its move by the square root of how big its recent gradients have been (√v̂), while steering by their running average. A weight that's been swinging wildly takes tiny, careful steps; a calm one strides. Like a tightrope walker: where the rope sways hardest, the feet go smallest.

Feed the spring through that, and the shrink goes lopsided.

\Delta w_{\text{decay}} \;\propto\; \dfrac{\lambda\,w_t}{\sqrt{\hat v_t}+\epsilon}

Here's the snag. Bolt the spring onto the gradient — the obvious way — and the optimizer divides the decay by that same per-weight scale. So the busiest weights, the ones you most want reined in, get shrunk the least. Like one shove on a curling stone versus a hockey puck: the same push barely creeps the heavy stone but sends the puck flying. In plain words: the shrink a weight feels is its size divided by how busy it's been (√v̂).

The fix: take the shrink off the dial.

w_{t+1} = w_t - \eta\,\dfrac{\hat m_t}{\sqrt{\hat v_t}+\epsilon} - \eta\lambda w_t

Like unhitching a trailer: drop the shrink out of the powered machinery and let it stand on its own. Take the normal adaptive step for the data — then, separately, pull every weight toward zero by the same flat fraction, no matter how busy it's been. That splitting-out is the whole idea. In plain words: the −ηλw term now sits outside the √v̂ division, so every weight shrinks by the same (1−ηλ) each step.

Now two knobs that no longer fight each other.

With the decay set free, the two controls finally pull apart: one sets how fast the model learns, the other how hard it simplifies — and turning one no longer smears the other. Like two burners on a stove: raise the heat under one pot without touching the other. The honest catch: for the plain optimizer the two recipes were truly identical — decoupling only earns its keep once each weight steps at its own pace.

🌱 The tidying worked best kept apart from the work.

The whole gain came from one quiet act: lifting the urge to forget out of the machinery that learns, and letting it work on its own. Two pressures — one reaching toward the data, one easing everything back toward nothing — held side by side, never blended. So when does tidying belong inside the work, and when must it stand apart? And is forgetting, done evenly and on purpose, a part of learning — or a quiet craft of its own?