How a model actually trains — one pace per knob, and a memory. · Source

Plain descent takes one step size for a billion knobs.

Gradient descent has one knob you set by hand: the step size. It uses that same number to move every weight in the model — all billion of them, identically. But the slopes they sit on are wildly different. The good optimizers fix this by giving every weight its own pace — and a short memory of where it has been heading.

The slope you measure jitters — follow it raw and you stagger.

Each step's slope is read from just a handful of examples, so it jitters — pointing a little wrong every single time. Follow it raw and you stagger downhill. Like a windsock in a gusty crosswind: it snaps to a new angle with every gust, and no single flick tells you the steady wind behind them.

Don't chase each slope. Follow their running average.

m_t = \beta_1\, m_{t-1} + (1-\beta_1)\, g_t

Don't chase each twitchy slope. Keep a running blend — mostly your old heading, nudged by the newest reading. The jitter cancels; the true direction builds. Like a heavy boulder rolling downhill: it smooths over every little bump and holds its line. That blend is momentum: the next heading is mostly where you were going, tilted a little toward the latest slope.

One step size can't fit a billion different knobs.

\theta_{t+1} = \theta_t - \eta\, g_t

Here's the catch plain descent never fixes. That single step size η multiplies every knob's slope the same way. But some knobs sit on steep, violent slopes that need tiny, careful steps; others sit on flat, quiet ones and could stride. Like one bike gear for every hill: fine on the flat, brutal on the climb, lazy on the descent. One ratio — one shared η — can't suit them all.

So give every knob its own pace — set by its own roughness.

v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, g_t^{2}, \qquad \tilde{\eta}_t = \dfrac{\eta}{\sqrt{v_t}+\epsilon}

So give each knob its own pace. Track how rough its slope has been — a running average of its squared gradient — and divide that knob's step by it. Knobs whose slopes thrash get reined in; quiet ones get freed to move. Like sanding around a knot: ease off and crawl where the grain fights you, glide fast where it's smooth. In plain words: a big recent slope shrinks the step there; a small one lets it grow.

Both averages start at zero, so the first steps read too small.

\hat{m}_t = \dfrac{m_t}{1-\beta_1^{\,t}}, \qquad \hat{v}_t = \dfrac{v_t}{1-\beta_2^{\,t}}

One snag. Both running averages start at zero, so for the first steps they read far too small — not because the slopes are small, but because the tally just began. Like a coin jar started empty this week: the running total looks tiny only because it began at nothing. The fix divides that head-start out, scaling each average up while it is young — a correction that fades to nothing as the tally fills.

Steady heading, knob-by-knob stride — that's the whole optimizer.

\theta_t = \theta_{t-1} - \eta\, \dfrac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}

Put it together. Momentum picks the direction — the smoothed, jitter-free heading. The per-knob rate picks each stride — long where the slope is calm, short where it thrashes. One line does both. Like a kayaker reading a rapid: hold the line of the main current, but meter each stroke — gentle in the smooth water, firm where it churns. That's Adam, the optimizer that trains almost everything (typically β₁=0.9, β₂=0.999).

🌱 Same hill, same start — but the optimizer chose the valley.

Momentum and the per-knob pace mean the path downhill is no longer the landscape's — it is the optimizer's. Start two of them at the very same spot on the very same hill, and a heavier memory or a bolder pace can carry each into a different valley. So when training stops, did we find the bottom — or just the one our momentum happened to roll us toward?