Why the step size should warm up before it winds down. · Source

The step size isn't one number. It's a whole journey.

A model learns by stepping downhill — but how big a step? Pick one number and freeze it, and you leave real accuracy on the table. The best training changes the rate over time: timid at the start, bold through the middle, gentle at the end. That curve — the schedule — is as much the recipe as the model itself.

At step one, the model is fragile. Don't leap.

Fresh from random numbers, the model barely knows which way is down — the very first slopes it reads are wild and untrustworthy. Take a big step now and it can fling itself somewhere it never recovers from. Like easing a cold engine: floor a frozen motor on a frosty morning and it stalls or grinds — you let it warm before you ask for power. So the rate creeps at first.

Warmup: raise the rate in a straight line.

\eta_t = \eta_{\max}\,\dfrac{t}{t_{\text{warmup}}}, \qquad t \le t_{\text{warmup}}

For the first few thousand steps, lift the rate evenly from near-zero up to its peak — a clean ramp, not a jump. Like a horse breaking into a gallop: it doesn't explode from a standstill, it builds stride by stride to full speed, then flies. In plain words: the longer the warmup, the gentler the climb to full pace.

Near the bottom, big steps overshoot.

Once the model is close to the valley floor, a large step bounds straight past the lowest point and bounces around the basin, never settling. The fix is to shrink the steps as you near the target. Like a hummingbird landing on a blossom: bold darts to get close, then the tiniest hovering nudges to touch down exactly — overshoot and it misses the flower.

Cosine decay: ease down, gentle at both ends.

\eta_t = \eta_{\min} + \tfrac{1}{2}\,(\eta_{\max}-\eta_{\min})\left(1 + \cos\!\frac{\pi\, t}{T}\right)

After warmup, glide the rate from its peak toward zero along a half-cosine: slow at the top, quickest through the middle, slowing again as it lands. Like a ski slope's profile: a gentle crest, a steep middle, then a long gentle runout. In plain words: it spends extra time at the high rate and extra time settling, rushing only in between.

The first recipe fused both into one line.

\eta = d_{\text{model}}^{-1/2}\cdot \min\!\left(t^{-1/2},\; t\,\cdot\, t_{\text{warmup}}^{-3/2}\right)

The earliest transformer schedule fused warmup and decay into one rule: rise straight to a peak at the end of warmup, then fall off as one over the square root of the step. Like a spinning top: flicked to full speed, then a long, ever-slower wind-down. The honest part — the exact shape matters less than having both phases; warmup mainly buys an adaptive optimizer time to trust its own sense of scale before taking big swings.

One curve: find your feet, then cover ground, then settle.

Put it together and the whole run rides one arc: a timid start to find footing on a wild landscape, a bold middle to cover real ground, a gentle finish to settle into the lowest point. Like firing a kiln: ramp the heat slowly, hold it, then cool slowly — rush any stage and the pot cracks. The schedule isn't a detail bolted on; it's half of what training even means.

🌱 Is there a right pace — or only a right pace for now?

The rate that's perfect at step one would wreck the model at the end; the rate that lands it gently would never have moved it at the start. The right step keeps changing as the model learns. So maybe there was never one correct pace — only the pace that fit where it stood. And if that's true of a learning machine, how sure are we it isn't true of us?