How a model improves at all — one small step downhill. · Source

A model isn't written. It's nudged downhill, one step at a time.

Nobody hand-codes a model's billions of settings. It finds them — by measuring how wrong it is, then nudging every knob a hair toward less wrong. Do that a million times and a blank network becomes a mind. The method has a plain name: gradient descent — walking downhill on your own mistakes.

One number says how wrong it is. And a billion knobs to turn.

Training crushes every mistake into a single score — the loss. Lower is better, and it depends on all the knobs at once. Like a wall of identical valves stretching out of sight: turning them one at a time, trying combinations, would outlast the universe. You can't search this. You need a direction — which way each knob should turn, right now.

Which way is downhill? Ask the slope.

\nabla L(\theta) = \left( \dfrac{\partial L}{\partial \theta_1},\ \dots,\ \dfrac{\partial L}{\partial \theta_n} \right)

For one knob, its slope tells you which way the loss climbs, and how steeply. Bundle every knob's slope into a single arrow and you have the gradient — the steepest way uphill. Like reading a slope through your boots in fog: blind to the whole mountain, your feet still feel exactly which way the ground tilts. In plain words: the gradient is every knob's slope at once, packed into one arrow pointing at the fastest climb.

Want to go down? Step against the arrow.

\theta_{t+1} = \theta_t - \eta\, \nabla L(\theta_t)

The gradient points uphill — the last place we want to be. So flip it: take the knobs you have and step a little the opposite way. Then again. And again. Like tuning a guitar: too sharp, loosen a hair; too flat, tighten a hair — never yanking, always nudging toward the note. In plain words: new knobs = old knobs minus a small step (the η) down the slope. That one line, run a million times, is learning.

The step size is everything. Too big or too small, both fail.

L(\theta - \eta \nabla L) \approx L(\theta) - \eta\, \lVert \nabla L \rVert^{2}

That step size — η, the learning rate — is the most important dial in training. In plain words: one small step downhill lowers the loss by about (step size) × (steepness)², honest, predictable progress — but only while the step stays small enough that this straight-line estimate holds. Like a putt: rap it too hard and you race clean past the hole; tap it too soft and you stop short. The right pace drops it in.

Can't weigh every example each step. So take a handful.

\nabla L(\theta) \approx \dfrac{1}{|B|} \sum_{i \in B} \nabla L_i(\theta)

The true slope means running every training example — far too slow to do each step. So grab a small random batch and average just their slopes. Like judging a sack of coffee by one handful: you don't pour out the whole sack — one honest scoop estimates the rest. In plain words: a cheap, jittery guess at the slope — and the jitter even helps shake you out of shallow ruts. This is stochastic gradient descent.

That's the whole engine. You're done when the ground goes flat.

\nabla L(\theta^{*}) = 0

Strip off the names and learning is four moves on a loop: measure how wrong, read the slope, step against it, repeat. Billions of knobs, one rule. In plain words: you've arrived when the gradient hits zero — no slope left, the ground is flat, nowhere down to go. That flat spot is the bottom the whole walk was bending toward. A blank network became a mind, and this — patient, blind, relentless descent — is the only thing that put it there.

🌱 But which bottom? It depends on where you began.

🌱 A landscape this vast has many valleys, not one. Flat ground only means a bottom, not the bottom — start somewhere else, step a little differently, and you come to rest in a wholly different basin. So the model you get isn't the only one that fit; it's the one your path happened to settle into. Is there a single true answer waiting down there at all — or only the hollow you rolled into, and the road that took you there?