How one runaway gradient can wreck a training run — and the cap that saves it. · Source

Most steps nudge. One freak step can erase the weeks before it.

Training is millions of tiny downhill steps. But every so often one batch produces a monstrous gradient — and a single giant step hurls the model clean off the map, undoing weeks of progress in one update. The whole run dies. The cure isn't a better step. It's a leash on the rare violent one.

Why it explodes: multiply enough slopes above one.

\frac{\partial \mathcal{L}}{\partial h_0} = \frac{\partial \mathcal{L}}{\partial h_t}\prod_{k=1}^{t}\frac{\partial h_k}{\partial h_{k-1}}, \qquad \left\|\prod_{k=1}^{t}\frac{\partial h_k}{\partial h_{k-1}}\right\| \sim \lambda^{t}

Like microphone feedback: the speaker feeds the mic feeds the speaker, each loop louder, until the room screams. Backprop does the same down a deep net — it multiplies one slope per layer. If each slope is just above one, fifty of them compound into a monster: at λ=1.5 over 50 layers, a whisper swells to ≈600 million.

A giant gradient means a giant leap — off the map.

\theta \leftarrow \theta - \eta\, g, \qquad \lVert \Delta\theta \rVert = \eta\, \lVert g \rVert

Like an over-drawn slingshot: pull it a normal amount and the stone lands on target; haul it back wildly and the stone rockets clean over the hill, gone for good. The step you take is the learning rate times the gradient's length. Let that length explode to millions and the model leaps off the loss surface entirely — into nonsense, into NaN.

First, measure how big the whole gradient is.

\lVert g \rVert_2 = \sqrt{\sum_i g_i^{2}}

Before you can cap it, you need one number for the size of a vector with millions of parts. Like the diagonal of a box: square every edge, add them, take the root — and every separate side collapses into one straight length. That's the gradient's norm: the single length of the whole step, no matter how many millions of numbers feed it.

The fix: cap the length, keep the heading.

\hat{g} = g \cdot \min\!\left(1,\ \frac{c}{\lVert g \rVert}\right)

Like reining a bolting horse: you pull the reins to bring a wild gallop down to a safe canter — but the horse still runs exactly where you aim it. Clip-by-norm does this to the gradient: under the threshold, leave it alone; over it, shrink it back to that limit. Same direction, shorter stride. The descent still points downhill — it just can't lunge.

The blunt cousin: cap each number, and bend the aim.

\hat{g}_i = \operatorname{clip}(g_i, -v, v) = \max\!\left(-v,\ \min(v,\ g_i)\right)

There's a cheaper way: clamp every single component at ±v. Like sawing a fence flush: a picket line that climbed in a clean slope, sawn level along the top, no longer leans the way it did — you enforced 'nothing too tall' but lost the shape. Capping each number on its own saves the run, yet quietly rotates the gradient off true. By-norm keeps the heading; by-value trades it for simplicity.

Now the worst step is leashed to a known limit.

\lVert \Delta\theta \rVert = \eta\, \lVert \hat{g} \rVert \le \eta\, c

Like a safety net under a trapeze: the acrobat dares the hardest catch because the net is there for the one time they miss. After clipping, the step can never exceed the learning rate times the cap — the violent gradient becomes a firm shove, not a fatal leap. It doesn't fix why the gradient exploded. It just makes sure the one time it does, you survive.

We silenced the wildest correction. Was it a warning?

A gradient that violent was the model screaming that something — the data, the rate, the architecture — was badly wrong. Clipping doesn't listen; it just turns the scream into a polite nudge so the run can go on. We kept our footing. 🌱 But when we cap the rare, violent signal, do we cure the trouble — or only stop hearing it?