How a model trains on memory it doesn't have. · Source

To learn, it must remember every step. That's the wall.

To improve, a model runs your input forward through every layer — and to learn from its mistake, it must keep every one of those in-between results, ready for the backward pass. The math was never the problem. The memory is. On a deep enough model, holding the whole forward pass at once is exactly what runs you out of room.

Why keep it all? To trace the blame back.

M_{\text{store}} = \sum_{\ell=1}^{L} a_{\ell} = O(L)

Learning asks each layer one question: how much did you add to the error? Answering it needs that layer's saved output from the trip up — so the model keeps them all. Like a chef leaving every prep bowl on the pass: to find which step soured the sauce you keep every bowl out, and the counter fills before the dish is done. One stored result per layer means the memory bill climbs in lockstep with the depth L.

So just don't save it? Then the work explodes.

C_{\text{recompute}} = \sum_{\ell=1}^{L} \ell = \frac{L(L+1)}{2} = O(L^{2})

There's an obvious dodge: keep nothing, and whenever the backward pass needs a layer's value, just compute it again from the input. The memory vanishes — but the work blows up. Like walking back to the ground floor every time: to reach the ninth floor you start again from the bottom, again and again. Redo every layer from scratch and the total effort grows with the square of the depth.

The fix: keep a few waypoints, redo the rest.

M(k) = \underbrace{\tfrac{L}{k}}_{\text{waypoints}} + \underbrace{k}_{\text{one segment}}

Neither extreme works, so take the middle. Save the activations at just a few layers and throw the rest away; when the backward pass needs a discarded value, re-run only the short stretch from the nearest saved point. Like a hiker planting a flag each mile: you don't memorize every footstep — to retrace, walk back to the last flag and redo that one short leg. Checkpoint every k layers and you hold just L/k flags plus the k steps of the leg you're redoing.

How far apart? The square root of the depth.

\frac{dM}{dk} = -\frac{L}{k^{2}} + 1 = 0 \;\Rightarrow\; k^{*} = \sqrt{L}, \quad M^{*} = 2\sqrt{L} = O(\sqrt{L})

Space the waypoints too far and each re-walk is long; pack them too close and you've saved no memory. The two costs pull against each other, and the total is smallest when they're balanced. Like a relay team spacing its handoffs: legs too long and each runner is spent, too short and you've hired runners for nothing — the team is fastest when every leg is the same short length. Balance lands at one flag every √L layers — and the memory drops to about √L.

The cost? One extra pass. Worth it.

C_{\text{ckpt}} \approx F_{\text{fwd}} + F_{\text{bwd}} + F_{\text{recompute}} = F + 2F + F = \tfrac{4}{3}\,C_{\text{full}}

Nothing is free: every discarded leg gets re-walked once during the backward pass, which sums to redoing the whole forward trip exactly one extra time. Like renting a small van and making two trips: you couldn't afford the big truck, so you drive the route twice — a little more time, and everything still moves. A normal step is one trip forward and two back; one extra forward makes four parts where there were three — about a third more compute for a near-square-root cut in memory.

Now memory is a dial, not a wall.

That's the whole trick: memory stops being a wall and becomes a dial you can turn. Spend a little more time and you fit a model that should never have fit — train something far deeper than your hardware can hold, just by agreeing to rebuild what you chose to forget. Like a ship built inside a bottle: the thing that can't fit through the neck gets there anyway, assembled by a patient, clever method. The work you threw away was never lost — only cheap to make again.

🌱 Is a memory you can rebuild a memory at all?

It chose to forget — then, when it needed the thought again, it simply rebuilt it. We rarely get that option: our past is stored, not re-derived, and what we lose is usually gone. But if a thing can always recompute what it forgot, was forgetting ever a loss? Maybe remembering everything is the expensive habit — and redoing the work, quietly, is the cheaper way to think.