How understanding arrives long after the memorizing is done. · Source

Perfect on the test it studied. Then, ages later — it clicks.

A model can nail its training set in a flash and still flunk anything new — then, after a small eternity of training that looks like nothing is happening, it suddenly generalizes. Like a still jar of syrup: clear and quiet for ages, then in one breath it locks into crystal. Memorizing came fast. Understanding arrived much, much later.

First it just memorizes — flawless on what it saw, blank on the rest.

At first it simply crams: training accuracy rockets to 100%, while accuracy on unseen questions sits at pure chance. It fit every example one by one without finding the rule underneath. Like a plaster mold cast from a single object — it reproduces that one shape perfectly and fits nothing else. A perfect score that has understood nothing.

The leap comes long after — and progress was hidden all along.

Then comes a long, flat valley — thousands of steps where the test score barely twitches. It looks stuck. But underneath, a real circuit is quietly assembling; the flat curve hides it. Like bamboo: years with nothing above the soil while the roots spread wide — then, almost overnight, it shoots for the sky. The work was always happening out of sight.

Why keep changing after the errors hit zero? A quiet second pressure.

\min_{\theta}\; L(\theta) + \frac{\lambda}{2}\,\lVert\theta\rVert^2

Once it can't lower its mistakes any further, one force still pushes: a small rent on size. The objective minimizes error and the total bulk of the weights — so even at zero error it keeps shrinking, with λ setting the rent. Like a slow river long after it reaches the sea: a faint, ceaseless current still straightening its own bends.

Two ways to ace it. The leaner one quietly wins.

\theta^\star = \arg\min_{\theta:\,L(\theta)\approx 0}\ \lVert\theta\rVert

Two different inner machines can both score perfectly: a bulky memorizer and a lean rule. Same score — but the rule does it with far smaller weights. So the rent picks the small one: of all settings that ace the data, it drifts toward the smallest. Like two ropes on one post — a wasteful tangle and a single neat hitch both hold, but only the hitch spares the rope.

What did it actually learn? To add by spinning a circle.

\cos(\theta_a + \theta_b) = \cos\theta_a\cos\theta_b - \sin\theta_a\sin\theta_b

Peer inside the lean circuit and there's a real algorithm. In the cleanest case we can read — adding numbers on a clock — it learned to turn each number into an angle and add the angles, wrapping around on its own. The identity below is the gear that turns one angle onto the next. Like a railway turntable: roll the engine on, rotate by so much, and it points a new way — no lookup table, a rule.

Fitting the data and grasping the rule were never one moment.

Here's the lesson: memorizing and understanding are two separate events, and the long delay is just the slow migration from the first to the second. It shows up sharpest in small, clean tasks with that rent turned up — a clear proof, not a promise every model will suddenly grok. Like a cellar of aging barrels: idle-looking for a season while a quiet ripening finishes — then, at last, it's good.

🌱 It crammed in a blink and understood in an age. When is a mind done?

It reached a perfect score almost at once — and only much later did the understanding surface. So was the rule already there the moment it memorized, just waiting to be drawn out? And if a mind can keep quietly grasping long after it looked finished, when do we ever call its learning done?