Why making it bigger keeps making it better — predictably, then suddenly. · Source

Make it bigger — it improves on a curve, and new skills switch on.

You don't rewrite the model to make it smarter. You just scale it — more parameters, more data, more compute — and its skill climbs a smooth, forecastable curve. Then, at certain sizes, abilities it was never taught simply appear. Like a balloon at dawn: you rise steadily, and all at once a whole new horizon opens below.

The catch: a giant training run is a fortune spent before you know it pays.

Training a frontier model costs millions and runs for weeks. The old way to know whether a bigger one was worth it? Build it and find out. You commit everything up front, and the verdict lands far too late to change course. Like sowing a vast field at dusk: all your seed goes in now, and whether it was worth it won't show until the harvest, a season away.

Plot error against size, and the chaos straightens into one clean line.

L(N) \approx \left(\dfrac{N_c}{N}\right)^{\alpha_N},\qquad \alpha_N \approx 0.076

Measure the model's error as you grow the number of parameters N. On log-log axes the points fall on a straight line: error drops as a fixed power law of size. In plain words, every 10× in size shaves off the same fixed slice of error — so you can read tomorrow's model off today's line. Like rails to the horizon: dead straight, and you can see where they lead long before you arrive.

Size, data, compute — and compute is just the other two multiplied.

C \approx 6\,N\,D

The curve has three dials: parameters N, training tokens D, and the compute C you spend — and they aren't independent. Each token costs about six operations per parameter, two on the way forward and four on the way back. In plain words, total work is size times data, six times over. Like weaving cloth: the yardage you get is the warp threads times the weft passes — neither alone makes a single yard.

You never reach zero. A floor of pure randomness is always left.

L(N,\,D) = E + \dfrac{A}{N^{\alpha}} + \dfrac{B}{D^{\beta}}

Grow the model and the data forever and the error still won't hit zero. The full law stacks two shrinking penalties — one for a too-small model, one for too little data — on top of a fixed floor E. That floor is the true randomness of language itself: even a perfect model can't call a coin flip. Like polishing a mirror: each pass gains a little less, and physics sets a smoothness you grind toward but never beat.

On a fixed budget, grow the model and the data together — not one alone.

N_{\text{opt}} \propto C^{a}, \quad D_{\text{opt}} \propto C^{b}, \quad a \approx b \approx \tfrac{1}{2}

Fix the compute budget and ask: bigger model, or more data? The answer is both, in near-equal measure — when the budget doubles, grow each by about the same factor. The sweet spot sits near twenty tokens of text per parameter. In plain words, a giant model fed too little is just an undertrained giant. Like rowing with two oars: pull one far harder than the other and you don't go faster — you only spin.

Ride that smooth curve down far enough, and new abilities snap on.

Here's the twist that makes scale feel like magic. The loss curve falls smoothly — yet a specific skill, like multi-digit arithmetic, can sit flat at chance for model after model, then snap to working almost at once. It was building underneath all along; the sharp jump is partly the yardstick, which scores it all-or-nothing. Like a sand pile at its tipping angle: add grains one by one and nothing moves — until a single grain sends the whole face sliding.

🌱 The line never bends. So is there a top — or only more of the same?

Every law here says the same thing: spend more, get more, along a curve that just keeps going — no bend, no ceiling in sight. But a line that never turns is also a line that never arrives; it only promises the next rung is in reach, never that it's the last. So when a model finally seems to understand, is that something genuinely new — or just the same smooth climb, sharpened until it surprises even us?