How a model runs in four bits without falling apart. · Source

Throw away three of every four bits — it barely notices.

A trained model is billions of long, exact numbers. Quantization rounds each to a much coarser one — sixteen bits down to four — and it still answers almost the same. Like a bonsai: a whole tree's sweep, kept in a fraction of the wood. The win is real: a model that filled a server now fits on hardware a fraction its size.

The cost isn't the thinking. It's hauling the numbers.

M = \dfrac{P \cdot b}{8}\ \text{bytes}

To write each word, the model hauls every one of its billions of weights out of memory — then again for the next word. The bottleneck isn't the math; it's the moving. Like movers carrying a house upstairs: the labor is the lifting, trip after trip — not deciding where each piece goes. The bill is just count times size: seven billion weights at sixteen bits is fourteen gigabytes; at four bits, three and a half.

The move: snap every number to the nearest rung.

q = \mathrm{round}\!\left(\dfrac{x}{s}\right), \qquad \hat{x} = s\,q

Drop the smooth sixteen-bit value; keep only the nearest rung on a short ladder, stored as a tiny integer. Like shoe sizes: feet vary by the hair, but stock comes in a few fixed sizes, so each foot takes the closest one. In plain words: divide by the step, round to a whole number — that integer is all you save — then multiply back to use it.

Where do the rungs go? Set them by the biggest number.

s = \dfrac{\max_i |x_i|}{2^{\,b-1} - 1}

How wide is each step? Take the largest weight, let it land on the top rung, and space the rest evenly down to zero. Like cutting one plank into equal boards: ask for more and each board is finer; fewer, and each is coarser. The cost of few bits is plain: four bits buys just seven rungs each side of zero — sixteen bits buys over thirty thousand.

Why it survives: the numbers huddle, each barely moves.

\lvert x - \hat{x}\rvert \le \dfrac{s}{2}

It works because trained weights aren't scattered — almost all huddle near zero. A short ladder sized to that huddle catches nearly every one, and rounding moves each by at most half a step. Like a bait ball: the whole school packs into one tight sphere, the water around it empty — so you place your few markers where the fish really are. Across billions, the tiny overs and unders wash out.

One giant outlier wrecks it for everyone.

s_g = \dfrac{\max_{i \in g} |x_i|}{2^{\,b-1} - 1}

But a few weights are freaks — far larger than the rest. Size the ladder to one freak and the whole huddle collapses onto a single rung, all detail lost. The fix: don't share one ladder — split the weights into small groups, each with its own step sized to its own biggest value. Like a giant sequoia over saplings: measure everything by the sequoia and each sapling reads as zero — so you measure the small ones on their own scale.

A quarter of the bits. Nearly the same mind.

\dfrac{M_{16}}{M_{4}} = \dfrac{16}{4} = 4\times

Put it together: a sixteen-bit giant runs in four bits — a quarter of the memory, almost the same answers. What carried the knowledge was the pattern of the weights — which leans which way, how they relate — not their last fine decimals. Like a shadow puppet: flatten a stag to a plain black outline, no color, no texture, and it is still, unmistakably, a stag.

If three-quarters was disposable, what was it for?

We burned twelve of every sixteen bits and the model never missed them. So were they waste — dead weight all along? Or was that slack the very room it needed while it learned, scaffolding you can strike once the building stands? 🌱 How much of any mind is load-bearing — and how much is just the space it grew in?