How a model holds many specialists — and wakes only a few. · Source

A model can hold a hundred minds — and wake just a few per word.

Imagine a model holding the knowledge of a hundred specialists at once. The trick: for any single word, almost all of them stay asleep. A few wake, give their answer, and the rest never fire. Like a vast warehouse: rows of racks vanishing into the dark, and you switch on the light over just one aisle. Enormous to hold — cheap to use.

Normally, every word pays for every last parameter.

C_{\text{token}} \approx 2P

In an ordinary dense model, each word flows through all the weights — every one touched, every time. The cost per word runs about two operations for every parameter you own, so a smarter model is a costlier model, lockstep. Capacity and cost are welded together. Like a full orchestra: the whole ensemble blasts away — even for one quiet note.

The fix begins: not one big block, but many small ones.

Take that one big processing block and break it into many smaller ones — call them experts. Each is free to drift into a specialty: one leans toward code, another toward grammar, another toward numbers. No one assigns the split — it falls out of training. Like a craftsman's pegboard: not one do-everything gadget, but a wall of dedicated tools, each superb at its own narrow job.

A tiny router reads each word and picks a few experts.

g = \operatorname{softmax}(x W_g), \qquad y = \sum_{i \in \operatorname{TopK}(g,\,k)} g_i\, E_i(x)

Running every expert would defeat the point. So a small router scores the experts for this particular word and keeps only the top few — those run, the rest stay dark. In symbols: score them with a softmax, take the k highest, and the answer is their weighted blend. Like a sorting line: a parcel hits the diverter, gets read in a blink, and is sent down one chute — not every belt in the building.

Only a few fire. So capacity and cost finally split apart.

\text{capacity} \propto N, \qquad \text{cost} \propto k, \qquad k \ll N

Here's the prize. Add more experts and the model knows more — its capacity grows with their number N. But only k of them ever run for a word, so the compute grows only with k. Stock a hundred experts, fire two: a hundredfold the knowledge at twofold the cost. Like a row of taps: the whole pipe is lined with them, yet only two are ever left running.

One snag: the router plays favorites. So we make it share.

\mathcal{L}_{\text{aux}} = \alpha\, N \sum_{i=1}^{N} f_i\, P_i

Left alone, the router falls into a rut — calling the same few experts while the rest never train. So we add a small fairness penalty: it multiplies the share of words each expert got by the share of confidence the router sent it, and bottoms out only when the load is spread evenly. Like irrigating a field: open every gate a fair amount, or a few rows flood while the rest crack dry.

The result: a giant that runs like something small.

Put it together and you get a quietly radical machine — the broad knowledge of a model many times its running cost. The whole library sits in memory; each word summons only the drawer it needs, and the rest of the specialists wait in the dark until called. Like an apothecary's cabinet: a wall of little drawers, and for the remedy in hand you slide open exactly one.

🌱 If only a few specialists wake, who is doing the thinking?

🌱 A mind split into experts, only a handful lit for any one thought — and a quiet router deciding which. So where does the understanding really live: in the specialists who answer, or in the chooser that knows whom to ask? We work this way too — not summoning all of ourselves for every thought, only the part that fits. Maybe knowing whom to wake is the deeper skill.