How we pull a model's tangled ideas into concepts we can read. · Source

One neuron means a dozen things. We learned to pull them apart.

A trained model folds thousands of ideas into every signal — so peeking at one neuron tells you almost nothing. Like a prism: plain white light looks like nothing in particular, yet split it and a band of pure colors fans out. We built a prism for a model's mind, and read its concepts one clean color at a time.

Why not just read a neuron? It moonlights for everything.

There are far more ideas than neurons, so each cell has to moonlight — one fires for poetry, for Python, for the color teal. Like a smoke alarm: one shrill beep for burnt toast, a hot shower, or a real fire. The signal screams something happened — never what. So reading a neuron tells you nothing you can name.

The fix: a dictionary longer than the model is wide.

x \approx b_{dec} + \sum_{i=1}^{M} f_i\, d_i, \qquad M \gg d

Stop reading neurons. Instead learn a long dictionary of concept-directions — far more entries than there are neurons — and rebuild each signal as a short sum of a few. Like a hardware store's wall of drawers: thousands of little drawers, one kind of part each, so nothing ever shares. In plain words: the signal is a baseline plus a handful of dictionary directions switched on, and the dictionary M dwarfs the neuron count d.

Force just a few on at once — that's what makes them clean.

f = \mathrm{ReLU}(W_{enc}\, x + b_{enc}), \qquad \lVert f \rVert_0 \ll M

An overcrowded dictionary could still smear. The cure is sparsity: at any moment only a handful of entries may switch on. Starved of room to share, each is forced to specialize — to stand for one nameable thing. Like a pipe organ: hundreds of pipes, yet any chord sounds only a few, and each that sounds is one pure note. In plain words: the code keeps almost every entry at zero, so the few that fire each carry a single concept.

Two rules pulling against each other: rebuild it, but cheaply.

\mathcal{L} = \lVert x - \hat{x} \rVert_2^2 + \lambda \lVert f \rVert_1

How do we train it? Squeeze the signal into that sparse code, rebuild it from the dictionary, and grade on two things at once. Like a recipe pared to its fewest ingredients: recreate the full flavor, but pay for every item you add, so only what truly changes the taste survives. The first term says rebuild faithfully; the second charges a fee, λ, for every entry left on. That tug is what carves clean concepts from the blur.

Are the concepts real? Turn one up and find out.

A name you read off a feature could be coincidence. So test it: clamp one entry wide open and watch the model move. Push the 'all things nautical' feature and every sentence drifts toward the sea. Like a drop of dye in clear water: one drop, and that exact color blooms through the whole glass. The feature isn't just a label — it's a handle you can pull.

Name the parts, and the machine's wiring appears.

Now the real prize. With features named in every layer, you can see which ones feed which — this concept switching on that one, layer after layer. A small, readable wiring diagram of an actual thought falls out. Like the exploded view of a watch: every gear laid bare, meshing with the next, the movement finally legible. We stop describing the parts and start reading the machine.

We read its concepts. But whose concepts are they?

We chose the dictionary's size, and how sparse to make it. We read the names off the features ourselves. And some of every signal never fits any clean entry — a residue the prism won't split. So did we discover the mind's own concepts, or quietly hand it ours? 🌱 The door is open. We still can't see the whole room.