Why one neuron means many things at once. · Source

Peek at one neuron — and it lights up for a dozen unrelated things.

You'd expect one neuron to mean one thing. Instead, a single one fires for DNA, for French, for a perfect free throw. Why the jumble? Because the model is holding more ideas than it has neurons — so it makes them share. Read one neuron and you read a blur. The true unit of meaning is hiding somewhere else.

An idea isn't a neuron. It's a direction.

\mathbf{x} \approx \sum_{i} f_i\,\mathbf{d}_i

Here's the reframe: a concept isn't one neuron firing — it's a whole direction across all of them. What you observe is simply those directions added up, each turned up by how strongly its idea is present. Like a chord: press several strings at once and you don't hear them apart — you hear a single sound, every note still folded inside.

Want zero confusion? Make every idea perpendicular.

\mathbf{d}_i \cdot \mathbf{d}_j = 0 \;\Rightarrow\; \text{at most } d \text{ such directions in } \mathbb{R}^{d}

Two directions at a right angle never bleed into each other — perfectly separate, zero overlap. So why not give every idea its own perpendicular direction? Because you run out. Like the corner of a room: three edges meet at clean right angles, and there's simply nowhere to stand a fourth. A space of d dimensions hands you exactly d clean directions — no more.

Loosen 'exactly 90°' to 'about 90°' — and room explodes.

N \sim \exp\!\big(c\,\varepsilon^{2} d\big), \qquad |\mathbf{d}_i \cdot \mathbf{d}_j| \le \varepsilon

Here's the escape. Drop the demand for perfect right angles, allow a sliver of overlap, and the number of usable directions stops matching the dimensions and starts exploding past them. Like a sea urchin: from one small body, hundreds of spines fan out, no two quite parallel, each its own near-distinct heading. Almost-perpendicular turns out to be almost as good — and there's room for vastly more of it.

Overlap should cause chaos. Sparsity keeps it quiet.

\langle \mathbf{d}_i, \mathbf{x}\rangle = f_i + \underbrace{\sum_{j \ne i} f_j\,(\mathbf{d}_i \cdot \mathbf{d}_j)}_{\text{interference}}

But overlap means cross-talk: read off one idea and you pick up faint smudges of every idea sharing its slant. Why doesn't that wreck everything? Because ideas are sparse — at any moment only a handful are switched on. Like a spice drawer: a hundred jars, but a single dish reaches for only three. Two flavors that would clash almost never go in together — so the clash almost never happens.

A faint hum remains. A simple floor wipes it.

Even when ideas are sparse, a little interference always hums underneath. The fix is almost crude: set a floor — zero out anything below it, let everything above pass straight through. Faint cross-talk never clears the line; only a real, present idea rises over it. Like a snow line on a mountain: below a certain height nothing sticks and the rock stays bare; only high enough does the white hold. That one threshold is what keeps all the crowding safe.

So one neuron ends up meaning many things at once.

n_k = \mathbf{e}_k \cdot \mathbf{x} = \sum_{i} f_i\,(\mathbf{d}_i)_k

Now the opening puzzle dissolves. A neuron is just one axis of the whole space — and it lights up for every idea whose direction leans even slightly its way. Dozens of unrelated concepts, each tilting a little toward the same axis, all firing the same cell. Like a watering hole: zebra, stork, and warthog share one pool — not because they're alike, but because they all happen to pass through. The neuron was never confused. We were — for reading meaning off the wrong thing.

🌱 If no neuron holds an idea, where does it live?

We keep hunting for the cell that holds a memory, a word, a face — and there isn't one. Every idea is smeared as a direction across thousands of neurons; every neuron carries scraps of thousands of ideas. The meaning is real, yet it has no address. Maybe a thought was never in a place at all — only in a pattern, the way a chord is nowhere on the strings and everywhere in the air.