The one shared lane every layer writes to and reads from. · Source

It isn't a tower of boxes. It's one lane everything writes to.

We picture a model as a stack of layers, each handing its work to the next. Not quite. Running the full height of the model is a single shared vector — one lane of numbers, one per token. Every layer reads from this one lane and writes its answer back into it. Nothing is handed box to box. Everything meets in the same place.

How does the last layer use what the first one found?

If each layer heard only the one just before it, an early discovery would survive only by being copied forward, step after step — and whatever blurs along the way is lost. Like a game of telephone: a sentence whispered down a long line comes out mangled. The fix isn't a cleaner whisper. It's to stop whispering and let every layer write on the same shared lane.

Each layer reads the lane, adds to it, erases nothing.

x_{\ell+1} = x_\ell + F_\ell(x_\ell) \qquad x_L = x_0 + \sum_{\ell=0}^{L-1} F_\ell(x_\ell)

Here's the move. A layer reads the whole running lane, works out a correction, and adds it on top — it never overwrites what's there. Like a wall built course by course: each layer lays its row on the last, nothing torn down, so the finished wall is just every course summed. In plain words: the model's final state is the starting vector plus every layer's added note.

On that lane, a meaning is a direction.

\text{write: } x \leftarrow x + a\,\mathbf{u} \qquad \text{read: } s = \mathbf{u}^\top x \qquad x \approx \sum_i a_i\,\mathbf{u}_i

How do you store an idea inside one list of numbers everyone shares? As a direction. To write a feature, a layer adds a small arrow pointing a certain way; to read it, a later layer measures how far the lane leans along that arrow. Like colored stage lights on one screen: each lamp adds its hue to the same wall, and to read one lamp you check how much of its color shows. In plain words: writing adds an arrow, reading takes a shadow, and the whole lane is those arrows summed.

One lane, fixed width — yet it carries thousands of ideas.

\langle \mathbf{u}_i, \mathbf{u}_j \rangle = 0 \Rightarrow \text{at most } d \text{ directions} \qquad |\langle \mathbf{u}_i, \mathbf{u}_j \rangle| \le \varepsilon \Rightarrow N \sim e^{\,c\,\varepsilon^{2} d}

The lane has a fixed width — a set number of slots, shared by every layer and head. So how does it hold far more ideas than it has slots? It packs them nearly, not perfectly, apart. Like birds crowded on one wire: a few can sit with clear space between them, but let them perch almost wing to wing and the same wire holds far more — at the cost of a little jostling. In plain words: only a few directions can be exactly separate, but a huge number can be almost separate.

Stop at any layer and you can read its current guess.

p^{(\ell)} = \mathrm{softmax}\!\left( W_U\, x_\ell \right)

Because the lane is one running sum, you can stop at any layer, push that half-finished vector through the model's output map, and read a provisional answer — one that grows sharper with depth. Like a photo developing in the tray: the image is legible long before it's done, just fainter and rougher. In plain words: feed the partial lane to the same map that reads the final guess, and out comes the model's best answer so far (give or take one last rescaling).

So the model isn't a tower. It's one slowly-clearing message.

Put it together. There's no relay of sealed boxes — there is one stream, and the whole model is the first vector plus every layer's note, read off at the end. Attention and the feed-forward blocks never speak to each other directly; they speak by leaving marks on this shared lane. This is the residual stream — not the plumbing between the thinking, but the place the thinking is actually kept.

Nothing is erased — only written over. Is that forgetting?

Every change a layer makes is added to the lane — never deleted. To take back a thought, the model can't reach in and pull it out; it can only write the exact opposite on top, so the two cancel. The original is still there, buried inside the sum. 🌱 If a mind can only add, never erase, is anything it once thought ever truly gone?