How a model reads on and on without re-reading a thing. · Source

It reads the whole book — without ever re-reading a word.

Attention re-reads every word against every other; twice the text, four times the work. But there's another way. Carry one running state — a small, fixed summary — and fold each new word into it as you go. No looking back, no growing grid. The cost stays flat, however long the page. The question becomes: what's worth keeping?

Don't keep every word. Keep one running summary.

Here's the other way, in one move. Instead of storing every word and comparing them all, keep a single state — a compact running summary — and stir each new word into it. Like one stockpot you keep adding to: you don't line up every ingredient again at each step; you fold the new one in and taste. The pot holds the whole meal so far, in one place.

The whole machine in one line: fade, add, read.

h_t = A\,h_{t-1} + B\,x_t,\qquad y_t = C\,h_t

The state updates by a simple rule. Each step, the old state fades a little (the A term), the new word is added in (the B term), and what you report is a read-off of the state (the C term). Like a canyon echo: each fresh shout joins a softly-fading reverberation of every shout before — and what your ear catches is the blend.

Freeze the rule, and it becomes one fixed filter.

y_t = \sum_{k=0}^{t} C A^{k} B\,x_{t-k} = (K * x)_t,\qquad K_k = C A^{k} B

If A, B, C never change, unroll the recurrence and it collapses into a single filter slid across the whole input — computable all at once, blazingly fast. But there's a price: like one escalator at a fixed speed, it carries every word at the identical pace. It can't rush the filler or linger on the one word that matters. Fast, but blind.

The leap: let the word decide what to keep.

\Delta_t,\,B_t,\,C_t = f(x_t),\qquad h_t = \bar{A}_t\,h_{t-1} + \bar{B}_t\,x_t,\quad \bar{A}_t = \exp(\Delta_t A)

Now the real trick. Make the gates depend on the word itself: how much to write in (B), how fast to forget (the step Δ), what to read out (C) — all functions of the input. Like a surfer reading each wave: let the small swells slide under, paddle hard for the one worth riding. The model now chooses, word by word, what's worth remembering.

Selectivity broke the shortcut. A scan rebuilds it.

(A_j,\,b_j)\oplus(A_i,\,b_i) = (A_j A_i,\;\, A_j\,b_i + b_j)\quad\Rightarrow\quad O(\log n)\ \text{depth}

Letting the word steer means there's no single fixed filter anymore — so the all-at-once trick is gone. But the step is still associative: combining two runs of 'fade-then-add' gives one run of the same kind. So pair them up and merge in parallel. Like a knockout bracket: whole rounds resolve at once, and the champion is known in just a handful of rounds — not one match at a time.

Linear cost — and a memory that never grows.

\text{time } O(n\,d\,N),\qquad \text{state size } O(d\,N)\ \text{(fixed in } n)\qquad \text{vs. attention } O(n^2 d)

Add up the bill. Work grows straight-line with length, not as its square. And at generation time you carry just one fixed-size state — it never swells the way a stored history does, no matter how long the talk runs. Like a tightrope walker's balance pole: the same single pole in your hands at step ten or step ten-thousand. Length stops being a wall.

It keeps a summary — never the past itself.

An all-pairs reader keeps every word and can always glance back. This one keeps only an evolving state — and must decide, the instant each word arrives, what to fold in and what to let fade. It can never un-forget. 🌱 If a mind chooses what to keep the moment it sees it, and never revisits the raw past — is what it carries memory, or only the story it told itself about what mattered?