How attention heads share their memory — and shrink it many times over. · Source

Its memory of the chat balloons — so the heads share it.

Each attention head keeps its own running notes on every word so far — its Keys and Values. Stack enough heads, layers and words and that pile, reread on every step, becomes the real bottleneck. The fix is almost cheeky: most heads don't need their own copy. Like many straws in a few shared jars: each head still sips its own way, but from a memory they hold in common.

Every head hoards its own copy of the past.

M = 2\,L\,n\,H\,d_{\text{head}}\,b \;=\; 2\,L\,n\,d_{\text{model}}\,b

Think of the cache as the model's memory of the chat. In standard attention every head keeps its own Keys and Values — for every layer, every word. Like a long banquet table with an identical bottle at every seat: one shape, copied down the line. In plain words: the memory grows as heads × layers × words, and you reread all of it just to add the next word.

What if every head shared one memory?

\mathrm{head}_i = \mathrm{softmax}\!\left(\dfrac{Q_i K^\top}{\sqrt{d_{\text{head}}}}\right) V \qquad i = 1,\dots,H

Here's the bold move. Keep each head's Query — its own question — but let every head read from one shared Key and Value. Like one espresso machine serving a whole row of baristas: each still pulls their own cup, from a single source. In plain words: the question stays unique to each head; only the memory is shared — so it no longer grows with the head count.

But one shared memory is too blunt.

Sharing one memory across every head goes too far. Each head used to keep notes shaped for its own question; forced through a single set, the sharpest ones blur and training can wobble. Like a whole choir crowding one microphone: you still hear singing, but each distinct voice flattens into the same channel. We saved memory — and quietly lost some of the model's range.

So don't share one — share in small groups.

\mathrm{head}_i = \mathrm{softmax}\!\left(\dfrac{Q_i\,K_{g(i)}^\top}{\sqrt{d_{\text{head}}}}\right) V_{g(i)},\qquad g(i)=\left\lceil iG/H\right\rceil,\quad 1\le G\le H

The fix splits the difference. Sort the heads into a few groups; every head in a group shares one Key and Value, but the groups stay separate. Like carpool vans at dawn: not one car each, not one bus for all — a handful of vans, each with its own little crowd. One dial sets it — call it G shared memories for H heads: G = H is ordinary attention, G = 1 is the blunt extreme, and a few groups in between is the sweet spot.

Eight times less to carry, every word.

M = 2\,L\,n\,n_{kv}\,d_{\text{head}}\,b,\qquad \text{shrink}=\dfrac{H}{n_{kv}}\quad (n_{kv}=H,\,G,\,1)

Now the win. With G shared memories instead of H, the cache shrinks by the factor H ∕ G — often eight-fold. And it pays twice: long context fits, and at every generated word the model reloads that whole memory from slow storage. Like a delivery van carrying one shared crate instead of eight identical ones: every single trip is lighter. Smaller memory, lighter traffic, the same answer.

And you can retrofit a model you already trained.

K^{(g)} = \dfrac{G}{H}\sum_{i\,\in\,g} K_i \qquad \big(\,V^{(g)}\text{ likewise},\ |g|=H/G\,\big)

Best part: you needn't train from scratch. Take a finished model where every head has its own memory, and average each group's memories down to one — then fine-tune briefly to settle it. Like several rivulets merging into a single pool: many near-identical trickles become one. It works because, within a group, those memories were nearly the same all along — which is the whole reason sharing barely hurts.

A hundred heads. A handful of memories.

Grouped-query works because dozens of heads turned out to keep nearly the same notes — different enough to ask their own questions, alike enough to share the answers. So how separate were they, really? Maybe a head's identity was never in what it remembers, only in what it asks. 🌱 Where, then, does a point of view actually live?