How a model decides what to look at. · Source

The whole trick: every word gets to look at every other word.

You read a sentence one word at a time. A model doesn't — it drops every word on the table at once and lets each one ask the rest, who here matters to me? That one move is attention, and it's the engine under almost everything: chat, code, images. The rest is just how a word asks.

A word means nothing on its own — only in context.

“Bank” — river or money? No way to tell from the word alone. The model keeps each word as one vector that's a blur of every meaning it's ever had; to pick the live one, it looks at its neighbors. Like a loud party: you don't hear a hundred voices equally — you tune in to the one that matters and let the rest go fuzzy.

Every word puts out three things: a want, an offer, a payload.

Q = xW_Q,\quad K = xW_K,\quad V = xW_V

Query = what I'm looking for. Key = what I'm about. Value = what I'll hand over if you pick me. A search bar, shelf labels, and what's actually in the box. Each word searches the room with its Query and reads everyone else's Keys. Like a murmuration: no starling tracks the whole flock — each just reads its neighbors and moves.

How much should A care about B? Multiply their vectors.

\text{scores} = QK^{\top}

Take A's Query and B's Key and multiply — a dot product, one number, big when they line up. Do it for every pair, then softmax each row so the scores become weights that add to 1: a fixed spotlight budget every word spends. Like one stage light: 80% on the lead, a sliver on two others, none on the rest.

One catch: divide by √d, or it chokes.

\text{weights} = \mathrm{softmax}\!\left(\dfrac{QK^{\top}}{\sqrt{d}}\right)

Long vectors make those dot products huge, and softmax of huge numbers snaps to all-or-nothing — one weight ≈ 1, the rest ≈ 0. That flatlines the gradient and learning stalls. The fix is almost too simple: shrink by √d first. Like a speaker cranked past its limit — it doesn't get louder, it distorts. Drop the gain and the detail comes back.

A word's new meaning is a blend of what it paid attention to.

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\dfrac{QK^{\top}}{\sqrt{d}}\right)V

Spend the spotlight: weight each word's Value by how much attention it got, and add them up. Now the word isn't just itself — it's a mix of the words it cared about. “bank” beside “river” is a different thing than “bank” beside “money.” The line below is the transformer — not a picture of it, the thing itself. Like mixing paint: several pigments pulled into the exact color the stroke needs.

It all happens at once — and attention has no clue what order things are in.

One: every Query hits every Key in a single matrix multiply — massively parallel, which is exactly why this architecture devours GPUs and why scale took off. Two: shuffle the words and the math returns the same answer. Attention sees a bag of words, not a sequence — order gets stapled on separately. Like a dumped jigsaw: every piece, no order; the only reason it knows “dog bites man” from “man bites dog” is a tiny coordinate written on each piece first.

🌱 If it reads a bag of words with coordinates clipped on, is the model really <em>reading</em>?

🌱 If it reads a bag of words with coordinates clipped on, is the model really reading?

You move through a sentence in order, each word pulling you to the next. A transformer sees the whole thing at once and reconstructs order from a position tag. That's the superpower — word 1 can talk to word 1,000 as easily as to word 2. It's also the crack: nothing inside insists on order, so the longer the text, the more weight that little bolted-on sense of place has to carry. So if order is the one thing attention can't feel on its own — maybe the next leap isn't a bigger model. It's a better way to tell it where it is.