How a model learns where each word stands. · Source

Same words, new order, opposite meaning — and attention can't tell.

Last time, attention read a bag of words — shuffle them and the math hands back the same answer. But “dog bites man” and “man bites dog” aren't the same story. Order is meaning. So before the model looks at anything, we do one quiet thing: hand every word a sense of where it stands.

Why not just number them 1, 2, 3…? Because the count runs away.

Tag each word with its raw position — 1, 2, 3, … — and the number keeps climbing. By word 5,000 that tag towers over the word's actual meaning and drowns it out. Worse, a slot the model never trained on is a total stranger. Like a tally that never resets: the marks just pile up until they're all you can see. We need a sense of place that stays small and stays familiar.

The fix: a wave that swings forever but never runs off.

\begin{aligned} \mathrm{PE}_{(pos,\,2i)} &= \sin\!\left(\dfrac{pos}{10000^{\,2i/d}}\right) \\[2pt] \mathrm{PE}_{(pos,\,2i+1)} &= \cos\!\left(\dfrac{pos}{10000^{\,2i/d}}\right) \end{aligned}

Swap the runaway count for a wave. A sine rises and falls between −1 and +1 forever, but always in that tidy band — read its height and you know how far along you are, with no number ever blowing up. Below, pos is the word's slot and i picks the wave; every position becomes a stack of sine and cosine readings. Like a pendulum: it swings on and on, yet never leaves its arc — the phase of the swing tells you the moment.

One wave is ambiguous. A stack of them is a fingerprint.

A single slow wave can't tell slot 3 from slot 300 — both sit at nearly the same height. So use many waves at once: some fast, some slow. The fast ones pin down the fine position, the slow ones the big picture; together they spell one code no other slot shares. Like meshed gears: a tiny one whirs, a great one barely creeps — read them all and you know exactly how far the machine has turned.

The position isn't bolted on the side — it's stirred in.

\mathbf{x}_{pos} = E_{w_{pos}} + \mathrm{PE}_{pos}

You'd expect the position code to ride next to the word. It doesn't — it's added straight onto the meaning vector: same length, same slots, summed. Sounds reckless, yet the model has room to keep the two readable apart. Like salt stirred into soup: you don't serve it in a side dish — you fold it through, and the tongue still tells broth from seasoning.

The quiet payoff: one step ahead is always the same turn.

\mathrm{PE}_{pos+k} = R_k\,\mathrm{PE}_{pos}

Here's the gift hidden in the sines. Move k words ahead and every position vector twists by the same fixed rotation — it depends only on k, never on where you started. So the model can feel “these two are 5 apart” directly, not just where each one sits. Like a spiral staircase: climbing one floor is the same quarter-turn, whatever floor you're on.

That's positional encoding: a smooth address folded into meaning.

Put it together. Attention sees an unordered bag; we hand each word a bounded, smooth, multi-scale address built from waves, add it right into the word, and get relative distance for free. Some models skip the formula and simply learn a position vector for each slot — same job, taught by data instead of by math. Either way, order stops being lost and starts being felt.

🌱 Is a place stapled to a word the same as living in time?

We gave the model an address — but it's still a tag added on top of meaning, read fresh every time it looks. You don't carry a coordinate; you feel order as rhythm, as cause, as one moment pulling into the next. So maybe order shouldn't be a label we attach at all. What if where a word sits should live inside the very act of one word turning to look at another — not stapled on before, but felt in the glance itself?