How a model encodes where a word sits — by turning, not tagging. · Source

What counts isn't where two words sit — but how far apart.

The usual fix adds an address to every word. But attention only ever compares two words at a time, and a phrase means the same on page 1 or page 50 — all that counts is the gap between them. Rotary position embedding (RoPE) spins each word's viewpoint by where it sits, so when two words meet, the comparison feels only the distance.

Add an address, and the comparison drags it along.

(x_m + p_m)^{\top}(x_n + p_n) = x_m^{\top}x_n + x_m^{\top}p_n + p_m^{\top}x_n + p_m^{\top}p_n

Like swapping street addresses to learn how close you live: you blurt two absolutes when all you wanted was the gap. Adding a position glues where onto what. Multiply two such words and the score splits into four — and the position pieces ride on the absolute seats, not the distance. The model has to dig the gap back out.

The fix: don't add a place. Turn to it.

\begin{pmatrix} x'_1 \\ x'_2 \end{pmatrix} = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}

Like a lighthouse beam: the same light, swung to a bearing set by the moment. RoPE takes each word's numbers two at a time and rotates every pair by an angle set by its position (times a fixed rate). Nothing is bolted on — the word's own vector is simply turned. Position stops being a tag and becomes a direction.

Two turns meet, and only the difference is left.

R(m\theta)^{\top} R(n\theta) = R\big((n-m)\theta\big) \;\Longrightarrow\; \langle R(m\theta)\,q,\; R(n\theta)\,k\rangle = q^{\top} R\big((n-m)\theta\big)\,k

Like two dancers in hold: spin them anywhere across the floor, and the angle between their bodies never changes. Spin the query by its place, spin the key by its place — and when they meet, the two absolute turns cancel. Only n − m, the gap, survives. The score reads pure distance, exactly what we wanted.

One speed blurs. So spin at many at once.

\theta_i = b^{-2(i-1)/d}, \quad i = 1, \dots, d/2, \quad b = 10000

A single turning rate wraps around — far enough apart, two gaps look the same. Like a peal of graduated bells: the little ones swing fast, the great ones swing slow, and only all of them together ring one unmistakable moment. RoPE gives each pair its own rate — quick pairs pin nearby words, slow pairs reach across the page.

It adds nothing — and never fades.

\tilde{q}_m = R_{\Theta,m}\,q_m, \qquad \tilde{k}_n = R_{\Theta,n}\,k_n \qquad (\text{0 new parameters})

Like a revolving door: the turn is the whole mechanism — no extra machinery, and everyone who passes takes the same spin again. RoPE touches only the queries and keys, with a fixed turn and not one new weight to train. It stacks all those little 2-D turns into one block-diagonal spin, re-applied inside every layer — so deep in the stack, the sense of distance never washes out.

Position becomes a turn — and the turn keeps going.

m' = m \cdot \dfrac{L_{\text{train}}}{L_{\text{target}}}

Put it together: position is no longer a tag you bolt on, but a turn you apply — and the model reads only the gap between words. Like the thread on a long screw: one fixed pitch that keeps mating however far the rod runs. Push past the length it trained on, gently shrink each turn to fit, and the same rule still bites.

It knows every distance. It knows no place.

A word never feels its own seat — only the angle between itself and another. It can tell you the gap to any word, yet point to none and say there. 🌱 If a mind holds only distances and never a single fixed place, does where exist for it at all — or only how far apart?