How a word becomes a point in space. · Source

A model can't read a word. So it turns it into a place.

To a model, cat isn't three letters — it's a point in space, a few hundred numbers long. The wild part: the model arranges those points so that nearness means similarity. cat lands a block from dog and a whole city away from Tuesday. Meaning becomes a map — and that map is an embedding.

The clumsy first try: give every word its own switch.

\text{cat}=[\,0,\dots,1,\dots,0\,]\in\{0,1\}^{V},\qquad \text{cat}\cdot\text{dog}=0

The obvious encoding: one slot per word in the vocabulary, all zeros but a single 1. But then every two words sit equally far apart — cat is no closer to dog than to Tuesday. Their overlap is exactly zero. Fifty thousand slots to say almost nothing. Like a wall of switches: one flipped per word, every word a lonely island.

The fix: describe a word with a few hundred dials.

\text{cat}\mapsto\mathbf{v}=[\,0.21,\,-1.3,\,\dots,\,0.07\,]\in\mathbb{R}^{d},\quad d\ll V

Trade fifty thousand mostly-empty slots for a few hundred real numbers — coordinates. Now a word isn't a label, it's a location, and locations can sit near or far. Each number is one shade of meaning; together they pin the word to a spot. Like white light through a prism: one plain beam fans into a whole spectrum, each band a different part of the same thing.

How close are two words? Measure the angle between them.

\cos\theta=\dfrac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert}

To compare two words, look at the angle between their vectors, not the raw gap. Cosine similarity: pointing the same way scores 1 (near-synonyms), at right angles 0 (unrelated), dead opposite −1. Direction carries the meaning; length doesn't. Like two flashlight beams: aimed nearly together they all but merge; the wider the angle between them, the less they share.

Where do the numbers come from? The company a word keeps.

\max\ \sum_{t}\sum_{-m\le j\le m,\ j\ne 0}\ \log p\!\left(w_{t+j}\mid w_t\right)

No one hand-sets the coordinates. The model reads oceans of text and nudges each word's vector to predict the words around it. Words that show up in the same company drift together — cat and dog become neighbors because they keep the same friends. Like a market laid out like-with-like: lemons beside limes, berries beside berries — what keeps similar company ends up shelved together.

The twist: you can do arithmetic on meaning.

\mathbf{v}_{\text{king}}-\mathbf{v}_{\text{man}}+\mathbf{v}_{\text{woman}}\approx\mathbf{v}_{\text{queen}}

Because meaning is now geometry, directions carry concepts. The step from man to woman is the same step as king to queen — so subtract man, add woman, and you land right beside queen. Nobody built a 'gender' axis; it simply fell out. Like two parallel ski tracks: the arrow joining one pair runs parallel to the arrow joining the other.

Put it together: a whole geography of meaning, self-drawn.

A few hundred numbers per word, arranged by company, compared by angle — and out falls a map: neighborhoods are topics, directions are concepts no one named (tense, plurality, sentiment), and the model drew every line itself from raw text. That is an embedding — meaning turned into a place you can measure. Like cracking open a geode: a dull rock outside, a structured world of crystal within.

🌱 If meaning lives in the gaps, what fills the empty ones?

The model never read a definition. It learned what words mean purely from which words they sit near — meaning living in the distances between points, never inside any one of them. So is a word's meaning something it has, or only a web of how far it stands from every other? And out in the empty stretches of that space, where no word has ever landed — do thoughts wait there that we simply never coined a word for?