How text becomes the pieces a model reads. · Source

The model never sees a letter. Or a word.

Before it 'thinks' a single thought, your sentence is shattered into chunks — and each chunk becomes a plain number. The model's entire world is that stream of integers: not letters, not words, but tokens. Everything it knows about language, it knows about these pieces.

Why not just feed it words? Or letters?

Two obvious choices, both broken. A slot for every word means an endless dictionary — and it still trips on the first name, typo, or emoji it has never seen. Go the other way, one letter at a time, and a short sentence balloons into a hundred near-empty steps. Like chopping vegetables: leave them whole and they won't cook through; mince them to paste and you've lost the dish. The right cut is in between.

The fix: split into <em>pieces</em>, not words or letters.

The fix: split into pieces, not words or letters.

Keep the common words whole; break the rare ones into reusable parts. So 'tokenization' splits into 'token' + 'ization' — and that 'ization' turns up again in 'civilization,' 'organization.' A few thousand standard parts can spell almost anything. Like building bricks: you never mold a new piece for every shape — you snap together the ones you already have.

Where do the pieces come from? Greed, one merge at a time.

(x,y)^{\star} \;=\; \arg\max_{(x,y)} \, \mathrm{count}(x,y)

Start from raw characters. Scan a mountain of text, find the most frequent neighboring pair, and fuse it into one new piece. Then repeat — thousands of times. 't'+'h' become 'th'; later 'th'+'e' become 'the.' In plain words: the two symbols seen side by side most often get glued together, again and again, until you have your vocabulary. Like a blacksmith's forge-weld: the two rods you reach for together over and over, you finally hammer into a single bar — and never join by hand again.

Every finished piece is just a number.

\text{ids} = \big[\, \mathrm{id}(t) \;:\; t \in \text{tokens} \,\big], \quad \mathrm{id}(t) \in \{0, 1, \dots, V-1\}

A network does math, and you can't multiply 'cat.' So every piece in the vocabulary gets a fixed ID — one row number out of V. Your sentence turns into a short list of integers, and that is all that flows in. In plain words: a frozen table swaps each chunk for its number. Like a coat-check: you hand over your coat, take a numbered tag, and from then on everyone deals only in tags — never coats.

This is why it can't count the r's in 'strawberry.'

The pieces ignore the seams we care about. 'strawberry' arrives as two or three lumps, never ten separate letters — so the model simply can't see the three r's to count them. A leading space even makes ' dog' a different token than 'dog.' It reads in lumps, never letters. Like a mosaic up close: you can read the whole picture, but each tile is one blunt chunk of color — ask which letter hides inside a tile, and there isn't one.

So a token is the model's real alphabet — a learned code.

\text{tokens} \approx \dfrac{\text{characters}}{4} \quad (\text{English, on average})

Put it together: the vocabulary — often 50,000 to 100,000 pieces — is a compression code, built once and frozen. Common text gets one short token; rare text is spelled out from many. In plain words: on average, about four English characters fold into a single token. Context length, speed, cost — all of it is counted in tokens, not words. The model doesn't read English; it reads this code that English passes through. Like baling hay: loose straw is endless, so you press it into tight bales — then you count, carry, and price the whole field in bales, not stalks.

🌱 It never meets a word. Only the pieces we chose for it.

Its entire world is built from a hundred thousand fragments, fixed in place before it ever learned to think. It never meets a letter, never meets a whole word — only these chunks, and the numbers behind them. So when it reads us, is it reading our language? Or a private dialect, stitched from fragments, that our words just happen to pass through?