How one table reads every word in — and writes every word out. · Source

One table reads each word in. The same table writes it out.

A model keeps a giant table that turns each word into a vector — its way of reading. To speak, it needs the reverse: a vector turned back into a word. The quiet trick is that these can be the same table, used both ways. Weight tying: read a word and write a word through one shared map.

Normally it lugs two giant tables, not one.

E \in \mathbb{R}^{V \times d}, \qquad U \in \mathbb{R}^{V \times d}

Reading uses a table with one row per word — as tall as the whole vocabulary V, as wide as the model d. Writing needs its mirror, another V×d table, to score every candidate word. Trained apart, they are two separate giants doing reflected jobs. Like a porter carrying two identical trunks: same contents, doubled weight, both to haul and keep.

But it's one vocabulary. So make it one table.

\text{in: } x = e_t \qquad \text{out: } z_i = e_i^{\top} h \qquad \text{tie: } U = E

Both jobs touch the same words, so let one table serve both. The vector e that reads a word in becomes the yardstick it is scored against coming out: a word's score is just its vector dotted with the running thought h. Set the writing table equal to the reading one — that is the tie. Like a tuning fork: strike it to sound the note, hold it to a string to detect that same note.

A guess is just: which vector lines up best?

z_i = e_i^{\top} h = \lVert e_i \rVert \, \lVert h \rVert \cos\theta_i

Tie the tables and prediction turns geometric. A word's score is its vector's alignment with the current thought — large when they point the same way, near zero when they don't. To choose the next word, find the vector that lines up most. Like a weather vane: it swings until it sits along the wind, and the direction it favours wins.

Count what you save: half the table vanishes.

\underbrace{2Vd}_{\text{untied}} \;\longrightarrow\; \underbrace{Vd}_{\text{tied}}

Two tables of V×d numbers become one. For a 50,000-word vocabulary and a 768-wide model, that is about 38 million fewer numbers — near a third of a small model. A bonus: a word's read-in and write-out meaning cannot drift apart, since they are the same numbers. Like one water main feeding a house instead of two parallel pipes — half the digging, one shared pressure.

Reading and writing aren't quite the same job.

The honest catch: the best space for understanding a word and the best space for predicting it are related, not identical. Forcing them equal is a strong, helpful bias — fewer knobs, less overfitting — but it spends a little flexibility. Some models stay tied yet add a small learned tweak. Like one wrench for many bolts: it fits the great majority, and now and then a bolt wants a slim adapter.

One map of meaning, read in both directions.

So the model carries a single geography of words. To read, it looks up where a word sits; to write, it asks which place its thought is nearest. The same map, travelled two ways — built once, serving both halves of language. Like a bridge whose one deck carries traffic each way: you don't pour a second span to come home.

If reading and speaking share one address…

When the place a word is understood is the very place it is spoken from, the line between taking a word in and giving one back grows thin. 🌱 Does the model truly tell knowing a word from saying it — or are they, underneath, one motion read forwards and back?