Two ways to learn a language — and what each one becomes. · Source

Two ways to learn a language — and two different minds.

Give a model a mountain of text and it learns by playing a guessing game. But there are two games. One hides what comes next and asks it to predict forward. The other blanks out a word in the middle and asks it to fill the gap from both sides. Same text, same machine — yet the game you choose decides what the model can ever become.

Game one: guess the next word, seeing only the past.

\log p_\theta(x_{1:T}) = \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})

Like a jazz soloist: each note is chosen from the ones already played, the next bar still unwritten. A predict-the-next-word model reads left to right and bets on what follows — never peeking ahead. The math is just the chain rule: a whole sentence's odds are the running product of 'given all this so far, what comes next?'

Game two: blank a word, rebuild it from both sides.

\mathcal{L}_{\mathrm{MLM}} = -\sum_{i \in M} \log p_\theta\!\left(x_i \mid x_{\setminus M}\right)

Like a mosaic restorer: one tile is missing, and you read the pattern flowing in from every side to know exactly which one fits. A fill-in-the-blank model hides scattered words and reconstructs each from the text on its left and right at once. It never predicts forward — it reasons inward, from all around the gap.

The only real difference: what each word may look at.

A = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}} + M\right),\quad M_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases}

Like blinkers on a carriage horse: they don't change the road, they just hide all but what's already passed. Both games run the very same attention — one merely hangs a curtain over the future. Add −∞ to every score pointing ahead and softmax crushes its weight to zero; the past stays lit, the future goes dark. Lift the curtain, and every word sees every other.

But the fill-in game pays a price in feedback.

\text{targets per sequence} = \begin{cases} N & \text{predict-next} \\ \rho N,\ \rho \approx 0.15 & \text{fill-in} \end{cases}

Like panning for gold: you sift a whole pan of gravel to keep only a few bright flecks. The predict-next game grades every word — N words, N lessons. The fill-in game grades only the few it hid, about one word in seven, while the rest is just context it reads but is never tested on. A richer two-sided view, but sparser feedback — so it needs far more passes to drink in the same text.

Only the forward game can actually write.

x_t \sim p_\theta(x_t \mid x_{<t}), \quad t = 1, 2, 3, \dots

Like a glassblower drawing molten glass: the shape can only grow from the glowing leading edge — you can't add to the middle of what's already set. To generate, the model samples the next word from 'given so far,' fuses it on, and asks again. The fill-in model has no leading edge: it always assumed both sides already existed, so it has no honest place to begin from nothing and grow.

The objective is destiny.

Same transformer, same data — only the game differs, and out come two different minds. Predict-next grows a talker: a model that generates, the kind you converse with. Fill-in grows a reader: a model that takes in a whole passage at once, powering search and sorting. The strangest twist? Scaled far enough, the forward-only talker quietly learned to understand too.

🌱 One root, two ways — or two minds?

One model learned to speak by only ever looking forward. The other learned to understand by looking both ways at once. We split language into two games and got two different gifts. So which one is thinking — the voice that always moves ahead, or the gaze that takes in the whole at a glance? And could a single game ever grow both?