Why grabbing the best word now can lose the best sentence. · Source

The best word now can cost you the best sentence.

A model hands you the odds for the next word — but a sentence is hundreds of choices in a row. Grab the single likeliest word every time and you write yourself into corners: a strong opener can dead-end, while a humbler one opens onto something far better. Like the shortest checkout line: you join the one that looks fastest now, then crawl while the longer line flies. Best-this-instant is rarely best-overall.

The goal isn't the best word — it's the best <em>sentence</em>.

The goal isn't the best word — it's the best sentence.

P(y_{1:T}\mid x)=\prod_{t=1}^{T} p(y_t\mid y_{<t},x),\qquad \log P=\sum_{t=1}^{T}\log p(y_t\mid y_{<t},x)

Score a sentence as a whole and its odds are every word's odds multiplied together — in logs, simply added up. One brilliant word can't rescue a path that turns unlikely later. Like one load on a chain of links: the chain holds only if every link does; a single weak link drops the whole weight. Greedy maximizes each link alone — never the whole chain.

Greedy decoding: take the top word, then commit.

y_t=\arg\max_{w\in V}\ p(w\mid y_{<t},x)

The simplest method takes the single highest-probability word at each step — fast, certain, and blind. It never looks ahead and never takes anything back. Like writing in pen: each word is inked the instant you choose it, so when a confident opening corners you three words later, there's no erasing the choice that trapped you. It optimizes each step, not the sentence.

So why not just try every sentence? Look at the count.

|V|^{T}\quad\text{(e.g. } 50000^{20}\approx 10^{94}\text{ sequences)}

If greedy is too myopic, why not search all sentences and keep the best? Because the count explodes. Every slot can hold any of tens of thousands of words, so the candidates multiply as the vocabulary raised to the length. Like a library of every possible book: a single twenty-word reply already has more versions than there are atoms in the observable universe. You can't check them all.

Beam search: keep the best few alive, not just one.

s(y_{1:t})=\sum_{\tau=1}^{t}\log p(y_\tau\mid y_{<\tau},x),\qquad \mathcal{B}_t=\text{top-}B\ \text{of}\ \mathcal{B}_{t-1}\times V

The middle path keeps the top B partial sentences alive at every step. Extend each by every possible next word, score the lot, then prune back to the best B — and repeat. Like thinning seedlings: plant many, and at each stage pull all but the strongest few, letting the survivors grow on. Widen the beam for more foresight; B = 1 is just greedy again.

One catch: it secretly prefers short sentences.

s_{\text{norm}}(y)=\frac{1}{T^{\alpha}}\sum_{t=1}^{T}\log p(y_t\mid y_{<t},x),\qquad 0\le\alpha\le 1

Every extra word multiplies in another probability below one, so longer sentences always score lower — left raw, beam search clips its answers short. The fix is to divide the total by the length, judging the average word, not the sum. Like scoring a long routine: more moves means more chances to wobble, so you grade per move, not by counting total slips. Now a long, honest sentence competes fairly.

The honest catch: likeliest isn't best.

\hat{y}=\arg\max_{y}\ p(y\mid x)\quad\text{vs}\quad y\sim p(\,\cdot\mid x)

Push search hard enough and a surprise appears — the single most probable sentence is often flat, generic, even looping. Probable and good are different axes. Like the safest beige: the colour that offends no one is the one no one remembers. So precise tasks search for the likeliest, while open, creative ones sample a surprise instead — two ways to read the same odds.

🌱 Is the most probable sentence ever the best one?

Search hunts the likeliest words; the likeliest words often make the dullest reply. The safest path and the best path are not the same path — and a model that always walks the likeliest one may be the least likely to say anything you'll remember. If the most probable sentence is the most forgettable, what exactly were we maximizing?