How a small model drafts and a big one just checks — same words, far faster. · Source

The big model writes one word at a time. What if it didn't have to?

A giant model is slow for one reason: it writes one word at a time, each a full trip through billions of parameters. Like a master inking over a quick pencil sketch: let a fast little hand rough in the next few words, and the master just confirms the good strokes and fixes the rest. Same finished line — drawn in a fraction of the time.

Why so slow? Each word must wait for the one before it.

\text{latency}\approx n\times t_{\text{pass}}

The giant can't run ahead of itself. To write word ten it needs word nine, and word nine needed word eight — a single thread, no skipping. Like laying a plank footbridge: you can only nail the next plank while standing on the last one. So n words means n full trips through the giant, strictly one after another — and the chip spends most of each trip just waiting.

First move: let a small fast model guess the next few words.

Bring in a featherweight model — far smaller, far faster, often right. Let it guess the next handful of words in a blink. Like a breakaway cyclist: a light rider sprints out ahead of the pack and calls the road back. The guesses are cheap and sometimes wrong — that's fine. They're only a proposal, waiting to be checked.

The trick: the giant checks all the guesses in one pass.

T_{\text{check}}(\gamma{+}1)\approx T_{\text{pass}}(1)

Here's what makes it pay. A single pass of the giant can score a whole row of positions at once — the γ guesses plus its own next word — for almost the price of scoring one. Like a lifeguard on the tower: one sweep of the eyes takes in the whole crowded beach. Why nearly free? The slow part is hauling the giant's weights onto the chip, not the arithmetic — so a few extra words ride along for almost nothing.

The rule: keep each guess while the giant agrees.

\text{keep } x\sim q \ \text{ with probability } \ \min\!\left(1,\ \frac{p(x)}{q(x)}\right)

Walk the guesses left to right. Let p be the giant's probability for a word and q the draft's. Keep the guess outright when the giant likes it at least as much (p ≥ q); when the giant likes it less, keep it only p/q of the time. The first guess that fails — stop, throw out the rest. Like a zipper: the teeth mesh smoothly one by one, then jam dead at the first tooth out of line.

The magic: a rejected guess is corrected, not just dropped.

p'(x)=\dfrac{\big(p(x)-q(x)\big)_+}{\sum_{x'}\big(p(x')-q(x')\big)_+},\qquad \min\big(p,q\big)+\big(p-q\big)_+=p

When a guess is rejected, the giant doesn't stall — it samples a replacement from exactly the gap between what it wanted and what the draft offered. Add that correction back and a small miracle falls out: every word emitted is distributed exactly as the giant alone would have produced. Like water in a U-tube: top up the low side and the level settles to precisely the true mark. Faster, yet provably lossless — not an approximation.

The payoff: several words per single pass of the giant.

\mathbb{E}[\#\text{tokens}]=\frac{1-\alpha^{\,\gamma+1}}{1-\alpha}

Count the winnings. If the draft proposes γ words and is right a fraction α of the time, one giant pass yields, on average, this many words. When the little model is usually right, you collect close to γ+1 words per check; when it's usually wrong, you simply fall back to one — never worse. Like skimming a flat stone: a single throw touches down again and again. One pass, several words.

🌱 Two minds drew the path. Where did the deciding happen?

The fast guesser and the slow giant, together, produce exactly the words the giant would have written alone. So which one was thinking? Like two trails merging into one: the path you walk is single and sure — yet two of you laid it down. If a quick guess and a careful nod give the very same answer as the careful mind alone, where in all that was the choosing — in the leap, or only in the check?