How a giant model learns your task by training almost nothing. · Source

Adapt a giant model by training a sliver you could email.

Full fine-tuning bends every weight in a model that has billions of them — and stores a whole fresh copy for each new task. You can't keep a dozen of those around. LoRA freezes the giant and trains a tiny add-on instead: small enough to email, swap, and stack. Same model underneath, a brand-new skill, almost nothing saved.

The old way retrains all of it — a fresh copy per task.

W_{\text{FT}} = W_0 + \Delta W, \quad \Delta W \in \mathbb{R}^{d \times k}

To specialize a giant, classic fine-tuning learns a change ΔW exactly the same size as the original weights — every one of its d×k numbers free to move. Do that for ten tasks and you store ten whole models. Like reprinting the entire atlas to redraw a single street: the change is tiny, yet you remake the whole book.

First move: don't touch the giant at all.

h = W_0 x + \Delta W x, \quad W_0 \;\text{frozen}

LoRA leaves the pretrained weights frozen — not a single one learns. The output stays the giant's own answer, plus a small, separate correction that you do train. Like a clear acetate sheet laid over a blueprint: the drawing underneath never changes; you ink your edits onto the overlay above it.

Second move: make that correction simple.

\Delta W = B A, \quad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ \ r \ll \min(d,k)

Here's the trick: force the change to be low-rank. Instead of a full d×k block, write ΔW = B·A — two skinny matrices sharing a narrow waist of width r, far smaller than either side. Like a marionette: just a few strings move the whole sprawling figure. The big update really has only a handful of independent pulls.

Now count what you actually train.

r(d+k) \;\ll\; d\,k

A full update is d×k numbers. Two skinny matrices are only r(d+k). For a layer 12288 wide with r = 8, that's hundreds of times fewer — a model's worth of tuning shrunk to a sliver you can store by the dozen. Like a tailor's few nips and tucks: nobody sews a new suit, a handful of stitches makes the off-the-rack one yours.

Start it adding nothing, then ease in.

h = W_0 x + \frac{\alpha}{r} B A x, \qquad B = 0,\ \ A \sim \mathcal{N}(0,\sigma^2)\ \ \text{(at init)}

How do you begin safely? Set B = 0 and fill A with small random noise, so the add-on starts at exactly zero — on step one you behave as the untouched giant. Then training eases the correction in, scaled gently by α/r. Like stage lights swelling up from black: nothing is added at first, then the new scene arrives slowly, never with a jolt.

When it runs, the add-on disappears.

W = W_0 + \frac{\alpha}{r} B A

Once trained, fold the two skinny matrices back into the big one. The model is a single matrix again — zero extra work at run time, no slower than the original. And since each skill is just its own small B·A, one frozen giant can wear many of them. Like one power drill with a tray of bits: keep the motor, click in a different tip for each job.

🌱 If the new skill is so small, who already knew it?

The change that turns a general giant into your specialist is astonishingly small — a few thin strips of numbers. Which leaves a quiet question: did you add the new skill, or did the giant already hold it, waiting for a few strings to pull it into the open? Maybe fine-tuning doesn't teach — it reveals.