How a model picks up a new task from the prompt — without changing a thing. · Source

It learns a new task from the prompt — and not one weight moves.

Show it a task it was never trained on — a few examples, a new question — and it just does it. No fine-tuning. No gradient steps. Nothing inside it is rewritten. This is in-context learning: the model picks up the job from the prompt alone, in a single forward pass, then lets it go. Teaching, with nothing to install.

Usually learning means changing. Here, nothing changes.

\Delta\theta = 0

Training rewrites a model — gradient descent nudges a billion weights until it's better. In-context learning does none of that: the weights are frozen solid, Δθ = 0. So where does the new skill come from? Like a kaleidoscope: the mirrors and beads never change — give it a turn and a wholly new pattern appears, built only from what was already inside.

You don't tell it the rule. You show it a few examples.

\hat{y} = \arg\max_{y}\ p_\theta\!\left(y \mid (x_1,y_1),\dots,(x_k,y_k),\,x_q\right),\quad \theta\ \text{fixed}

Lay down a handful of pairs — input, then answer — then a fresh input, all as one long prompt. The model just continues the pattern. You never stated the rule; the examples are the instruction. Like an impressionist: give them a few lines in someone's voice and they carry on in it perfectly — nobody wrote down how that voice works. Same next-word engine, newly aimed.

It isn't learning the task. It's finding one it already has.

p(y \mid x, D) = \int p(y \mid x, T)\, p(T \mid D)\, dT

Pretraining already walked it past a thousand tasks like yours. The examples don't teach a new skill — they tell it which skill to run. Formally, its answer averages over every task it might be doing, weighted by how well each fits your demonstrations. Like a janitor's keyring: you already carry every key — the examples just say which one opens this door.

Each example rules tasks out. The first few do the most.

p(T \mid D_{k+1}) \propto p(T \mid D_k)\, p(x_{k+1},y_{k+1} \mid T)

Every new demonstration reweights the field: tasks that predicted it well rise, the rest fade. So more examples sharpen the guess — but with diminishing returns. A handful usually does most of the work, and piling on more barely moves it (and can even confuse it). Like focusing binoculars: the first turns snap the scene sharp; after that, you're barely turning at all.

The wild part: the forward pass may be doing the learning.

\hat{w} = w_0 - \eta\,\nabla_{w}\textstyle\sum_{i}\left(w^\top x_i - y_i\right)^2,\qquad \hat{y}_q = \hat{w}^\top x_q

Strip it to a toy — a simple linear task, simplified attention — and you can read an actual learning step out of the math: one pass fits a throwaway model to your examples, then predicts. Like a card counter: from the few cards already shown, they quietly fit the odds in their head — a whole calculation hidden inside one glance. Proven for the toy case; a leading guess for the real thing.

The prompt is a program. The machine never changes.

No retraining, no new copy, no weights touched — you reprogram it with what you set in front of it, and the next prompt reprograms it again. The whole skill lives in the input. Like a chameleon: the animal itself never changes — set it on a new background and it becomes that color, instantly, then the next.

🌱 It learned for a moment. Then the window closed.

Whatever it picked up lived only in this context. Close the window and it's gone — the model is exactly as it was, holding nothing. So was that learning, or only remembering long enough to answer? And if a skill can appear with the prompt and vanish with it, where was the skill the whole time it slept?