How a model learns to see by reading. · Source

It learns to see by reading — and you never hand it a label.

Show it a photo and the caption that happened to come with it — then do that a billion times. Slowly it ties what it sees to what we say, until pictures and words live in one shared space. Ask in words, it finds the picture; show a picture, it finds the words. Nobody labeled anything — the captions the world already wrote were the whole lesson.

The old way: shove every picture into one of a fixed set of slots.

To teach a model to see, we used to hand-label millions of pictures against a closed list — a thousand fixed boxes and not one more. Slow, costly, and blind to anything off the list. Like a tray with a fixed set of slots: whatever doesn't match a slot has nowhere to land. But the open web already pairs pictures with the words sitting beside them — billions of captions, already written, free. So stop labeling. Start pairing.

Two separate readers, pixels and words, aimed at one shared space.

I = \dfrac{f_{\text{img}}(x)}{\lVert f_{\text{img}}(x)\rVert},\quad T = \dfrac{f_{\text{txt}}(t)}{\lVert f_{\text{txt}}(t)\rVert},\quad I,\,T \in \mathbb{R}^{d}

A picture and a sentence are nothing alike — so the model runs each through its own reader, and each emits a point in one shared space, every point stretched to the same length so only its direction means anything. Train the two readers together until a matched picture and caption point the same way. Like two tuning forks struck to one note: strike one and its twin across the room hums on its own — different bodies, one shared frequency. The lesson isn't a label. It's agreement.

Pull each true pair together. Shove every wrong pair apart.

\ell_i = -\log \dfrac{\exp(\langle I_i,\,T_i\rangle/\tau)}{\sum_{j=1}^{N}\exp(\langle I_i,\,T_j\rangle/\tau)}

A batch holds a thousand picture–caption pairs: a thousand right matches, nearly a million wrong ones. The rule is blunt — for each picture, make its own caption score highest, and push the rest down. Like a heap of two-piece puzzles tipped out together: each picture has exactly one caption that interlocks; tighten every true mate, pry the impostors loose. The formula just says: top = the true pair's score, bottom = every candidate summed, and τ sets how sharply the contest is judged.

Closeness is just direction — and it's judged both ways.

\operatorname{sim}(u,v)=\dfrac{u\cdot v}{\lVert u\rVert\,\lVert v\rVert},\qquad \mathcal{L}=\tfrac{1}{2}\left(\mathcal{L}_{\text{img}\to\text{txt}}+\mathcal{L}_{\text{txt}\to\text{img}}\right)

How does it score 'closest'? By direction, not distance — the cosine of the angle between two points. Same heading, high score; at right angles, nothing. Like two weathervanes on a roof: what matters isn't how big each arrow is, only whether they swing to face the same way. And the contest runs both directions at once — each picture hunts its caption, each caption hunts its picture — so the loss is the average of the two.

Now it can name a thing it was never trained to see.

\hat{y}=\arg\max_{c}\ \operatorname{sim}\big(f_{\text{img}}(x),\,f_{\text{txt}}(\text{prompt}_c)\big)

Here's the gift of one shared space: to sort a picture you need no labeled examples at all. Just describe each option in plain words, drop those descriptions into the same space, and pick whichever one sits nearest the picture. Like birding with a field guide: you read a description of a bird you've never once seen, then spot the one in the tree that fits it best. A new category is just a new sentence — no retraining, no new labels.

One shared map for pixels and words — the seed of multimodal AI.

Put it together and you've built one coordinate system where a picture and its words sit side by side. Search a photo library by typing what you want. Caption an image by reading off its nearest words. And — quietly — this is the joint that text-to-image generators turn on: a text reader that already speaks the language of pictures, steering what gets drawn. Like two oxen under one yoke: two very different animals, joined at a single beam, finally pulling as one.

They land on one point. Do they share one meaning?

A photograph of a dog and the word for that dog land on the very same spot. So it understands? Or did it only learn that the two tend to arrive together — the way thunder follows lightning with no idea what a storm is? Sharing a coordinate is not the same as sharing a meaning. We taught it that picture and word belong side by side. We never taught it what either one truly is. 🌱