How Language AI Reads Text
A language model never reads a word. It chops text into numbered pieces, turns each number into a point in space, lets the points pull on each other, and then guesses one next piece. Again and again.
Step 01 of 08
1 · Text in, one word out
Autocomplete, translators and chatbots all run on a loop like this box. A sentence feeds in on the left: “The cat didn’t eat the fish because it was”. One word lights up on the right: stale. Nothing in between reads. Here is what it does instead.
Step 02 of 08
2 · Chop it into numbered pieces
First, a cut. The tokenizer slices the text into pieces from a fixed menu of 50,257: whole common words, or fragments of rarer ones. Watch “didn’t” split into didn and ’t. Then every piece is swapped for its number on the menu: fish is 5916, it is 340. From here on the machine only ever sees whole numbers.
Step 03 of 08
3 · Each number pulls out a row
The number is an address. It picks one row out of a table 50,257 rows tall, and every row holds 768 numbers learned in training. That row becomes the token’s card: each coloured cell is one of its 768 numbers. The whole table is 38.6 million numbers, and the machine hasn’t read anything yet.
Step 04 of 08
4 · Meaning is a place
Read those 768 numbers as coordinates and every token becomes a point in space. Training nudges the points until words used the same way sit together: animals here, spoiled food there. Even directions carry meaning. In classic word vectors, the step from man to king is the same step that takes woman to queen.
Step 05 of 08
5 · Every token looks back
A point on its own can’t tell which “it” you mean. So each token scores itself and every earlier token for relevance, and the scores always add up to 100%. Here “it” spends most of its attention on the two nouns it could stand for: fish and cat. Its card then blends in theirs. “was” comes later, so it is masked out.
Step 06 of 08
6 · Twelve times over
One round of attention, plus a small network that reworks each card, makes a layer. GPT-2 small stacks twelve. Every card rises through all of them, and each layer mixes in more context and reshapes the numbers. By the top, the card for “was” has soaked up the cat, the fish and the refusal.
Step 07 of 08
7 · Score, pick, append
Only the last card matters now. It is scored against every row of the same 50,257-row table, and the scores become odds. “stale” comes out on top, but “too”, “full” and “rotten” are all in the running. One piece is picked, dropped onto the end of the text, and the whole machine runs again for the next.
Step 08 of 08
8 · Predict one piece. Append. Repeat.
That is the whole trick, run forever: one piece at a time, each fed back in as input. A long chatbot reply is this loop run a few hundred times. The understanding you feel is geometry: points that sit close together, and tokens that pull on each other.