THE BIG 4
Four matrices of random numbers, a trillion corrections, and a working language model at the end — built here, from nothing, with no install and no account.
A pile of random fractions gets shown nearly everything humans have written.
It guesses wrong a trillion times.
Each wrong guess nudges the numbers a hair closer to right.
Slowly, without anyone telling it what a word means, the randomness organises itself into geometry — and the geometry turns out to be meaning.
The strange part isn't that it works.
The strange part is that nudging numbers toward better guesses was enough.
A matrix of random numbers
Picture a matrix with sixty-five thousand rows and four thousand columns. Every cell holds a number. Every one of those numbers was just picked at random.
This is the moment a model switches on. It isn't smart. It isn't dumb. It's balanced — like a coin that hasn't been flipped.
The row that will one day mean apple holds 4,096 meaningless fractions. The row that will mean monitored holds 4,096 different meaningless fractions. Neither knows a thing.
Those 4,096 numbers are coordinates — like latitude and longitude, except in a space with 4,096 directions instead of two. Words that sit near each other in that space mean similar things. That's the whole idea the model is eventually going to discover.
Which is what makes day one so bleak. 65,536 × 4,096 = 268,435,456 — 268 million numbers in the lookup table alone, and not one of them is anywhere near where it belongs. Apple is currently parked next to monitored. The map is nonsense.
Before it wakes up, language gets chopped
Models don't read letters, and not quite words either. They read tokens — frequent chunks of text found by scanning enormous piles of writing.
Every chunk gets a number. That's the whole vocabulary: usually 30,000–100,000 entries.
And notice what these are right now — empty labels. Token 8901 doesn't mean speed. It's a barcode. A locker number. Nothing is in the locker yet.
Give every token 4,096 coordinates
Rows are tokens. Columns are dimensions — abstract directions of meaning, thousands of them.
Type the word pace and the model does something almost disappointing: it goes to row 8901 and pulls out that row's numbers. That list is the word, as far as the machine is concerned.
Feed in a whole sentence and it does that eight times over and stacks the results. That rectangle — 8 × 4,096 — is the raw material handed to Layer 1.
Guess. Be wrong. Nudge. Repeat.
One game, played across essentially everything humans have written down: cover the next word and guess it.
The newborn model does a mountain of arithmetic and confidently blurts out banana. And the training algorithm says the most important sentence in machine learning: "Wrong. It was 'pace.'"
Then it traces exactly how wrong, backward through every number that contributed. That's backpropagation — less like a teacher marking an exam, more like a river carving a canyon.
Every number nudges a hair. One nudge is noise. Do it a trillion times.
Meaning is a location
After enough rounds something genuinely beautiful happens, and nobody programmed it. Structure appears on its own.
Pace, speed, velocity, tempo keep showing up near running, driving, fast, slow — so the nudges keep pushing them the same way. Eventually they cluster like stars.
Elsewhere, his, her, their drift into their own huddle. Somewhere else again: monitored, watched, tracked.
The model never saw a dictionary. It just noticed, a trillion times over, the company each word keeps. That turned out to be enough.
Same row. Two meanings.
Here's what the spreadsheet cannot solve.
Both sentences use the token pace. Same locker, same row 8901, the identical 4,096 numbers pulled out both times. But those are two different meanings, and any decent reader knows it instantly.
So the frozen dictionary definition can only ever be raw material. Something else has to bend it into what the word means right here, right now, in this sentence.
A crowded party, all shouting at once
Every word at the party holds three things: a Query (a question it shouts), a Key (a name tag on its chest), and a Value (what it hands over if you go talk to it).
Words don't invent those. They manufacture them, by running their 4,096 numbers through three learned grids — W_Q, W_K, W_V — 4,096 × 4,096 each, about 16.8 million numbers apiece.
One rule of matrix multiplication, and it's easier than it looks: the inner numbers must match, and they cancel.
softmax pi = ezi ⁄ Σj ezjQKT scores every query against every key. Dividing by √dk keeps those scores from growing with width, which would drive the softmax to a hard one-hot and kill the gradient. Softmax turns the row into weights that sum to 1; multiplying by V collects the values in those proportions.
Eight tokens. Each builds its own Q, K and V.
"his" shouts who do I belong to? — and "coach" is wearing the matching name tag.
The 8 × 8 attention map. Darker = listening harder.
4,096 splits into 32 heads × 128. Thirty-two conversations at once — one tracks grammar, one tracks who-owns-what, one tracks the topic.
The rectangle never changes size
Layer 2 doesn't get the original dictionary words. It gets Layer 1's upgraded versions, and asks brand-new questions of them.
After Layer 1, his no longer means a generic pronoun — it means "the coach's." And pace means "marathon pace."
So when monitored reaches for pace in Layer 2, it isn't grabbing a generic speed word. It's grabbing one that already has marathon baked inside.
Because the output shape matches the input shape, you can add the original straight back on top (the residual connection) and hand it upward. Forty layers of this — all stackable, because the rectangle never changes size.
Powerful, and completely unemployable
At the end of pre-training you have a machine that is spectacular at predicting text and has no idea it's meant to help anyone.
Ask it a question and it might just generate five more questions — because that's what a list of questions looks like on the internet.
So there's a final phase. It's shown examples of good, honest, useful answers, and human feedback steers it toward being an assistant instead of an autocomplete engine.
Two ways to do it. Pick one →
Both get you an assistant. One costs a fortune.
By the time it reaches you, the ink is dry
When you actually talk to a language model, the spreadsheet is frozen. Completely read-only.
It looks up the exact same 4,096 numbers for pace every single time anyone, anywhere, types that word.
Think of a printed dictionary. Somewhere, over months, in a building drawing more power than a small town, the publisher set the type and ran the press. Nothing you say edits a single definition.
But you can still read those frozen definitions and combine them into something nobody has ever written before. And when the conversation ends, that structure is gone. The thought dissolves. The dictionary sits there, unchanged, waiting.
Everything that happens, in four moves
Everything in the last ten chapters is one loop, run an enormous number of times. Here it is end to end, at speed.
Text goes in, chopped into tokens and looked up as rows of numbers.
The matrices reshape it — queries meet keys, softmax turns raw scores into a distribution, and a feed-forward block does the rest of the thinking.
Being wrong is measured, and the error is traced backwards to every number that contributed. On the surface to the right, that is a ball finding its way down a funnel.
Then it stops. The weights lock, and the same machinery runs forward only — which is all inference ever is.
Where the "7B" in a model's name comes from
Not 7 billion words memorised. 7 billion individual numbers — every one of which started as a random fraction and got dragged into position by being wrong a trillion times.
vocabulary 65,536
model width 4,096
layers 32
heads per layer 32 × 128
feed-forward width 16,384