Inside a tiny language model

A 122k-parameter nanoGPT trained from scratch on word tokens, shown three ways: the map its embeddings form, the actual arithmetic one prompt goes through, and how both change over 3,000 training steps. Every number on this page was read from the saved runs — nothing is illustrative.

Talk to it

loading the weights…

The map on the left is the model's own vocabulary: two words are tied together when each is among the other's most similar vectors, measured across all 64 dimensions. Nothing labelled those groups — the model only ever read sentences and was only ever asked for the next word. Type on the right and the map turns into an explanation: the words of your prompt light up, dashed lines show which of them the last token actually attended to, and solid rays fan out to the words the model is considering next, thick in proportion to their probability. The word it picks is the one the ray leads to.