GlassBox: a transformer you can watch work
A 1,352-parameter transformer that trains in your browser and shows every number it has. Why I built it, what to do first at glassformer.com, and the one thing it is not.
Every explanation of a transformer I’ve read has the same shape. There’s a diagram with boxes labelled attention and feed-forward, there’s a paragraph about queries, keys and values, and then there’s a leap: “and so the model learns to predict the next token.” The leap is where the understanding was supposed to happen, and it never does, because nothing in the diagram moves.
GlassBox exists to make it move. It’s a deliberately tiny transformer, 48 words of vocabulary, eight-token context, an embedding of eight dimensions, 1,352 parameters in its default shape, written in plain NumPy with the backpropagation done by hand, one line per forward operation. A fresh model is untrained, so its first predictions are nonsense. You train it, in the browser, on a laptop CPU, in about a second, and watch nonsense become “the cat sat on the → mat”. Nothing is hidden. Every forward pass emits exactly twenty-four trace events, one per stage from tokenize to selected token, and the interface draws only from that trace; it does no maths of its own.
What to do first
Open glassformer.com. It lands on the Training tab for a reason: train ten epochs and look at the loss curve against the uniform-guess baseline before you do anything else. Then run a prompt and press Step Through. Walk the fourteen groups of the pipeline with the arrow keys. At the softmax stage, click one cell of the attention heatmap and read the calculation inspector: the query and key vectors, each dimension’s product with its bar, the dot product, the scaling, the shift, the row sum, the weight. That inspector is the whole point of the project. Attention is not a metaphor; it is that arithmetic, and now you’ve seen it.
Then open the 3D view, which is the Inside a language model sketch on this site rebuilt from a live trace, so every cell and bar is a real number from the run you just made.
When you want more than a demonstration
The Lab tab lets you change the architecture: one to four layers, heads, layer norm or RMS norm, cosine attention, sparsemax, sinusoidal or rotary positions, GELU or SiLU or a formula of your own, gated by a gradient check so a bad formula can’t be built. The Experiments tab runs a baseline against a variant over matched seeds and gives you a paired 95% interval and a verdict on a metric you chose in advance. Compare puts two models side by side with KL both ways. A logit lens shows what each layer already believes before the next one touches it. These are not toys; they are the tools you’d use to ask whether an idea about transformers is true, at a scale where the answer comes back in seconds.
What it is not
It isn’t a frontier model and it makes no claim about what happens inside one. It’s the mechanism, at a size where the mechanism is visible, and I hold the unfashionable view that the next real breakthroughs in this field are still sitting in the mathematics GlassBox lets you see, not in anything that needs a data centre.
If you’d rather build the mechanism by hand first, the Grounded Reader pages do the bigram version with cards on a table before any of this machinery arrives.