Build a reader, not a recaller
A language model you can make with index cards. The first page of a guidebook that goes from a bigram table on the kitchen table to a working reader, with no commercial AI anywhere in the chain.
Here is the shortest honest description of a language model: a table of what tends to come after what, and a way of drawing from it. Everything else is engineering to make the table bigger and the drawing cleverer.
Most people never see the table. They meet the finished product, which talks, and they reach for the only word that fits a thing that talks, which is mind. That’s where the trouble starts, and it’s why I’m writing a guidebook that begins with the table. Not a metaphor for it; the actual thing, built by hand, with the mechanism in full view at every step.
The progression runs in four stages, pitched at four ages, though adults are welcome at all of them:
- By hand, age eight and up. Three sentences, a pile of index cards, one pile per word. Scan the text, drop a card for each pair. Then draw: pick the pile for the last word, draw a card at random, write it down, repeat. You’ve built and run a bigram model. It produces sentences that are not in the source, and never could have been, and were built entirely from it. That last fact is the one worth sitting with.
- Scratch or Python, ten to twelve. The same thing, as code you wrote.
- A small fine-tune, twelve to fourteen. A 360-million-parameter open model, nudged toward a corpus of your choosing, so you see what training does to a thing that already works.
- The full pipeline, sixteen and up. A reader that answers from what it has in front of it and says so, with a gate that refuses to go beyond its grounds.
No stage calls a commercial API. The point is to own every step, so that when someone tells you what a model “knows”, you have the table in your head and can ask the right question back.
Start here
Stage 1, Part 1: building the model is the first interactive page. It does the card exercise on screen: the source text, the current pair being extracted, the piles filling up, and then generation, drawing one card at a time with the random draw shown. Press Play and watch it. Then press Generate until you’ve seen it say something the source never said.
More parts follow, and the guidebook with them. Its working title is the title of this page, and I’m still looking for the right Irish one.