Learning from prediction 1: n-gram language models
N-grams, character-level language models, and the fundamentals of next-token prediction
Open slidesOverview
Predicting the next token is the idea behind every modern language model — this lecture builds the smallest possible version of it, one character at a time. From 32,033 names we count a 27×27 bigram table that can already generate new names; then a one-layer neural network trained by gradient descent rediscovers the same solution (loss 2.46 vs 2.45); then an MLP with embeddings and a five-character context beats both (2.17). From there to GPT is scale, not magic — the loss function never changes.
Topics
- The chain rule, the Markov assumption, and n-gram models
- Counting a character-level bigram model: estimation, sampling, smoothing
- Evaluating with negative log likelihood — the loss every LLM minimises
- The same model as a neural network; why the learning framework scales
- The MLP upgrade: embeddings, longer context, hidden layer
Readings
- J&M chapters 3 and 7
Asynchronous work
- Watch Andrej Karpathy’s The spelled-out intro to language modeling: building makemore — the notebooks follow it closely
Materials
- Slides: use the Open slides button above
- Notebooks — download all three plus the data file, keep names.txt in a
data/folder next to them:- Predict_the_next_token_part1.ipynb — the count-based bigram model
- Predict_the_next_token_part2_nn.ipynb — the same model learned by a neural network
- Predict_the_next_token_part3_mpl.ipynb — the MLP with embeddings and a five-character context
Before lecture 5
- Run the three notebooks — practical day 1 builds on them
- Watch the Karpathy video if you haven’t