Learning from Prediction 1: N-gram Language Models

<p class="kicker">GRA 4164 · Lecture 4 · Fall 2026</p>

## Learning from Prediction 1:<br>N-gram Language Models

### Leif Anders Thorsrud & Vegard H. Larsen — BI Norwegian Business School

---

Predict the **next token** — the idea behind every modern language model.

Today: the smallest possible version, one character at a time.

---

## A new assumption

<div class="cols">
<div class="callout">
<h3>Lectures 2–3</h3>
<p>Bag-of-words: order doesn't matter. A document is a pile of counts.</p>
</div>
<div class="callout">
<h3>Today</h3>
<p>Language models: order is the whole point. Text is a sequence.</p>
</div>
</div>

---

## What a language model is

---

## A probability over sequences

$$P(w\_1, w\_2, \ldots, w\_T) = \prod\_{t=1}^{T} P(w\_t \mid w\_1, \ldots, w\_{t-1})$$

- The **chain rule** — exact, no assumptions yet
- Each factor: *given everything so far, what comes next?*

---

## The Markov shortcut

$$P(w\_t \mid w\_1, \ldots, w\_{t-1}) \;\approx\; P(w\_t \mid w\_{t-n+1}, \ldots, w\_{t-1})$$

- **N-gram model**: only the last $n-1$ tokens matter
- $n=2$: the **bigram** model — predict from one previous token

---

## Character-level, on purpose

- Vocabulary: **27 tokens** — a–z plus `.` for start/end
- `"hello"` → `.→h`, `h→e`, `e→l`, `l→l`, `l→o`, `o→.`
- Small enough to **see every number** — same idea as GPT

---

## Estimating it: just count

$$P(c\_n \mid c\_{n-1}) = \frac{\text{count}(c\_{n-1}, c\_n)}{\text{count}(c\_{n-1})}$$

Corpus `"hello hello"`:   $P(e \mid h) = \dfrac{2}{2} = 1$

---

## Model 1: counting

---

## The data

<div class="cols cols-3">
<div class="stat stat--blue">
<span class="stat-value">32,033</span>
<span class="stat-label">names</span>
</div>
<div class="stat">
<span class="stat-value">27</span>
<span class="stat-label">token vocabulary</span>
</div>
<div class="stat stat--green">
<span class="stat-value">228k</span>
<span class="stat-label">character transitions</span>
</div>
</div>

Most common bigrams: `n→.` (6,763), `a→.` (6,640), `a→n` (5,438).

---

## The whole model in one picture

![27 by 27 heatmap of bigram transition probabilities](../figures/bigram-heatmap.svg)

---

## Sampling from it

- Random baseline: `nzheqtuvbputkx` — garbage, as expected
- Bigram model: `alan` · `mea` · `lia` · `dan` · `koukeliphes`
- Some are real names. Some are… **almost** names

---

## How good is it? One number

$$\text{loss} = -\frac{1}{n}\sum \log P(c\_{t} \mid c\_{t-1})$$

- Average **negative log likelihood** of the data — lower is better
- The counting model scores **2.45**

---

## Model 2: learning

---

## The same model, as a network

- Input: one-hot vector of the current character (27 dims)
- One weight matrix $W$ (27 × 27), then **softmax** → probabilities
- Train by **gradient descent** on the same loss

```python
logits = x_onehot @ W
probs  = logits.exp() / logits.exp().sum(1, keepdim=True)  # softmax
loss   = -probs[range(n), ys].log().mean()
```

---

## What does it learn?

<div class="cols">
<div class="stat stat--blue">
<span class="stat-value">2.45</span>
<span class="stat-label">counting</span>
</div>
<div class="stat">
<span class="stat-value">2.46</span>
<span class="stat-label">neural net, trained</span>
</div>
</div>

---

Gradient descent **rediscovers** the counting solution.

So why bother with the network?

---

## Model 3: more context, more layers

---

## The MLP recipe

- **Embed** each character: 27 one-hots → 10-dim vectors
- **Context of 5**: concatenate embeddings → 50 numbers
- **Hidden layer** of 300 (tanh) → softmax over 27
- **23,697 parameters** · mini-batches · train/dev/test split

---

## Same task, three models

![Loss comparison: counting 2.45, neural net 2.46, MLP 2.17](../figures/loss-ladder.svg)

---

From here to GPT is **scale**, not magic.

23,697 parameters → 175 billion. Same loss function.

---

## In sum

- A language model assigns **probability to sequences** — chain rule + Markov
- Bigram counting, a **27×27 table**, already generates names
- A network trained by gradient descent **matches it** — and then **scales past it**

</div>

---

## Questions

## Smoothing

- Problem: unseen bigram → $P = 0$ → log likelihood $= -\infty$
- **Laplace**: $P(c\_n \mid c\_{n-1}) = \dfrac{\text{count} + 1}{\text{count}(c\_{n-1}) + 27}$
- One pseudo-count per pair; barely moves the loss (2.454 → 2.455)

## The three models, side by side

| | Counting | Neural net | MLP |
|---|---|---|---|
| Context | 1 char | 1 char | 5 chars |
| Parameters | 0 learned | 729 | 23,697 |
| Training | instant | ~1k steps | 200k mini-batches |
| Loss | 2.45 | 2.46 | 2.17 |

## Notation cheat-sheet

- `stoi` / `itos` — character ↔ index lookup tables
- `N` — 27×27 count matrix; `P` — row-normalised probabilities
- `torch.multinomial(P[i], 1)` — sample the next character
- `F.cross_entropy(logits, ys)` — the loss, in one call

---

## What's next

- **Lecture 5 (Mon):** practical day 1 — bring code, we work on the assignments
- **Lecture 6:** word embeddings — the `C` matrix from part 3 becomes the star
- **Before then:** run the three notebooks; watch **Karpathy's "building makemore"**