Introduction to LLMs and Tokenization

<p class="kicker">GRA 4164 · Lecture 7 · Fall 2026</p>

## Introduction to LLMs<br>and Tokenization

### Leif Anders Thorsrud & Vegard H. Larsen — BI Norwegian Business School

---

You've already built a language model. Now make it **large**.

Same loss function as lecture 4 — everything else is scale.

---

## The plan for the LLM block

- **Today:** what LLMs are · what goes in (**tokenization**)
- **Lecture 8:** the architecture (**attention, transformers**)
- **Lecture 9:** how they're trained and **fine-tuned**
- **Lecture 11:** how to use them well (**prompting, alignment**)

---

## What LLMs are

---

## An LLM, in a nutshell

- Concretely: **two files** — the weights, and code to run them
- A **transformer** network trained to predict the **next token**
- Training **compresses** a huge text corpus into the weights
- Predicting well forces the model to **encode facts** about the world

---

## Input, output, learning

- **In:** a sequence of tokens
- **Out:** a probability for every possible next token
- *"What comes next in this ___"* → `course` 20% · `class` 30% · …
- **Self-supervised**: the next token is the label — data labels itself

---

## From raw model to assistant

- **Pre-training** — next-token prediction on a massive corpus
- **Mid-training** — conversation format, special tokens, tool use
- **Supervised fine-tuning** — curated instruction–response pairs
- **Reinforcement learning** — optimise against a reward model

Today: stage one's input. Lecture 9: the whole pipeline.

---

## Why "large"?

<div class="cols cols-3">
<div class="stat stat--blue">
<span class="stat-value">10¹³</span>
<span class="stat-label">tokens of training text</span>
</div>
<div class="stat">
<span class="stat-value">10¹¹</span>
<span class="stat-label">parameters</span>
</div>
<div class="stat stat--red">
<span class="stat-value">10⁷+</span>
<span class="stat-label">GPU-hours to train</span>
</div>
</div>

<p class="muted">Orders of magnitude — exact numbers age fast. Training is industrial; running a small model is not.</p>

---

## What they're used for

**Today**   `customer support` `drafting & content` `code` `translation` `summarisation`

**Emerging**   `education` `healthcare` `research` `agents & workflows`

</div>

---

## Should we trust them?

---

## Four open problems

- **Reliability** — hallucinations, no sources, sensitive to phrasing
- **Bias** — models reproduce what the corpus contains
- **Privacy** — training data exposure, logged queries, GDPR
- **Concentration** — few firms, mostly US, enormous cost of entry

---

**Treat LLM output as a starting point, never a definitive answer.** Verify anything that matters against a source you trust.

</div>

---

## Tokenization

---

LLMs don't see words. They see **tokens**.

Half of LLM weirdness is explained right here.

---

## Tokenizer weirdness, explained

- Can't **spell** or count letters — it never sees letters
- Bad at **arithmetic** — `12345` is chunks, not digits
- Worse in **Norwegian** — fewer, clumsier merges than English
- Early models bad at **code** — whitespace tokenized terribly

---

## First: how is text even bytes?

```python
ord('a')                      # 97
ord('👍')                     # 128077  (Unicode code point)
list('å'.encode('utf-8'))     # [195, 165]   — 2 bytes
list('👍'.encode('utf-8'))    # [240, 159, 145, 141]  — 4 bytes
```

- **Unicode** numbers every character; **UTF-8** stores them in 1–4 bytes

---

## The old answer: stemming

```python
PorterStemmer().stem("running")   # 'run'     ✓
PorterStemmer().stem("ran")       # 'ran'     ✗  irregular
PorterStemmer().stem("easily")    # 'easili'  ✗  not a word
```

- Hand-written rules normalising words — fine for **search**, lossy for **generation**

---

## The modern answer: byte-pair encoding

- Start from **bytes**; count adjacent **pairs**
- **Merge** the most frequent pair into a new token
- **Repeat** until the vocabulary is big enough
- `low, lower, newest, widest` → `low · er · new · est · wide`

Data decides the vocabulary — no linguist required.

---

## What the model actually sees

![Real BPE token splits for English, Norwegian, arithmetic, and spelling examples](../figures/tokenization-examples.svg)

---

## Why a business student should care

- **Cost** — API pricing is per token; Norwegian costs ~30% more than English here
- **Context** — model limits are in tokens, not pages
- **Quality** — if it tokenizes badly, it reasons badly: know when not to trust

---

## In sum

- An LLM is a **transformer predicting the next token** — lecture 4 at scale
- Capability comes with **open problems**: reliability, bias, privacy, concentration
- **Tokenization** is the lens: bytes → BPE merges → tokens — and it explains the weirdness

</div>

---

## Questions

## BPE by hand

Corpus: `low lower newest widest` — start from characters:

1. Most frequent pair `e+s` → merge to `es`
2. `es+t` → `est`   3. `l+o` → `lo`   4. `lo+w` → `low` …
3. Vocabulary grows: `est`, `low`, `er`, `new`, `wide` emerge as units

## UTF-8, the details

| Character | Code point | UTF-8 bytes |
|---|---|---|
| `a` | 97 | `[97]` |
| `å` | 229 | `[195, 165]` |
| `👍` | 128077 | `[240, 159, 145, 141]` |

- ASCII unchanged in 1 byte; everything else 2–4 bytes — **backwards compatible**

## Running models yourself

- **Hugging Face** — open-weight models, datasets, the `transformers` library
- **Ollama** — one-command local models
- The **course server** hosts open-weight models for the assignments

---

## What's next

- **Lecture 8 (Tue):** attention and the transformer — how token sequences become predictions
- **Before then:** run the tokenizer notebook; play with **tiktokenizer.vercel.app**
- Async: the **Hugging Face** intro tutorial · Reading: J&M ch. 2 and 7