LLM Input/Output, Training, and Fine-tuning

GRA 4164 · Lecture 9 · Fall 2026

## LLM Input/Output, Training, and Fine-tuning

### Leif Anders Thorsrud & Vegard H. Larsen — BI Norwegian Business School

---

You know the architecture. Today: where the **weights** come from.

…and how to make someone else's weights work for **you**.

---

01

## From tokens to the next token

---

## The full stack

![Full LLM pipeline: tokens, embeddings plus positions, transformer blocks, head, next-token distribution, generation loop](../figures/llm-stack.svg)

---

## How much fits? The context window

- $N$ is **finite** — attention cost and memory grow fast with length
- BERT era: **512** tokens · today: **hundreds of thousands**
- Overflow → silent **truncation** or an error: know which your tool does

---

02

## Picking the next token

---

## The distribution is not a decision

- **Greedy** (always argmax) → generic, repetitive text
- **Pure random** over everything → nonsense from the long tail
- **Top-k**: sample within the k most likely — but the right k varies
- **Top-p**: sample within the smallest set covering probability p

---

## Temperature

$$\boldsymbol{y} = \text{softmax}(\boldsymbol{u} / \tau)$$

![The same next-token distribution at temperature 0.5, 1, and 2](../figures/sampling-temperature.svg)

---

03

## Training

---

## The loss, one more time

$$L\_{CE}(\hat{y}\_t, y\_t) = -\log \hat{y}\_t[w\_{t+1}]$$

- **Self-supervised**: the next word in real text *is* the label
- Sequence loss = the **average** over positions; **teacher forcing**: condition on the true history, not the model's guesses
- The same loss since **lecture 4** — third and last appearance

---

## Gradient descent

![Loss curve with two computed learning-rate trajectories: small eta converges, large eta overshoots](../figures/gradient-descent.svg)

---

## Backpropagation: where gradients come from

- The network is a **graph of simple operations**
- Chain rule, applied **backwards** through the graph: every parameter's gradient in **one backward pass**
- You've used it: `loss.backward()` in the lecture-4 notebooks

---

## Making it scale: SGD

- **Mini-batches**: estimate the gradient on 32–4M tokens, not the whole corpus
- **Epoch** = one pass over the data; **shuffle** between epochs
- Noise is a feature — helps generalisation; **Adam** & friends adapt η per parameter

---

Pre-training is **industrial**. Adapting is **affordable**.

The rest of the lecture is the affordable part.

---

04

## Adapting a pre-trained model

---

## Transfer learning

- A model pre-trained on *everything* already encodes language, facts, style
- **Fine-tuning**: keep the body, retrain for *your* task
- Update **all** weights, or just the **new parts** — a cost/quality dial

---

## Route 1: bolt on a task head

- Replace the language-model head with a **classifier head**
- Example: sentiment — sequence embedding → softmax(positive / negative)
- Needs **labelled data**; train the head (and optionally the body) with SGD

---

## Route 2: teach it to follow instructions

- **Instruction tuning (SFT)**: fine-tune on curated instruction → response pairs
- **RLHF**: humans rank outputs → reward model → optimise against it
- Newer: RL on **verifiable rewards** — math, code, tests → reasoning models
- This — not pre-training — is what turned text-completers into **assistants**

---

## Route 3: barely touch it

- **LoRA / PEFT**: freeze the body, train tiny low-rank add-ons — fine-tuning on one GPU
- **Feature extraction**: use the model's **embeddings** as inputs to your own model
- **Prompting**: adapt with **no** training at all → lecture 11

---

## Whichever route: how do you know it works?

- A **golden set** — held-out examples with known good answers, before you start
- **Metrics where you can**, human or **LLM-as-judge** review where you can't
- **Audit the errors**, not just the score — *which* cases fail, and why?

The train/validate/test discipline from lecture 3 — it never went away.

---

## Questions

## Positional encodings, the original recipe

$$PE\_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right) \qquad PE\_{(pos, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)$$

- Unique per position; relative offsets are linear functions of each other
- Extends beyond training lengths; modern models often **learn** positions instead

## Backpropagation, the algorithm

- Feedforward: $z^{l} = w^{l} a^{l-1}$, $\;a^{l} = \sigma(z^{l})$
- Output error: $\delta^{L} = \nabla\_a L \odot \sigma'(z^{L})$
- Backwards: $\delta^{l} = \big((w^{l+1})' \delta^{l+1}\big) \odot \sigma'(z^{l})$
- Update: $w^{l} \leftarrow w^{l} - \frac{\eta}{m} \sum\_x \delta^{x,l} (a^{x,l-1})'$

## Encoder–decoder and cross-attention

$$\text{CrossAttention}(Q, K, V) = \text{softmax}\!\left(\frac{QK'}{\sqrt{d\_k}}\right)V \quad \begin{aligned} Q &= W^Q H^{dec} \\\\ K, V &= W^K H^{enc},\; W^V H^{enc} \end{aligned}$$

- The original transformer: encoder reads the source, decoder writes the target
- Queries from the **decoder**, keys/values from the **encoder** — translation's workhorse

---

## What's next

- **Lecture 10 (Tue): practical day 2** — pre-trained models hands-on; **bring a laptop**
- Course-server access details come with the practical-day materials
- Reading: J&M chapters 8–10 · finish **HF course ch. 1–3** if you haven't