LLM input/output, training, and fine-tuning
How LLMs are specified, estimated, and adapted — pre-training, fine-tuning, and what it means for applied work
Open slidesOverview
You know the architecture — this lecture covers where the weights come from
and how to make someone else’s weights work for you. First the run-time
mechanics: tokens to embeddings to transformer blocks to a next-token
distribution, and the sampling knobs (temperature, top-k, top-p) that turn
that distribution into text. Then training: cross-entropy (the same loss for
the third time in this course), gradient descent, backpropagation —
demystified rather than re-derived, since you already called loss.backward() in lecture 4 — and SGD at industrial scale. The punchline
is an asymmetry: pre-training is industrial, adapting is affordable — so
the last section is the adaptation menu, ordered by budget: task-head
fine-tuning, instruction tuning and RLHF (what turned text-completers into
assistants), and the barely-touch-it routes — LoRA, embeddings as features,
and prompting.
Topics
- The full stack at run time: embeddings + positions, blocks, the tied head, and the generation loop
- Context windows and silent truncation
- Sampling: greedy, top-k, top-p, and temperature — on a real distribution
- Training: cross-entropy, teacher forcing, gradient descent, backprop, SGD
- Adapting pre-trained models: task heads, instruction tuning + RLHF (and RL on verifiable rewards — where reasoning models come from), LoRA/PEFT, feature extraction, prompting
- Evaluating adapted models: golden sets, LLM-as-judge, error audits
Readings
- J&M chapters 8, 9, and 10
Asynchronous work
- Hugging Face NLP course, chapters 1–3 — directly preparation for practical day 2
Materials
- Slides: use the Open slides button above
- No dedicated notebook — practical day 2 (lecture 10) is the hands-on session for this material
Before lecture 10
- Bring a laptop — practical day 2 works with pre-trained models
- Finish HF course chapters 1–3 if you haven’t