Learning from prediction 2: word embeddings
Pointwise mutual information, word embeddings, and what you can do with vector representations of words
Open slidesOverview
“You shall know a word by the company it keeps” (Firth, 1957) — this lecture makes that idea mathematical. Every word becomes a dense vector in which similarity and even direction carry meaning (king − man + woman ≈ queen). Two routes get us there: counting (co-occurrence windows → pointwise mutual information → SVD) and predicting (word2vec’s skip-gram with negative sampling) — and, echoing lecture 4, the two land in nearly the same place. As proof the counting route works, the slides include embeddings computed for this course from 223 State of the Union addresses: war, money, law, and nations vocabulary cluster on their own, with no labels and no neural network. We close with research applications that turn embedding arithmetic into measurement: cultural dimensions (the assigned Kozlowski et al. reading), credit-market risk indices, and climate transition risk.
Topics
- From bag-of-words to context: what embeddings fix
- The counting route: co-occurrence, PMI/PPMI, truncated SVD
- The predicting route: word2vec, skip-gram, negative sampling
- Why the two routes agree (Levy et al. 2015) — and the one-vector-per-word limit that transformers break
- Embedding arithmetic as a research tool
Readings
- J&M chapters 5 and 6
- Kozlowski, Taddy & Evans (2019), The Geometry of Culture — discussed in the lecture
Materials
- Slides: use the Open slides button above
- No dedicated notebook — embeddings are exercised on practical day 2 and in assignment 2
Before lecture 7
- Finish the Kozlowski et al. reading if you haven’t
- Lecture 7 starts the LLM block — no other preparation needed