Bag-of-words, TF-IDF, and Cosine Similarity

<p class="kicker">GRA 4164 · Lecture 2 · Fall 2026</p>

## Bag-of-words, TF-IDF,<br>and Cosine Similarity

### Leif Anders Thorsrud & Vegard H. Larsen — BI Norwegian Business School

---

Computers can't read. Today, text becomes **numbers**.

Counting words gets you further than you'd expect.

---

## Where we are

![Five-step pipeline from raw text to business decisions](../figures/text-as-data-workflow.svg)

---

## From text to a matrix

---

## Getting a corpus

- **Corpus** — all the documents you analyse
- **Extraction** — third-party data, APIs, scraping, manual collection
- **Storage** — CSV works, XML is ok, **JSON is good**

```json
{ "docid": "Doc1", "date": 2018,
  "meta": { "author": "LAT" },
  "rawtext": "Using text as data is new to most economists..." }
```

---

## What counts as a document?

- A **sentence**? An **article**? Everything published in a **day**?
- Notation: corpus $D$ with documents $D\_i$, $i = 1, \ldots, n$
- The unit you pick changes every number downstream

It's a modelling decision — remember lecture 1.

---

## Preprocessing, in five lines

```python
text = text.lower()                                      # casing
text = text.translate(str.maketrans('', '', punctuation)) # punctuation
tokens = text.split()                                     # tokenize
tokens = [t for t in tokens if t not in stop_words]       # stop words
tokens = [t for t in tokens if not t.isnumeric()]         # numbers
```

---

## The document-term matrix

| | using | text | as | data | … |
|---|---|---|---|---|---|
| **Doc1** | 1 | 1 | 2 | 2 | … |
| **Doc2** | 0 | 3 | 0 | 2 | … |
| **Doc3** | 0 | 0 | 5 | 2 | … |

- $\boldsymbol{C}$ is $n \times p$: documents × vocabulary
- Typically **sparse** — and typically **huge**

---

## Word order? Use n-grams

- **1-gram** — every word on its own: order is lost
- **2-gram** — "low unemployment" vs "high unemployment" — now distinguishable
- **3-gram** and up — more context, **exploding vocabulary**

---

## TF-IDF

---

## Why not just raw counts?

- Long documents get **big counts** — for every term
- Frequent words like "market" swamp informative ones
- Simple fixes: shares $\boldsymbol{c}\_i / m\_i$, or binary $\boldsymbol{1}\_{c\_{ij} > 0}$

The standard answer is **TF-IDF**.

---

## The definition

$$tf\_{ij} = c\_{ij} \qquad\qquad idf\_j = \log\!\left(\frac{n}{d\_j}\right)$$

$$\text{tf-idf}\_{ij} = tf\_{ij} \times idf\_j$$

$d\_j$ = number of documents containing term $j$.

---

## What it does

<div class="cols">
<div class="callout">
<h3>Rare terms ↑</h3>
<p>High IDF — "litigation" in one report out of hundreds is a signal.</p>
</div>
<div class="callout">
<h3>Common terms ↓</h3>
<p>Low IDF — "company" in every report distinguishes nothing.</p>
</div>
</div>

Used for **weighting** features — and for **selecting** them.

---

## Cosine similarity

---

## Documents are now vectors

- Each row $\boldsymbol{c}\_i$ is a point in $p$-dimensional space
- Similar documents → similar vectors
- **But what does "similar" mean for vectors?**

---

## Two answers, one picture

![Three document vectors compared by Euclidean distance and by angle](../figures/vector-space-cosine.svg)

---

## The formula

$$\cos \theta\_{12} = \frac{\overrightarrow{d\_1} \cdot \overrightarrow{d\_2}}{\lVert \overrightarrow{d\_1} \rVert \, \lVert \overrightarrow{d\_2} \rVert}$$

- **1** — same direction · **0** — nothing in common
- Document length **cancels out** in the denominator

---

Cosine similarity compares **direction**, not **length**.

What a document is about — not how long it is.

---

## This can be enough

- Engle et al. (2020): hedge **climate risk** by tracking climate news
- Method at its core: **TF-IDF** vocabulary + **cosine similarity** of news to it
- Published in the *Review of Financial Studies*

No deep learning required.

---

## Try it: Shakespeare

```python
tfidf = TfidfVectorizer().fit_transform(plays['Content_clean'])
sims  = cosine_similarity(tfidf)
```

- 5 MB of raw text → a play-by-play similarity matrix
- **Which play is closest to Hamlet?** The notebook answers it

---

## In sum

- Text → **document-term matrix**: the bag-of-words move
- **TF-IDF** reweights counts by informativeness
- **Cosine similarity** finds similar documents — length-free

</div>

---

## Questions

## Other ways to slice the matrix

| Sentence level | using | text | as | data |
|---|---|---|---|---|
| **Sen1** | 1 | 1 | 1 | 1 |
| **Sen2** | 0 | 0 | 0 | 0 |

| 2-grams, doc level | using text | text as | as data |
|---|---|---|---|
| **Doc1** | 1 | 1 | 2 |
| **Doc2** | 0 | 3 | 0 |

## The normalization menu

- Raw counts: $\boldsymbol{x}\_i = \boldsymbol{c}\_i$
- Shares: $\boldsymbol{x}\_i = \boldsymbol{c}\_i / m\_i$ with $m\_i = \sum\_j c\_{ij}$
- Binary: $x\_{ij} = \boldsymbol{1}\_{c\_{ij} > 0}$
- TF-IDF: $x\_{ij} = c\_{ij} \log(n / d\_j)$

## Scraping in six lines

```python
from urllib.request import urlopen
from bs4 import BeautifulSoup

with urlopen('https://www.bi.edu') as r:
    soup = BeautifulSoup(r.read().decode('utf-8'), 'html.parser')
print(soup.find_all('h2'))
```

---

## What's next

- **Lecture 3 (Mon):** methods built on this matrix — dictionaries, text regression, topic models
- **Before then:** run the Shakespeare notebook; read **Engle et al. (2020)**
- Reading: lecture notes; Bholat et al. (2015)