{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical Day 1\n",
    "## Vegard H. Larsen\n",
    "## Course: Text as data (GRA 4164)\n",
    "### Lecture 5,  October 30th 2024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plan for today:\n",
    "1. Tips for working with code\n",
    "2. Introduction to some useful Python libraries\n",
    "3. Talk about assignment\n",
    "4. Coding/practical session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Tips for Working with Code\n",
    "\n",
    "1. **Use Environments to Manage Your Python Installation**  \n",
    "   - Set up virtual environments (e.g., with `venv` or `conda`) to keep your project dependencies organized and avoid conflicts. Each project can have its own environment with the specific libraries it needs, making it easier to manage and debug.\n",
    "\n",
    "2. **Use Generative AI as a Coding Aid**  \n",
    "   - Generative AI tools like ChatGPT can be invaluable for brainstorming code, troubleshooting, and getting quick explanations of concepts. However, make sure to understand the code you receive—use AI to support your learning, not replace it.\n",
    "\n",
    "3. **Start Small and Test Often**  \n",
    "   - Break down your tasks into smaller, manageable pieces and test your code frequently. This approach makes it easier to identify errors early and gives you more confidence in each part of your code as it builds up.\n",
    "\n",
    "4. **Use Version Control (e.g., Git)**  \n",
    "   - Even if you're just starting, using version control can save you from losing work and help you understand how your code changes over time. Platforms like GitHub are helpful for tracking progress and collaborating.\n",
    "\n",
    "5. **Use Descriptive Naming and Comment Your Code**  \n",
    "   - Use clear and descriptive variable names, and add comments to explain the purpose of more complex sections. This makes your code easier to read and understand for both you and others who might look at it.\n",
    "\n",
    "6. **Debugging is Part of the Process**  \n",
    "   - Don’t be discouraged by errors! Learning to read and understand error messages will make debugging less intimidating. Start by searching for the exact error message online—chances are, others have encountered it before.\n",
    "\n",
    "7. **Build a Habit of Reading Documentation**  \n",
    "   - Official documentation and library docs (like pandas, NumPy, etc.) are great resources. Familiarize yourself with them to quickly find explanations and examples that can deepen your understanding.\n",
    "\n",
    "8. **Google is Your Friend**  \n",
    "   - Even experienced programmers search for solutions and examples frequently. Don’t hesitate to look up questions you have—sites like Stack Overflow often have relevant answers for common coding challenges.\n",
    "\n",
    "9. **Coding is changing fast**  \n",
    "    - New tools have significantly transformed the life of programmers, with platforms like GitHub Copilot, AI-based code generators, and improved development environments lowering barriers to entry. This shift is making coding accessible to a wider range of people, including those without formal technical backgrounds. Staying curious about new tools and being open to experimenting with them can enhance your productivity and skillset—coding is becoming less about memorizing syntax and more about problem-solving and leveraging resources.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "latex"
    }
   },
   "source": [
    "## Python Libraries\n",
    "\n",
    "- **NLTK - Natural Language Toolkit**  \n",
    "    - A foundational library for text processing, offering tools for tokenization, stemming, lemmatization, and part-of-speech tagging.\n",
    "    - Excellent for beginners due to its extensive documentation and built-in corpora for experimentation.\n",
    "\n",
    "- **Scikit-learn - Machine Learning in Python**  \n",
    "    - Provides a wide range of machine learning algorithms and utilities for classification, clustering, regression, and dimensionality reduction.\n",
    "    - Integrates well with text data for tasks like text classification, sentiment analysis, and feature extraction (e.g., TF-IDF).\n",
    "\n",
    "- **Gensim - Topic Modeling for Humans**  \n",
    "    - Specialized in topic modeling and document similarity, making it ideal for analyzing and discovering patterns in large text corpora.\n",
    "    - Implements popular algorithms like LDA (Latent Dirichlet Allocation) and Word2Vec, helping to capture semantic structure in text data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Part-of-Speech (POS) Tagging\n",
    "\n",
    "Part-of-Speech (POS) Tagging is the process of assigning grammatical labels (tags) to each word in a sentence based on its role and function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk import pos_tag\n",
    "\n",
    "# Download necessary NLTK resources\n",
    "nltk.download('punkt')\n",
    "nltk.download('averaged_perceptron_tagger_eng')\n",
    "\n",
    "# Sample sentence\n",
    "sentence = \"The quick brown fox jumps over the lazy dog.\"\n",
    "\n",
    "# Tokenize the sentence into words\n",
    "words = word_tokenize(sentence)\n",
    "print(words)\n",
    "\n",
    "# Perform POS tagging\n",
    "tagged_words = pos_tag(words)\n",
    "\n",
    "# Display the POS tags\n",
    "print(\"Word\\tPOS Tag\")\n",
    "for word, tag in tagged_words:\n",
    "    print(f\"{word}\\t{tag}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **DT** - Determiner\n",
    "- **JJ** - Adjective, describes a noun \n",
    "- **NN** - Noun, person, place, thing, or idea\n",
    "- **VBZ** - Verb, third-person singular present \n",
    "- **IN** - Preposition or subordinating conjunction "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Computing TF-IDF with Scikit-Learn\n",
    "\n",
    "TF-IDF (Term Frequency-Inverse Document Frequency) is a numerical statistic that reflects the importance of a word in a document relative to a collection of documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# Sample documents\n",
    "documents = [\n",
    "    \"The quick brown fox jumps over the lazy dog.\",\n",
    "    \"Never jump over the lazy dog quickly.\",\n",
    "    \"A fox is quick and jumps high.\"\n",
    "]\n",
    "\n",
    "# Initialize the TF-IDF Vectorizer\n",
    "tfidf_vectorizer = TfidfVectorizer()\n",
    "\n",
    "# Fit and transform the documents\n",
    "tfidf_matrix = tfidf_vectorizer.fit_transform(documents)\n",
    "\n",
    "# Get feature names to view the vocabulary\n",
    "feature_names = tfidf_vectorizer.get_feature_names_out()\n",
    "\n",
    "# Convert the TF-IDF matrix to a DataFrame for a better view\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(tfidf_matrix.toarray(), columns=feature_names)\n",
    "print(\"TF-IDF Matrix:\")\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each cell in the resulting DataFrame represents the TF-IDF score of a word in a particular document, with higher values indicating greater relevance of the term to that document. \n",
    "\n",
    "Each row represents a document, and each column represents a word's TF-IDF score in that document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling with Gensim: LDA Example\n",
    "\n",
    "Latent Dirichlet Allocation (LDA) is a popular topic modeling technique that discovers underlying topics in a collection of documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the 20 Newsgroups dataset\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "import gensim\n",
    "from gensim import corpora\n",
    "\n",
    "# Download NLTK stopwords if not already downloaded\n",
    "nltk.download('stopwords')\n",
    "\n",
    "# Load the 20 Newsgroups dataset\n",
    "newsgroups_data = fetch_20newsgroups(subset='train', remove=('headers', 'footers', 'quotes'))\n",
    "documents = newsgroups_data.data[:2000]  # Limit to 2000 documents for performance\n",
    "\n",
    "# Define stop words\n",
    "stop_words = set(stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic preprocessing to remove stop words and tokenize\n",
    "from nltk.tokenize import word_tokenize\n",
    "import re\n",
    "\n",
    "# Basic preprocessing to remove stop words, tokenize, and remove non-alphabetic characters\n",
    "# Update preprocessing to remove short words and keep only alphabetic characters\n",
    "processed_docs = [\n",
    "    [\n",
    "        word.lower() for word in word_tokenize(re.sub(r'\\W+', ' ', doc))\n",
    "        if word.isalpha() and word.lower() not in stop_words and len(word) > 2  \n",
    "    ]\n",
    "    for doc in documents\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Dictionary and Corpus for Gensim\n",
    "\n",
    "# Create a dictionary representation of the documents\n",
    "dictionary = corpora.Dictionary(processed_docs)\n",
    "\n",
    "# Filter out extreme terms\n",
    "dictionary.filter_extremes(no_below=15, no_above=0.5)\n",
    "\n",
    "# Convert the documents into a bag-of-words format\n",
    "corpus = [dictionary.doc2bow(doc) for doc in processed_docs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the LDA Model\n",
    "# Set parameters for LDA\n",
    "num_topics = 10  # Number of topics to extract\n",
    "\n",
    "# Build the LDA model\n",
    "lda_model = gensim.models.LdaModel(corpus, num_topics=num_topics, id2word=dictionary, passes=10, random_state=42)\n",
    "\n",
    "# Display the topics\n",
    "for idx, topic in lda_model.print_topics(-1):\n",
    "    print(f\"Topic {idx + 1}: {topic}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output shows the top terms in each topic, with weights indicating the relevance of each term within the topic. \n",
    "\n",
    "Each topic represents a grouping of related terms."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}