5-analysis-and-visualisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Steps 4 and 5: Analysing Data and Visualising Results\n",
    "---\n",
    "---\n",
    "Firstly, we need to reload the cleaned list of word tokens we were using in the [previous notebook](4-cleaning-and-exploring.ipynb) that we saved in a file. (You don't need to understand what is happening here in detail to follow the rest of the notebook.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import a module that helps with filepaths\n",
    "from pathlib import Path\n",
    "\n",
    "# Create a filepath for the file\n",
    "tokens_file = Path('data', 'CLEAN-2199-0.txt')\n",
    "\n",
    "# Create an empty list to hold the tokens\n",
    "tokens = []\n",
    "\n",
    "# Open the text file and append all the words to a list of tokens\n",
    "with open(tokens_file, encoding='utf-8') as file:\n",
    "    for token in file.read().split():\n",
    "        tokens.append(token)\n",
    "\n",
    "tokens[0:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Step 4: Analysing Data with Frequency Analysis\n",
    "Let's take a moment to remember our research question:\n",
    "\n",
    "> What are the top 10 words used in Homer's Iliad in English translation?\n",
    "\n",
    "In order to answer this question we need to count the number of each unique word in the text. Then we can see which are the most popular, or frequent, 10 words. This metric is called a *frequency distribution*. \n",
    "\n",
    "### English Stopwords\n",
    "Before we start, we need to take a moment to think about what sort of words we are actually interested in counting. \n",
    "\n",
    "We are not interested in common words in English that carry little meaning, such as \"the\", \"a\" and \"its\". These are called *stopwords*. There is no definitive list of stopwords, but most Python packages used for Natural Language Processing provide one as a starting point, and spaCy is no exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the spaCy standard stopwords list\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "stopwords = [stop for stop in STOP_WORDS]\n",
    "\n",
    "# Sort the stopwords in alphabetical order to make them easier to inspect\n",
    "sorted(stopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write code here to count the number of stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Exercise**: What do you notice about these stopwords?\n",
    "\n",
    "For your own research you will need to consider which stopwords are most appropriate:\n",
    "* Will standard stopword lists for modern languages be suitable for that language written 10, 50, 200 years ago?\n",
    "* Are there special stopwords specific to the topic or style of literature?\n",
    "* How might you find or create your own stopword list?\n",
    "\n",
    "Now we can filter out the stopwords that match this list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_nostops = [token for token in tokens if token not in stopwords]\n",
    "tokens_nostops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Frequency Distribution\n",
    "At last, we are ready to create a frequency distribution by counting the frequency of each unique word in the text.\n",
    "\n",
    "First, we create a frequency distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import a module that helps with counting\n",
    "from collections import Counter\n",
    "\n",
    "# Count the frequency of words\n",
    "word_freq = Counter(tokens_nostops)\n",
    "word_freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `Counter` maps each word to the number of times it appears in the text, e.g. `'coward': 17`. By scrolling down the list you can inspect what look like common and infrequent words.\n",
    "\n",
    "Now we can get precisely the 10 most common words using the function `most_common()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "common_words = word_freq.most_common(10)\n",
    "common_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Exercise**: Investigate what is further down the list of top words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Step 5: Presenting Results of the Analysis Visually\n",
    "There are many options for displaying simple charts, and very complex data, in Jupyter notebooks. We are going to use the most well-known library called [Matplotlib](https://matplotlib.org/), although it is perhaps not the easiest to use compared with some others.\n",
    "\n",
    "To create a Matplotlib plot we need to:\n",
    "\n",
    "* Import the matplotlib plot function\n",
    "* Arrange our data into a pair of lists: one for the x-axis, one for the y-axis\n",
    "* Set the appearance of titles, labels, ticks and gridlines\n",
    "* Pass the data into the plot function\n",
    "\n",
    "Let's display our results as a simple line plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the plot inline in the notebook with interactive controls\n",
    "# Comment out this line if you are running the notebook in Deepnote\n",
    "%matplotlib notebook\n",
    "\n",
    "# Import the matplotlib plot function\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Get a list of the most common words\n",
    "words = [word for word,_ in common_words]\n",
    "\n",
    "# Get a list of the frequency counts for these words\n",
    "freqs = [count for _,count in common_words]\n",
    "\n",
    "# Set titles, labels, ticks and gridlines\n",
    "plt.title(\"Top 10 Words used in Homer's Iliad in English translation\")\n",
    "plt.xlabel(\"Word\")\n",
    "plt.ylabel(\"Count\")\n",
    "plt.xticks(range(len(words)), [str(s) for s in words], rotation=90)\n",
    "plt.grid(visible=True, which='major', color='#333333', linestyle='--', alpha=0.2)\n",
    "\n",
    "# Plot the frequency counts\n",
    "plt.plot(freqs)\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this interactive plot you can:\n",
    "\n",
    "* Resize the plot by dragging the bottom right-hand corner.\n",
    "* Pan across the plot to see values further to the right (if there are any to display).\n",
    "* Zoom into the plot.\n",
    "\n",
    "> **Exercise**: Change the code to explore different data and ways of displaying your data. \n",
    "\n",
    "There are also lots of other graphs that Matplotlib can create, and alternative plotting libraries to use instead, but these are beyond the scope of our course.\n",
    "\n",
    "---\n",
    "---\n",
    "## Review and Reflection\n",
    "Now that you have seen the data and graph we have generated, no doubt you can see many ways we should improve. \n",
    "\n",
    "The process of text-mining a corpus (or individual text) is an iterative process. As you clean and explore the data, you will go back over your workflow again and again: from the collection stage, through to cleaning, analysis and presentation.\n",
    "\n",
    "> **Exercise**: List the ways you think we should improve the pipeline, from collection to plot.\n",
    "\n",
    "Fortunately, when you do your text-mining in code (and write explanatory text to document it) you know exactly what you did and can rerun and modify the process.\n",
    "\n",
    "---\n",
    "### Going Further: Libraries Libraries Libraries\n",
    "\n",
    "By now, you will be getting the idea that much of what you want to do in Python involves importing libraries to help you. Remember, libraries are _just code that someone else has written_.\n",
    "\n",
    "As reminder, here are some of the useful libraries we have used or mentioned in these notebooks:\n",
    "* [Requests](http://docs.python-requests.org/en/master/) - HTTP (web) requests library\n",
    "* [SpaCy](https://spacy.io/) - natural language processing library\n",
    "* [Matplotlib](https://matplotlib.org/) - 2D plotting library\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Summary\n",
    "\n",
    "Finally, we have: \n",
    "\n",
    "* **Loaded** clean token data from a file into a list.\n",
    "* Removed English **stopwords** from the list of tokens.\n",
    "* Created a **frequency distribution** and found the 10 most frequent words.\n",
    "* Visualised the frequency distribution in a **line plot**.\n",
    "\n",
    "---\n",
    "---\n",
    "## What's Next?\n",
    "You will get the most out of this course if you can follow up on the learning over the next few days and weeks before you forget it all. This is particularly important when learning to code. Abstract concepts need to be reinforced little and often.\n",
    "\n",
    "### Recommended book on Python\n",
    "\n",
    "* Sweigart, A. 2019. _Automate the Boring Stuff_ (2nd ed.) San Francisco: No Starch Press. [Available online](https://automatetheboringstuff.com/). This is just an all-round good book on Python and how to give yourself computer superpowers.\n",
    "\n",
    "### Text-mining and NLP in General\n",
    "\n",
    "* Work through this series of [Programming Historian tutorials](https://programminghistorian.org/en/lessons/working-with-text-files) to get some more practice with basic text files and basic text-mining techniques.\n",
    "* Follow a more in-depth set of Jupyter notebooks with [The Art of Literary Text Analysis](https://github.com/sgsinclair/alta/blob/master/ipynb/ArtOfLiteraryTextAnalysis.ipynb).\n",
    "* Read a practical and well-explained approach to Natural Language Processing (NLP) in Python: Srinivasa-Desikan, B., 2018. _Natural Language Processing and Computational Linguistics : A practical guide to text analysis with Python, Gensim, spaCy, and Keras._ Birmingham: Packt Publishing. This book has chapters on text pre-processing steps, various NLP techniques, and comes with Jupyter notebooks to follow.\n",
    "\n",
    "### Python for Digital Humanities\n",
    "\n",
    "* Work through [Python Programming for the Humanities](http://www.karsdorp.io/python-course/).\n",
    "* Browse a big list of resources for [Teaching Yourself to Code in DH](http://scottbot.net/teaching-yourself-to-code-in-dh/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}