code/ch08/ch08.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Python Machine Learning 2nd Edition* by [Sebastian Raschka](https://sebastianraschka.com), Packt Publishing Ltd. 2017\n",
    "\n",
    "Original Code Repository: https://github.com/rasbt/python-machine-learning-book-2nd-edition\n",
    "\n",
    "Adapted for this course: https://github.com/trungngv/python-machine-learning-book-2nd-edition/blob/master/code/ch08/ch08.ipynb\n",
    "\n",
    "Code License: [MIT License](https://github.com/rasbt/python-machine-learning-book-2nd-edition/blob/master/LICENSE.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 5 - Applying Machine Learning To Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [The movie reviews dataset](#The-IMDB-Moview-Review-dataset)\n",
    "- [Solving problems with machine learning](#Solving-problems-with-machine-learning)\n",
    "    - [Supervised learning](#Supervise-learning)\n",
    "        - [Text classification](#Text-classification)\n",
    "- [Transforming documents into feature vectors](#Transforming-documents-into-feature-vectors)\n",
    "    - [Word indicators representation](#Word-indicators-representation)\n",
    "    - [One-hot-encoding representation](#One-hot-encoding-representation)\n",
    "    - [Bag-of-words representation](#Bag-of-words-representation)\n",
    "    - [Tfidf representation](#Tfidf)\n",
    "- [Creating a modelling dataset](#Creating-a-modelling-dataset)\n",
    "    - [Cleaning text data](#Cleaning-text-data)\n",
    "- [Training a logistic regression model for document classification](#Training-a-logistic-regression-model-for-document-classification)\n",
    "- [Working with bigger data – online algorithms and out-of-core learning](#Working-with-bigger-data-–-online-algorithms-and-out-of-core-learning)\n",
    "- [Summary](#Summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The IMDB Moview Review dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The task: predict sentiment of movie reviews.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original IMDB movie review set can be downloaded from [http://ai.stanford.edu/~amaas/data/sentiment/](http://ai.stanford.edu/~amaas/data/sentiment/).\n",
    "\n",
    "You can download the dataset from Github or if you cloned this repository, you should have the file and don't have to do anything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data in with Pandas. Note that Pandas can read compressed file automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('movie_data.csv.gz', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice from the example reviews?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In 1974, the teenager Martha Moxley (Maggie Grace) moves to the high-class area of Belle Haven, Greenwich, Connecticut. On the Mischief Night, eve of Halloween, she was murdered in the backyard of her house and her murder remained unsolved. Twenty-two years later, the writer Mark Fuhrman (Christopher Meloni), who is a former LA detective that has fallen in disgrace for perjury in O.J. Simpson trial and moved to Idaho, decides to investigate the case with his partner Stephen Weeks (Andrew Mitchell) with the purpose of writing a book. The locals squirm and do not welcome them, but with the support of the retired detective Steve Carroll (Robert Forster) that was in charge of the investigation in the 70's, they discover the criminal and a net of power and money to cover the murder.&lt;br /&gt;&lt;br /&gt;\"Murder in Greenwich\" is a good TV movie, with the true story of a murder of a fifteen years old girl that was committed by a wealthy teenager whose mother was a Kennedy. The powerful and rich family used their influence to cover the murder for more than twenty years. However, a snoopy detective and convicted perjurer in disgrace was able to disclose how the hideous crime was committed. The screenplay shows the investigation of Mark and the last days of Martha in parallel, but there is a lack of the emotion in the dramatization. My vote is seven.&lt;br /&gt;&lt;br /&gt;Title (Brazil): Not Available</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK... so... I really like Kris Kristofferson and his usual easy going delivery of lines in his movies. Age has helped him with his soft spoken low energy style and he will steal a scene effortlessly. But, Disappearance is his misstep. Holy Moly, this was a bad movie! &lt;br /&gt;&lt;br /&gt;I must give kudos to the cinematography and and the actors, including Kris, for trying their darndest to make sense from this goofy, confusing story! None of it made sense and Kris probably didn't understand it either and he was just going through the motions hoping someone would come up to him and tell him what it was all about! &lt;br /&gt;&lt;br /&gt;I don't care that everyone on this movie was doing out of love for the project, or some such nonsense... I've seen low budget movies that had a plot for goodness sake! This had none, zilcho, nada, zippo, empty of reason... a complete waste of good talent, scenery and celluloid! &lt;br /&gt;&lt;br /&gt;I rented this piece of garbage for a buck, and I want my money back! I want my 2 hours back I invested on this Grade F waste of my time! Don't watch this movie, or waste 1 minute of your valuable time while passing through a room where it's playing or even open up the case that is holding the DVD! Believe me, you'll thank me for the advice!</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***SPOILER*** Do not read this, if you think about watching that movie, although it would be a waste of time. (By the way: The plot is so predictable that it does not make any difference if you read this or not anyway)&lt;br /&gt;&lt;br /&gt;If you are wondering whether to see \"Coyote Ugly\" or not: don't! It's not worth either the money for the ticket or the VHS / DVD. A typical \"Chick-Feel-Good-Flick\", one could say. The plot itself is as shallow as it can be, a ridiculous and uncritical version of the American Dream. The young good-looking girl from a small town becoming a big success in New York. The few desperate attempts of giving the movie any depth fail, such as the \"tragic\" accident of the father, the \"difficulties\" of Violet's relationship with her boyfriend, and so on. McNally (Director) tries to arouse the audience's pity and sadness put does not have any chance to succeed in this attempt due to the bad script and the shallow acting. Especially Piper Perabo completely fails in convincing one of \"Jersey's\" fear of singing in front of an audience. The only good (and quite funny thing) about \"Coyote Ugly\" is John Goodman, who represents the small ray of hope of this movie.&lt;br /&gt;&lt;br /&gt;I was very astonished, that Jerry Bruckheimer produced this movie. First \"Gone In 60 Seconds\" and now this... what happened to great movies like \"The Rock\" and \"Con Air\"? THAT was true Bruckheimer stuff.&lt;br /&gt;&lt;br /&gt;If you are looking for a superficial movie with good looking women just to have a relaxed evening, you should better go and see \"Charlie's Angels\" (it's much more funny, entertaining and self-ironic) instead of this flick.&lt;br /&gt;&lt;br /&gt;Two thumbs down (3 out of 10).</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           review  \\\n",
       "0  In 1974, the teenager Martha Moxley (Maggie Grace) moves to the high-class area of Belle Haven, Greenwich, Connecticut. On the Mischief Night, eve of Halloween, she was murdered in the backyard of her house and her murder remained unsolved. Twenty-two years later, the writer Mark Fuhrman (Christopher Meloni), who is a former LA detective that has fallen in disgrace for perjury in O.J. Simpson trial and moved to Idaho, decides to investigate the case with his partner Stephen Weeks (Andrew Mitchell) with the purpose of writing a book. The locals squirm and do not welcome them, but with the support of the retired detective Steve Carroll (Robert Forster) that was in charge of the investigation in the 70's, they discover the criminal and a net of power and money to cover the murder.<br /><br />\"Murder in Greenwich\" is a good TV movie, with the true story of a murder of a fifteen years old girl that was committed by a wealthy teenager whose mother was a Kennedy. The powerful and rich family used their influence to cover the murder for more than twenty years. However, a snoopy detective and convicted perjurer in disgrace was able to disclose how the hideous crime was committed. The screenplay shows the investigation of Mark and the last days of Martha in parallel, but there is a lack of the emotion in the dramatization. My vote is seven.<br /><br />Title (Brazil): Not Available                                                                                                                                                                                                                                                                                                \n",
       "1  OK... so... I really like Kris Kristofferson and his usual easy going delivery of lines in his movies. Age has helped him with his soft spoken low energy style and he will steal a scene effortlessly. But, Disappearance is his misstep. Holy Moly, this was a bad movie! <br /><br />I must give kudos to the cinematography and and the actors, including Kris, for trying their darndest to make sense from this goofy, confusing story! None of it made sense and Kris probably didn't understand it either and he was just going through the motions hoping someone would come up to him and tell him what it was all about! <br /><br />I don't care that everyone on this movie was doing out of love for the project, or some such nonsense... I've seen low budget movies that had a plot for goodness sake! This had none, zilcho, nada, zippo, empty of reason... a complete waste of good talent, scenery and celluloid! <br /><br />I rented this piece of garbage for a buck, and I want my money back! I want my 2 hours back I invested on this Grade F waste of my time! Don't watch this movie, or waste 1 minute of your valuable time while passing through a room where it's playing or even open up the case that is holding the DVD! Believe me, you'll thank me for the advice!                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
       "2  ***SPOILER*** Do not read this, if you think about watching that movie, although it would be a waste of time. (By the way: The plot is so predictable that it does not make any difference if you read this or not anyway)<br /><br />If you are wondering whether to see \"Coyote Ugly\" or not: don't! It's not worth either the money for the ticket or the VHS / DVD. A typical \"Chick-Feel-Good-Flick\", one could say. The plot itself is as shallow as it can be, a ridiculous and uncritical version of the American Dream. The young good-looking girl from a small town becoming a big success in New York. The few desperate attempts of giving the movie any depth fail, such as the \"tragic\" accident of the father, the \"difficulties\" of Violet's relationship with her boyfriend, and so on. McNally (Director) tries to arouse the audience's pity and sadness put does not have any chance to succeed in this attempt due to the bad script and the shallow acting. Especially Piper Perabo completely fails in convincing one of \"Jersey's\" fear of singing in front of an audience. The only good (and quite funny thing) about \"Coyote Ugly\" is John Goodman, who represents the small ray of hope of this movie.<br /><br />I was very astonished, that Jerry Bruckheimer produced this movie. First \"Gone In 60 Seconds\" and now this... what happened to great movies like \"The Rock\" and \"Con Air\"? THAT was true Bruckheimer stuff.<br /><br />If you are looking for a superficial movie with good looking women just to have a relaxed evening, you should better go and see \"Charlie's Angels\" (it's much more funny, entertaining and self-ironic) instead of this flick.<br /><br />Two thumbs down (3 out of 10).   \n",
       "\n",
       "   sentiment  \n",
       "0  1          \n",
       "1  0          \n",
       "2  0          "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df.shape)\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Solving problems with machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mapping problems to machine learning solutions\n",
    "\n",
    "http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised learning\n",
    "\n",
    "The goal of supervised learning is to learn a function that maps an input to an output based on example input-output pairs. \n",
    "\n",
    "Examples:\n",
    "\n",
    "- House properties => Price\n",
    "\n",
    "postcode | land size (sqm) | #bedrooms | #bathrooms | dist 2 station (m) | price (millions)\n",
    "---|---|---|---|---|\n",
    "2000|1000|4|2|200|2M\n",
    "2000|500|4|2|200|1.5M\n",
    "2100|1000|3|1|1000|0.4M\n",
    "\n",
    "- News => Topic\n",
    "\n",
    "news headline (Vietnamese) | topic\n",
    "--|--\n",
    "Jonh McCain - Thượng nghị sĩ nhiều duyên nợ với Việt Nam | Politics\n",
    "Thí sinh Hoa hậu Việt Nam diễn bikini | Entertainment\n",
    "Syria tập kín trước khi quyết đấu Việt Nam | Sports / Politics?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text classification\n",
    "\n",
    "- Input is text (can be a document, a sentence of varied length)\n",
    "- Output is categorical -- binary classification if two categories, multi-class if multiple categories\n",
    "- Is one instance of supervised learning\n",
    "- Is also one of natural language processing (NLP) tasks\n",
    "\n",
    "#### Examples\n",
    "\n",
    "- Categorization of news articles into defined topics\n",
    "- Understanding audience sentiment from social media\n",
    "- Detection of spam and non-spam emails\n",
    "- Auto tagging of customer queries\n",
    "- Categorization of research papers into research topics\n",
    "\n",
    "#### Main steps\n",
    "\n",
    "- **Feature engineering**: Transform text input into a numeric feature vector (i.e. vectorize a document); features can also be categorical\n",
    "- **Modelling**: Train classification models as a standard classification problem with numeric features\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transforming documents into feature vectors\n",
    "\n",
    "Suppose we have these 2 documents. How would you convert them into feature vectors? \n",
    "\n",
    "- The sun is shining\n",
    "- The weather is not sweet\n",
    "- 사랑 해요\n",
    "\n",
    "First, think of an approach to work with 2 documents. Then consider if it works for a large number of documents.\n",
    "\n",
    "In principles:\n",
    "\n",
    "- Represent a word by a numerical encoding (there are many different ways).\n",
    "- Create the document's vector by combining the word encodings.\n",
    "- Each document should be represented by vector of same length. Why?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word indicators representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make all documents have a fixed same length. Longer documents are trimmed and shorter documents are padded. \n",
    "\n",
    "- The sun is shining DUMMY\n",
    "- The weather is not sweet \n",
    "\n",
    "Padded documents:\n",
    "\n",
    "word1 | word2 | word3 | word4 | word5\n",
    "--|--|--|\n",
    "the | sun | is | shining | DUMMY \n",
    "the | weather | is | not | sweet \n",
    "\n",
    "Then replace each word with its index (the = 1, sun = 2, is = 3, shining = 4, weather = 5, not = 6, sweet = 7, DUMMY = 8)\n",
    "\n",
    "Vectorized documents:\n",
    "\n",
    "id | word1 | word2 | word3 | word4 | word5\n",
    "--|--|--|--|--|\n",
    "doc1 | 1 | 2 | 3 | 4 | 8\n",
    "doc2 | 1 | 5 | 3 | 6 | 7\n",
    "\n",
    "## Exercise\n",
    "- Does this vectorization work?\n",
    "- What are the problems?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One-hot-encoding representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the = [1 0 0 0 0 0]\n",
    "\n",
    "sun = [0 1 0 0 0 0]\n",
    "\n",
    "is  = [0 0 1 0 0 0]\n",
    "\n",
    "shining = [0 0 0 1 0 0]\n",
    "\n",
    "weather = [0 0 0 0 1 0]\n",
    "\n",
    "sweet = [0 0 0 0 0 1]\n",
    "\n",
    "The sun is shining = [1 1 1 1 0 0]\n",
    "\n",
    "Is the sun shining = []?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag-of-words representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By calling the fit_transform method on CountVectorizer, we just constructed the vocabulary of the bag-of-words model and transformed the following three sentences into sparse feature vectors:\n",
    "1. The sun is shining\n",
    "2. The weather is sweet\n",
    "3. The sun is shining, the weather is sweet, and one and one is two\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer()\n",
    "docs = np.array([\n",
    "        'The sun is shining',\n",
    "        'The weather is sweet',\n",
    "        'The sun is shining, the weather is sweet, and one and one is two'])\n",
    "bag = count.fit_transform(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us print the contents of the vocabulary to get a better understanding of the underlying concepts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'the': 6, 'sun': 4, 'is': 1, 'shining': 3, 'weather': 8, 'sweet': 5, 'and': 0, 'one': 2, 'two': 7}\n"
     ]
    }
   ],
   "source": [
    "print(count.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from executing the preceding command, the vocabulary is stored in a Python dictionary, which maps the unique words that are mapped to integer indices. Next let us print the feature vectors that we just created:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each index position in the feature vectors shown here corresponds to the integer values that are stored as dictionary items in the CountVectorizer vocabulary. For example, the  rst feature at index position 0 resembles the count of the word and, which only occurs in the last document, and the word is at index position 1 (the 2nd feature in the document vectors) occurs in all three sentences. Those values in the feature vectors are also called the raw term frequencies: *tf (t,d)*—the number of times a term t occurs in a document *d*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 0 1 1 0 1 0 0]\n",
      " [0 1 0 0 0 1 1 0 1]\n",
      " [2 3 2 1 1 1 2 1 1]]\n"
     ]
    }
   ],
   "source": [
    "print(bag.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise \n",
    "\n",
    "- Which featurization seems to be better than the previous model? How?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tfidf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.set_printoptions(precision=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are analyzing text data, we often encounter words that occur across multiple documents from both classes. Those frequently occurring words typically don't contain useful or discriminatory information. In this subsection, we will learn about a useful technique called term frequency-inverse document frequency (tf-idf) that can be used to downweight those frequently occurring words in the feature vectors. The tf-idf can be de ned as the product of the term frequency and the inverse document frequency:\n",
    "\n",
    "$$\\text{tf-idf}(t,d)=\\text{tf (t,d)}\\times \\text{idf}(t,d)$$\n",
    "\n",
    "Here the tf(t, d) is the term frequency that we introduced in the previous section,\n",
    "and the inverse document frequency *idf(t, d)* can be calculated as:\n",
    "\n",
    "$$\\text{idf}(t,d) = \\text{log}\\frac{n_d}{1+\\text{df}(d, t)},$$\n",
    "\n",
    "where $n_d$ is the total number of documents, and *df(d, t)* is the number of documents *d* that contain the term *t*. Note that adding the constant 1 to the denominator is optional and serves the purpose of assigning a non-zero value to terms that occur in all training samples; the log is used to ensure that low document frequencies are not given too much weight.\n",
    "\n",
    "Scikit-learn implements yet another transformer, the `TfidfTransformer`, that takes the raw term frequencies from `CountVectorizer` as input and transforms them into tf-idfs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.   0.43 0.   0.56 0.56 0.   0.43 0.   0.  ]\n",
      " [0.   0.43 0.   0.   0.   0.56 0.43 0.   0.56]\n",
      " [0.5  0.45 0.5  0.19 0.19 0.19 0.3  0.25 0.19]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tfidf = TfidfTransformer(use_idf=True, \n",
    "                         norm='l2', \n",
    "                         smooth_idf=True)\n",
    "print(tfidf.fit_transform(count.fit_transform(docs))\n",
    "      .toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw in the previous subsection, the word is had the largest term frequency in the 3rd document, being the most frequently occurring word. However, after transforming the same feature vector into tf-idfs, we see that the word is is\n",
    "now associated with a relatively small tf-idf (0.45) in document 3 since it is\n",
    "also contained in documents 1 and 2 and thus is unlikely to contain any useful, discriminatory information.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we'd manually calculated the tf-idfs of the individual terms in our feature vectors, we'd have noticed that the `TfidfTransformer` calculates the tf-idfs slightly differently compared to the standard textbook equations that we de ned earlier. The equations for the idf and tf-idf that were implemented in scikit-learn are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\text{idf} (t,d) = log\\frac{1 + n_d}{1 + \\text{df}(d, t)}$$\n",
    "\n",
    "The tf-idf equation that was implemented in scikit-learn is as follows:\n",
    "\n",
    "$$\\text{tf-idf}(t,d) = \\text{tf}(t,d) \\times (\\text{idf}(t,d)+1)$$\n",
    "\n",
    "While it is also more typical to normalize the raw term frequencies before calculating the tf-idfs, the `TfidfTransformer` normalizes the tf-idfs directly.\n",
    "\n",
    "By default (`norm='l2'`), scikit-learn's TfidfTransformer applies the L2-normalization, which returns a vector of length 1 by dividing an un-normalized feature vector *v* by its L2-norm:\n",
    "\n",
    "$$v_{\\text{norm}} = \\frac{v}{||v||_2} = \\frac{v}{\\sqrt{v_{1}^{2} + v_{2}^{2} + \\dots + v_{n}^{2}}} = \\frac{v}{\\big (\\sum_{i=1}^{n} v_{i}^{2}\\big)^\\frac{1}{2}}$$\n",
    "\n",
    "To make sure that we understand how TfidfTransformer works, let us walk\n",
    "through an example and calculate the tf-idf of the word is in the 3rd document.\n",
    "\n",
    "The word is has a term frequency of 3 (tf = 3) in document 3, and the document frequency of this term is 3 since the term is occurs in all three documents (df = 3). Thus, we can calculate the idf as follows:\n",
    "\n",
    "$$\\text{idf}(\"is\", d3) = log \\frac{1+3}{1+3} = 0$$\n",
    "\n",
    "Now in order to calculate the tf-idf, we simply need to add 1 to the inverse document frequency and multiply it by the term frequency:\n",
    "\n",
    "$$\\text{tf-idf}(\"is\",d3)= 3 \\times (0+1) = 3$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf-idf of term \"is\" = 3.00\n"
     ]
    }
   ],
   "source": [
    "tf_is = 3\n",
    "n_docs = 3\n",
    "idf_is = np.log((n_docs+1) / (3+1))\n",
    "tfidf_is = tf_is * (idf_is + 1)\n",
    "print('tf-idf of term \"is\" = %.2f' % tfidf_is)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we repeated these calculations for all terms in the 3rd document, we'd obtain the following tf-idf vectors: [3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]. However, we notice that the values in this feature vector are different from the values that we obtained from the TfidfTransformer that we used previously. The  nal step that we are missing in this tf-idf calculation is the L2-normalization, which can be applied as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\text{tfi-df}_{norm} = \\frac{[3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]}{\\sqrt{[3.39^2, 3.0^2, 3.39^2, 1.29^2, 1.29^2, 1.29^2, 2.0^2 , 1.69^2, 1.29^2]}}$$\n",
    "\n",
    "$$=[0.5, 0.45, 0.5, 0.19, 0.19, 0.19, 0.3, 0.25, 0.19]$$\n",
    "\n",
    "$$\\Rightarrow \\text{tfi-df}_{norm}(\"is\", d3) = 0.45$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the results match the results returned by scikit-learn's `TfidfTransformer` (below). Since we now understand how tf-idfs are calculated, let us proceed to the next sections and apply those concepts to the movie review dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.39, 3.  , 3.39, 1.29, 1.29, 1.29, 2.  , 1.69, 1.29])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = TfidfTransformer(use_idf=True, norm=None, smooth_idf=True)\n",
    "raw_tfidf = tfidf.fit_transform(count.fit_transform(docs)).toarray()[-1]\n",
    "raw_tfidf "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.5 , 0.45, 0.5 , 0.19, 0.19, 0.19, 0.3 , 0.25, 0.19])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l2_tfidf = raw_tfidf / np.sqrt(np.sum(raw_tfidf**2))\n",
    "l2_tfidf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a modelling dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven.<br /><br />Title (Brazil): Not Available'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0, 'review'][-50:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "def preprocessor(text):\n",
    "    text = re.sub('<[^>]*>', '', text)\n",
    "    emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)',\n",
    "                           text)\n",
    "    text = (re.sub('[\\W]+', ' ', text.lower()) +\n",
    "            ' '.join(emoticons).replace('-', ''))\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven title brazil not available'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor(df.loc[0, 'review'][-50:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is a test :) :( :)'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor(\"</a>This :) is :( a test :-)!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['review'] = df['review'].apply(preprocessor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a logistic regression model for text classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = df.loc[:25000, 'review'].values\n",
    "y_train = df.loc[:25000, 'sentiment'].values\n",
    "X_test = df.loc[25000:, 'review'].values\n",
    "y_test = df.loc[25000:, 'sentiment'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization and Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "porter = PorterStemmer()\n",
    "\n",
    "def tokenizer(text):\n",
    "    return text.split()\n",
    "\n",
    "def tokenizer_porter(text):\n",
    "    return [porter.stem(word) for word in text.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runners', 'like', 'running', 'and', 'thus', 'they', 'run']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer('runners like running and thus they run')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runner', 'like', 'run', 'and', 'thu', 'they', 'run']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_porter('runners like running and thus they run')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /Users/trung/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runner', 'like', 'run', 'run', 'lot']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop = stopwords.words('english')\n",
    "[w for w in tokenizer_porter('a runner likes running and runs a lot')[-10:]\n",
    "if w not in stop]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model training and hyperparameters search "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "tfidf = TfidfVectorizer(strip_accents=None,\n",
    "                        lowercase=False,\n",
    "                        preprocessor=None,\n",
    "                        stop_words='english')\n",
    "\n",
    "param_grid = {\n",
    "    'vect__ngram_range': [(1, 1)],\n",
    "    'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "    'vect__use_idf': [False],\n",
    "    'vect__norm': [None],\n",
    "    'clf__penalty': ['l1', 'l2'],\n",
    "    'clf__C': [1.0, 10.0, 100.0]\n",
    "}\n",
    "\n",
    "lr_tfidf = Pipeline([('vect', tfidf),\n",
    "                     ('clf', LogisticRegression(random_state=0))])\n",
    "\n",
    "rs_lr_tfidf = RandomizedSearchCV(lr_tfidf, param_grid,\n",
    "                           scoring='accuracy',\n",
    "                           cv=3,\n",
    "                           n_iter=10,\n",
    "                           verbose=1,\n",
    "                           n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important Note about `n_jobs`**\n",
    "\n",
    "Please note that it is highly recommended to use `n_jobs=-1` (instead of `n_jobs=1`) in the previous code example to utilize all available cores on your machine and speed up the grid search. However, some Windows users reported issues when running the previous code with the `n_jobs=-1` setting related to pickling the tokenizer and tokenizer_porter functions for multiprocessing on Windows. Another workaround would be to replace those two functions, `[tokenizer, tokenizer_porter]`, with `[str.split]`. However, note that the replacement by the simple `str.split` would not support stemming."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  30 out of  30 | elapsed: 36.5min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=3, error_score='raise',\n",
       "          estimator=Pipeline(memory=None,\n",
       "     steps=[('vect', TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=False, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), norm='l2', preprocessor=None, smooth_idf=True,\n",
       " ...nalty='l2', random_state=0, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False))]),\n",
       "          fit_params=None, iid=True, n_iter=10, n_jobs=-1,\n",
       "          param_distributions={'vect__ngram_range': [(1, 1)], 'vect__tokenizer': [<function tokenizer at 0x10eb22510>, <function tokenizer_porter at 0x10eb222f0>], 'vect__use_idf': [False], 'vect__norm': [None], 'clf__penalty': ['l1', 'l2'], 'clf__C': [1.0, 10.0, 100.0]},\n",
       "          pre_dispatch='2*n_jobs', random_state=None, refit=True,\n",
       "          return_train_score='warn', scoring='accuracy', verbose=1)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rs_lr_tfidf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameter set: {'vect__use_idf': False, 'vect__tokenizer': <function tokenizer at 0x10eb22510>, 'vect__norm': None, 'vect__ngram_range': (1, 1), 'clf__penalty': 'l2', 'clf__C': 1.0} \n",
      "CV Accuracy: 0.875\n"
     ]
    }
   ],
   "source": [
    "print('Best parameter set: %s ' % rs_lr_tfidf.best_params_)\n",
    "print('CV Accuracy: %.3f' % rs_lr_tfidf.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 0.882\n"
     ]
    }
   ],
   "source": [
    "clf = rs_lr_tfidf.best_estimator_\n",
    "print('Test Accuracy: %.3f' % clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with bigger data - online algorithms and out-of-core learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell is not contained in the book but\n",
    "# added for convenience so that the notebook\n",
    "# can be executed starting here, without\n",
    "# executing prior code in this notebook\n",
    "\n",
    "import os\n",
    "import gzip\n",
    "\n",
    "\n",
    "if not os.path.isfile('movie_data.csv'):\n",
    "    if not os.path.isfile('movie_data.csv.gz'):\n",
    "        print('Please place a copy of the movie_data.csv.gz'\n",
    "              'in this directory. You can obtain it by'\n",
    "              'a) executing the code in the beginning of this'\n",
    "              'notebook or b) by downloading it from GitHub:'\n",
    "              'https://github.com/rasbt/python-machine-learning-'\n",
    "              'book-2nd-edition/blob/master/code/ch08/movie_data.csv.gz')\n",
    "    else:\n",
    "        with in_f = gzip.open('movie_data.csv.gz', 'rb'), \\\n",
    "                out_f = open('movie_data.csv', 'wb'):\n",
    "            out_f.write(in_f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "\n",
    "# The `stop` is defined as earlier in this chapter\n",
    "# Added it here for convenience, so that this section\n",
    "# can be run as standalone without executing prior code\n",
    "# in the directory\n",
    "stop = stopwords.words('english')\n",
    "\n",
    "\n",
    "def tokenizer(text):\n",
    "    text = re.sub('<[^>]*>', '', text)\n",
    "    emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)', text.lower())\n",
    "    text = re.sub('[\\W]+', ' ', text.lower()) +\\\n",
    "        ' '.join(emoticons).replace('-', '')\n",
    "    tokenized = [w for w in text.split() if w not in stop]\n",
    "    return tokenized\n",
    "\n",
    "\n",
    "def stream_docs(path):\n",
    "    with open(path, 'r', encoding='utf-8') as csv:\n",
    "        next(csv)  # skip header\n",
    "        for line in csv:\n",
    "            text, label = line[:-3], int(line[-2])\n",
    "            yield text, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('\"In 1974, the teenager Martha Moxley (Maggie Grace) moves to the high-class area of Belle Haven, Greenwich, Connecticut. On the Mischief Night, eve of Halloween, she was murdered in the backyard of her house and her murder remained unsolved. Twenty-two years later, the writer Mark Fuhrman (Christopher Meloni), who is a former LA detective that has fallen in disgrace for perjury in O.J. Simpson trial and moved to Idaho, decides to investigate the case with his partner Stephen Weeks (Andrew Mitchell) with the purpose of writing a book. The locals squirm and do not welcome them, but with the support of the retired detective Steve Carroll (Robert Forster) that was in charge of the investigation in the 70\\'s, they discover the criminal and a net of power and money to cover the murder.<br /><br />\"\"Murder in Greenwich\"\" is a good TV movie, with the true story of a murder of a fifteen years old girl that was committed by a wealthy teenager whose mother was a Kennedy. The powerful and rich family used their influence to cover the murder for more than twenty years. However, a snoopy detective and convicted perjurer in disgrace was able to disclose how the hideous crime was committed. The screenplay shows the investigation of Mark and the last days of Martha in parallel, but there is a lack of the emotion in the dramatization. My vote is seven.<br /><br />Title (Brazil): Not Available\"',\n",
       " 1)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(stream_docs(path='movie_data.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_minibatch(doc_stream, size):\n",
    "    docs, y = [], []\n",
    "    try:\n",
    "        for _ in range(size):\n",
    "            text, label = next(doc_stream)\n",
    "            docs.append(text)\n",
    "            y.append(label)\n",
    "    except StopIteration:\n",
    "        return None, None\n",
    "    return docs, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "\n",
    "vect = HashingVectorizer(decode_error='ignore', \n",
    "                         n_features=2**21,\n",
    "                         preprocessor=None, \n",
    "                         tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "- You can replace `Perceptron(n_iter, ...)` by `Perceptron(max_iter, ...)` in scikit-learn >= 0.19."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from distutils.version import LooseVersion as Version\n",
    "from sklearn import __version__ as sklearn_version\n",
    "\n",
    "\n",
    "if Version(sklearn_version) < '0.18':\n",
    "    clf = SGDClassifier(loss='log', random_state=1, n_iter=1)\n",
    "else:\n",
    "    clf = SGDClassifier(loss='log', random_state=1, max_iter=1)\n",
    "\n",
    "\n",
    "doc_stream = stream_docs(path='movie_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0% [##############################] 100% | ETA: 00:00:00\n",
      "Total time elapsed: 00:00:35\n"
     ]
    }
   ],
   "source": [
    "import pyprind\n",
    "pbar = pyprind.ProgBar(45)\n",
    "\n",
    "classes = np.array([0, 1])\n",
    "for _ in range(45):\n",
    "    X_train, y_train = get_minibatch(doc_stream, size=1000)\n",
    "    if not X_train:\n",
    "        break\n",
    "    X_train = vect.transform(X_train)\n",
    "    clf.partial_fit(X_train, y_train, classes=classes)\n",
    "    pbar.update()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.867\n"
     ]
    }
   ],
   "source": [
    "X_test, y_test = get_minibatch(doc_stream, size=5000)\n",
    "X_test = vect.transform(X_test)\n",
    "print('Accuracy: %.3f' % clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}