3.1-sparse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sparse matrices\n",
    "\n",
    "somewhat you (probably) will need to work with the first competition efficiently.\n",
    "\n",
    "\n",
    "The scipy.sparse module provides data structures for 2D sparse matrices. There are [several](http://www.scipy-lectures.org/advanced/scipy_sparse/storage_schemes.html) available sparse matrix types, we'll use two of them: \n",
    "\n",
    "1. csc_matrix: Compressed Sparse Column \n",
    "2. csr_matrix: Compressed Sparse Row \n",
    "\n",
    "Actually, we already used them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some copy-paste from notebook 1.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "from scipy import sparse\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import codecs\n",
    "# python kung fu. No need to understand this code\n",
    "with codecs.open('./data/1-restaurant-train.csv') as f:\n",
    "    labels, reviews = zip(*[line.split('\\t') for line in f.readlines()])\n",
    "with codecs.open('./data/1-restaurant-test.csv') as f:\n",
    "    kaggle_test_reviews = f.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "from IPython.display import FileLink\n",
    "# supplementary function. Again, just use it\n",
    "def create_solution(predictions, filename='1-restaurant-predictions.csv'):\n",
    "    result = pandas.DataFrame({'Id': numpy.arange(len(predictions)), 'Solution': predictions})\n",
    "    result.to_csv('data/{}'.format(filename), index=False)\n",
    "    return FileLink('data/{}'.format(filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "answers = numpy.array(labels, dtype=int) >= 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "# take 100000 most frequent words\n",
    "vectorizer = CountVectorizer(max_features=100000)\n",
    "vectorizer.fit(reviews)\n",
    "\n",
    "counts = vectorizer.transform(reviews) #.toarray()\n",
    "kaggle_test_counts = vectorizer.transform(kaggle_test_reviews) #.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<82065x92353 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 9493523 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### output of vectorizer is sparse ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       ..., \n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# take first 100 lines, convert to numpy\n",
    "counts[:100].toarray() \n",
    "# but doing above operation with whole dataset is a very bad idea: ~10 ** 10 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "798.328391367\n"
     ]
    }
   ],
   "source": [
    "n_elements = counts.shape[0] * counts.shape[1]\n",
    "n_nonzero_elements = counts.getnnz()\n",
    "\n",
    "# real amount of elements is ~1000 times less\n",
    "print n_elements / float(n_nonzero_elements)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<82065x92353 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 9493523 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<92353x82065 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 9493523 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000 loops, best of 3: 41.7 µs per loop\n"
     ]
    }
   ],
   "source": [
    "# we can see that transposing is instant, because it simply changed rows and columns.\n",
    "%timeit counts.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14623064"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[2234,  349,    1, ...,    1,    1,    1]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2234,  349,    1, ...,    1,    1,    1])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# converted to numpy.array, then converted from matrix to a single row\n",
    "numpy.array(counts.sum(axis=0))[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Adding new features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<82065x1 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 82065 stored elements in COOrdinate format>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.max(axis=1) # max times one word was repeated in the review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_new_features(sparse_counts):\n",
    "    # added two new features to the dataset: number of words in the sentence and 'max repetitions'\n",
    "    # horizontal stacking of matrices. Important - second and third are also matrices, but with one column\n",
    "    return sparse.hstack([sparse_counts, sparse_counts.sum(axis=1), sparse_counts.max(axis=1)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "new_counts = add_new_features(counts)\n",
    "# the same operations should be done on kaggle test (to use the same model)\n",
    "new_kaggle_counts = add_new_features(kaggle_test_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Also let's add count of different characters\n",
    "\n",
    "why not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def add_character_count(counts, reviews, characters):\n",
    "    new_features = numpy.zeros(shape=[len(reviews), len(characters)])\n",
    "    for i, character in enumerate(characters):\n",
    "        for j in range(len(reviews)):\n",
    "            new_features[j, i] = reviews[j].lower().count(character)\n",
    "    print new_features\n",
    "    return sparse.hstack([counts, new_features])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   0.    0.    7.    1.  106.]\n",
      " [   0.    0.    1.    0.  158.]\n",
      " [   0.    0.    0.    0.  142.]\n",
      " ..., \n",
      " [   1.    0.    0.    0.  213.]\n",
      " [   0.    0.    0.    0.  115.]\n",
      " [   2.    2.    0.    0.  104.]]\n",
      "[[   1.    1.    1.    0.   96.]\n",
      " [   0.    0.    3.    0.  154.]\n",
      " [   0.    0.    0.    0.  149.]\n",
      " ..., \n",
      " [   0.    0.    0.    2.  317.]\n",
      " [   1.    1.    0.    2.  105.]\n",
      " [   0.    0.    0.    0.  189.]]\n"
     ]
    }
   ],
   "source": [
    "characters = [')', '(', '!', 'like', ' ']\n",
    "new_counts = add_character_count(new_counts, reviews, characters=characters)\n",
    "new_kaggle_counts = add_character_count(new_kaggle_counts, kaggle_test_reviews, characters=characters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train / test splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/antares/.virtualenvs/rep/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "train_counts, test_counts, train_answers, test_answers = \\\n",
    "    train_test_split(new_counts, answers, train_size=10000, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<10000x92360 sparse matrix of type '<type 'numpy.float64'>'\n",
       "\twith 1209508 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_counts # works correctly with sparse types too!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training models\n",
    "\n",
    "many (but not all!) models in scikit-learn work efficiently with sparse data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train:  0.999995552399\n",
      "test:   0.879890597236\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<a href='data/1-restaurant-predictions-logreg.csv' target='_blank'>data/1-restaurant-predictions-logreg.csv</a><br>"
      ],
      "text/plain": [
       "/Users/antares/Yandex.Disk.localized/2017-01-Imperial/data/1-restaurant-predictions-logreg.csv"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# logistic regression was explained in the lectures.\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# C is an inverse regularization, smaller C -> stronger regularization\n",
    "logreg_clf = LogisticRegression(C=1.)\n",
    "logreg_clf.fit(train_counts, train_answers)\n",
    "\n",
    "# important: logistic regression is a CLASSIFIER, so we use .predict_proba!\n",
    "print 'train: ', roc_auc_score(train_answers, logreg_clf.predict_proba(train_counts)[:, 1])\n",
    "print 'test:  ', roc_auc_score(test_answers, logreg_clf.predict_proba(test_counts)[:, 1])\n",
    "\n",
    "create_solution(logreg_clf.predict_proba(new_kaggle_counts)[:, 1], filename='1-restaurant-predictions-logreg.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train:  0.903638617114\n",
      "test:   0.87573470998\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<a href='data/1-restaurant-predictions-ridge.csv' target='_blank'>data/1-restaurant-predictions-ridge.csv</a><br>"
      ],
      "text/plain": [
       "/Users/antares/Yandex.Disk.localized/2017-01-Imperial/data/1-restaurant-predictions-ridge.csv"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# elastic net is a linear regression with both L1 and L2 regularizations, it's more general compared to Ridge\n",
    "# sidenote: logistic regression also has L1 regularization\n",
    "from sklearn.linear_model import ElasticNet, Ridge\n",
    "\n",
    "# ridge_reg = ElasticNet(l1_ratio=0.5, alpha=0.1, max_iter=5)\n",
    "ridge_reg = Ridge()\n",
    "ridge_reg.fit(train_counts, train_answers)\n",
    "\n",
    "# Ridge (and ElasticNet) is a regression method, so we use .predict method\n",
    "print 'train: ', roc_auc_score(train_answers, ridge_reg.predict(train_counts))\n",
    "print 'test:  ', roc_auc_score(test_answers, ridge_reg.predict(test_counts))\n",
    "\n",
    "create_solution(ridge_reg.predict(new_kaggle_counts), filename='1-restaurant-predictions-ridge.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# References\n",
    "\n",
    "- [scipy documentation on sparse matrices](https://docs.scipy.org/doc/scipy-0.18.1/reference/sparse.html)\n",
    "    - more verbose reference can be found in [scipy lectures](http://www.scipy-lectures.org/advanced/scipy_sparse/index.html)\n",
    "- ask in the gitter chat if you need something specific"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}