Scikit- Tutorial Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "\n",
    "## Binary Classification\n",
    "In linear regression we get a continuos real values output giving over the whole range of input which is useful for application like prediction of real valued output such as prices of house or stocks but these real valued output are not useful when it comes to get the probability of a particular set of features.\n",
    "\n",
    "Logistic regression is thus used to get probability of getting a class based on the whether a feature set belongs to a particular class or not.\n",
    "\n",
    "In logistic regression we take the output of linear regression and feed it as a input to logit ffunction which gives a real valued output in range of [0-1]. This can be thought of as probability of the feature either being 0 class or 1 class.\n",
    "\n",
    "Logistic function: \n",
    "![Image of logistic Function](https://analyticsdefined.com/wp-content/uploads/2017/10/sigmoid-function1.gif)\n",
    "\n",
    "Logistic Regression is when we feed in the linear regression to logistic function:\n",
    "\n",
    "![Image of Logistic regression](http://faculty.cas.usf.edu/mbrannick/regression/gifs/lo3.gif)\n",
    "\n",
    "### Application: Spam Filtering\n",
    "\n",
    "Our first problem is a modern version of the canonical binary classification problem: spam classification. In our version, however, we will classify spam and ham SMS messages rather than e-mail. We will extract TF-IDF features and classify the messages using logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                 Msg Class\n",
      "0  Go until jurong point, crazy.. Available only ...   ham\n",
      "1                      Ok lar... Joking wif u oni...   ham\n",
      "2  Free entry in 2 a wkly comp to win FA Cup fina...  spam\n",
      "3  U dun say so early hor... U c already then say...   ham\n",
      "4  Nah I don't think he goes to usf, he lives aro...   ham\n",
      "Total Ham:  4825\n",
      "Toatl Spam:  747\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.read_csv('SMSSpamCollection', delimiter='\\t', names=['Class', 'Msg'])\n",
    "#notice the delimiter option here\n",
    "#default delimiter used is ',' but in the dataset \\t is used as separator\n",
    "df = df[['Msg', 'Class']]\n",
    "df['Msg'] = df['Msg'].apply(lambda x: x.strip())\n",
    "print df.head()\n",
    "print 'Total Ham: ',df[df['Class']=='ham']['Class'].count()\n",
    "print 'Toatl Spam: ',df[df['Class']=='spam']['Class'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "X_train_raw, X_test_raw, y_train, y_test = train_test_split(df['Msg'], df['Class'])\n",
    "vectorizer = TfidfVectorizer()\n",
    "X_train = vectorizer.fit_transform(X_train_raw)\n",
    "X_test = vectorizer.transform(X_test_raw)\n",
    "clf = LogisticRegression()\n",
    "clf.fit(X_train, y_train)\n",
    "prediction = [ clf.predict(i)[0] for i in X_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1393\n"
     ]
    }
   ],
   "source": [
    "predicted = []\n",
    "print len(prediction)\n",
    "for i, output in enumerate(prediction):\n",
    "    predicted.append('Prediction: %s. Message: %s' % (output, X_test_raw.iloc[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction: ham. Message: Daddy, shu shu is looking 4 u... U wan me 2 tell him u're not in singapore or wat?\n",
      "Prediction: ham. Message: Good afternoon on this glorious anniversary day, my sweet J !! I hope this finds you happy and content, my Prey. I think of you and send a teasing kiss from across the sea coaxing images of fond souveniers ... You Cougar-Pen\n",
      "Prediction: ham. Message: Ok. C u then.\n",
      "Prediction: ham. Message: Ya they are well and fine., BBD(pooja) full pimples..even she become quite black..and ur rite here its too cold, wearing sweatter..\n",
      "Prediction: ham. Message: He says hi and to get your ass back to south tampa (preferably at a kegger)\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print predicted[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary classification performance metrics\n",
    "\n",
    "A variety of metrics exist to evaluate the performance of binary classifiers against trusted labels. The most common metrics are accuracy, precision, recall, F1 measure,and ROC AUC score. All of these measures depend on the concepts of true positives, true negatives, false positives, and false negatives. Positive and negative refer to the classes. True and false denote whether the predicted class is the same as the true class.\n",
    "\n",
    "For our SMS spam classifier, a true positive prediction is when the classifier correctly predicts that a message is spam. A true negative prediction is when the classifier correctly predicts that a message is ham. A prediction that a ham message is spam is a false positive prediction, and a spam message incorrectly classified as ham is a\n",
    "false negative prediction.\n",
    "\n",
    "### Confusion Matrix\n",
    "\n",
    "A **confusion matrix**, or **contingency table**, can be used to visualize true and false positives and negatives. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1213    1]\n",
      " [  31  148]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix = confusion_matrix(y_test, prediction)\n",
    "print confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARIAAAD0CAYAAABXXakIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAGSdJREFUeJzt3Xu0HWV9xvHvk0QINxNIMEIIJpWIK7IMQoqg1aJYCEgJWkCQSsC4WCreilqD2nLxXi8oS7SNBuXWyKVSUojEGIlKC0iiEcLFJMKihAZCAgTlFnLOr3/Me2Q4nMvsPXtn9uX5sGadmXfePfPO5uR33svMO4oIzMzKGFF1Acys/TmQmFlpDiRmVpoDiZmV5kBiZqU5kJhZaQ4kZlaaA0kLkbSDpP+StFnSVSWOc7KknzaybM0m6dOSvl91Oaw+DiR1kPRuScsl/UnSekk/kfRXDTj0ccAEYFxEHF/vQSLi8og4vAHleQFJh0oKSdf0S5+e0pfVe+yI+GJEvK90IbucpIskbZC0Kpf2VUn3SLpd0jWSxub2nSVpraTfSzoilz4zpa2VNHe4845q/KV0NklnAnOB9wOLgS3ATGAWcFPJw78CWB0RW0sep5keAQ6RNC4iNqW02cDqeg8oaVSLX3PTHPGWnWLToz2F8q64/dnFETFzmGw/BL4NXJJLWwKcFRFbJX0FOAv4lKRpwInAa4A9gZ9JelX6zIXA3wDrgNskLYyIuwY9a0R4KbgAY4A/AccPkWd74JvA/6Xlm8D2ad+h6X/Mx4ENwHrgtLTvXLKg9Fw6xxzgHOCy3LEnAwGMStunAvcCfwTuA07Opd+U+9wbgNuAzennG3L7lgGfA/47HeenwPhBrq2v/P8KnJHSRgIPAv8MLMvl/RbwAPAEsAJ4U27fOcDVwGVp//vy1wq8K13PS9P2kcBDwO5V/w40ejngtdvHc+tfWWgBlhf8PZ0MrBpk3zuAy9P6WWQBpm/fYuCQtCzOpb8g30CLmza1OQQYDVwzRJ7PAAcD+wPTgYOAz+b2v5wsIE0kCxYXSto1Is4GvghcERE7R8T8oQoiaSfgAuDIiNiFLFisHCDfbsD1Ke844BvA9ZLG5bK9GzgNeBmwHfCJoc5N9tfulLR+BLCKLGjm3Ub2HewG/DtwlaTRuf2zyILJWODy/Acj4grgf4ALUjnnA++LiEeGKVcbCnqit9ACjE9N6r7l9DpO+F7gJ2l9Ilmw77MupQ2WPigHktqMAzbG0NXwk4HzImJD+sU/F3hPbv9zaf9zEbGIrPaxb53l6QX2k7RDRKyPiDsHyPN2YE1EXBoRWyNiAXAP8Le5PD+IiNUR8TRwJVkAGFRE/A+wm6R9yQLKJQPkuSwiNqVzfp2sppa/zpsj4j8jojedt78zgLeS1Zj+KyKuG6pM7SqAXqLQQva7NyO3zKvlXJI+A2ylX+BuBAeS2mwi+6swVN/SnsD9ue37U9qfj9EvED0F7FxrQSLiSbImwPuB9ZKul/TqAuXpK1P+L8xDdZTnUuBDwFsYoIYm6ROS7k4jUI+T1cLG57I80P8zeRHxOHAVsB/w9QLlaVu9Bf8rQ9KpwNFkzd++R/4fBCblsu2V0gZLH5QDSW1uBp4Fjh0iz/+RdZr22ZsXV/uLehLYMbf98vzOiFgcEX8D7EFWy/hegfL0lWnIX4wCLgU+CCyKiKfyOyS9CfhH4ARg14gYS9Y/o3zxhzq4pP3JquELyJplHSkIeqLYUi9JM8n+fxzT7//VQuBESdtLmgJMBX5N1iydKmmKpO3IOmQXDnUOB5IaRMRmsk7FCyWdK2l1Gh67SNK/pGwLgM9K2l3S+JT/sjpPuRJ4s6S9JY0h6/QCQNIESbNSX8mzZE2kgf5sLQJelYasR0l6FzANKNVUiIj7gL8m6xPqbxeyKvQjwChJ/wy8tOixU1/KjWSjitOBiZI+WKa8rayGps2wJC0g+4O3r6R1kuaQjeLsAiyRtFLSvwKkpvCVwF3ADWQd6D2pxvwhss7Xu4ErB2k2/5mHf2sUEV+X9DDwfbJ/uE+Q3f/x/pTl82T/aG5P21eltHrOtUTSFelYG4GvAMek3SOAM8n6J4Is6HxggGNsknQ02SjKd4G1wNERsbGeMvU79mDD3YvJfjFXk9WqzmeYpkw/XwJ+T1bjuQT4e+BGSUsiYk39JW49AfQUDBKFjhdx0gDJg3bcR8QXgC8MkL6I7I9QIYoSVaZuJekQ4JyIOCJtnwUQEV+qtGAdRtJk4LqI2K/iojTN9OnbxeJF44fPCOyx1/oVETGjyUWqi5s29al5eMxsML0Fl1bmpo1ZhYJoaNOmKg4k9al5eMxsQAE97R9H3LSpU83DY2YDyW5Ia/+mjQNJHeoZHrPaDDKM2YFET8GllblpU6dah8esNoMMY3acAHo7oGnjQGJWoQC2dEDDwIHErGK90drNliIcSMwqlN3Z6kBiZiUEoqcDmjbtfwUVqnNiGatBN3zHvaFCSytzICmn43/JW0BHf8d9TRsP/5pZCaIn2v/veUsFkvG7jYzJk15SdTEK23viKGZMH91WdwGsvn3H4TO1kNHsyEu1W1t9x8/wJFvi2UJViACeY2STS9R8LRVIJk96Cb9ePGn4jFa3I/YccjpWa4BbY2nhvBGukZhZA/S2eP9HEQ4kZhXKOltdIzGzUty0MbOSsmkEHEjMrIRAbAmP2phZSb1u2phZGe5sNbPSAtHT4s/RFOFAYlYxd7aaWSkRePjXzMqS72w1s3IC2BLt/8+w/etUZm0sKDapUdGJjSRdJGmDpFW5tN0kLZG0Jv3cNaVL0gWS1kq6XdIBuc/MTvnXSJo93HkdSMwq1sOIQktBPwRm9kubCyyNiKnA0rQNcCQwNS2nA9+FLPAAZwOvBw4Czu4LPoNxIDGrUPZemxGFlkLHi/gl8Gi/5FnAxWn9YuDYXPolkbkFGCtpD+AIYElEPBoRjwFLeHFweoH2b5yZtbWaplEcL2l5bnteRMwr8LkJEbE+rT8ETEjrE4EHcvnWpbTB0gflQGJWob4aSUEbI2JGqfNFhKSGzzjnpo1ZxbbB5M8PpyYL6eeGlP4gkJ+ScK+UNlj6oBxIzCoUIZ7rHVVoKWEh0DfyMhu4Npd+Shq9ORjYnJpAi4HDJe2aOlkPT2mDctPGrELZfCSNuyFN0gLgULL+lHVkoy9fBq6UNAe4HzghZV8EHAWsBZ4CTgOIiEclfQ64LeU7LyL6d+C+gAOJWaUaO0NaRJw0yK7DBsgbwBmDHOci4KKi53UgMatQ1tnqW+TNrCTPR2JmpfTdIt/uHEjMKub5SMyslAh4rteBxMxKyJo2DiRmVlLJu1ZbggOJWYU8/GtmDeCmjZk1gOdsNbNSslnkHUjMrIRAbO31u3/NrCQ3bcysFI/amFlDeNTGzMqp4Z01rcyBxKxCjZ4hrSoOJGYVc43EzEoJYGsHPP3b1CuQNFPS79O7RecO/wmz7tLod/9WpWmBRNJI4EKy94tOA06SNK1Z5zNrV72o0NLKmlkjOQhYGxH3RsQW4Edk7xo1sz6BayTDKPT+UEmnS1ouafkjm3qaWByz1tN3Q5oDSUkRMS8iZkTEjN3Htf8zB2a16oRA0sxRm5rfH2rWbQLR41GbId0GTJU0RdJ2wIlk7xo1s5xO6GxtWo0kIrZK+hDZy4dHAhdFxJ3NOp9ZO4rwDWnDiohFZC8qNrNBRAcEkvZvnJm1tcbekCbpHyTdKWmVpAWSRqfuhVvTjaFXpK4GJG2fttem/ZPrvQoHErOKRajQMhxJE4GPADMiYj+yLoUTga8A50fEPsBjwJz0kTnAYyn9/JSvLg4kZhVqwn0ko4AdJI0CdgTWA28Frk77LwaOTeuz0jZp/2GS6mpnOZCYVSlN/lxkAcb33byZltNfcKiIB4GvAf9LFkA2AyuAxyNia8qWvzH0zzeNpv2bgXH1XIaf/jWrUFBTZ+vGiJgx2E5Ju5LVMqYAjwNXATPLlrEIBxKzSjX0rtW3AfdFxCMAkn4MvBEYK2lUqnXkbwztu2l0XWoKjQE21XNiN23MKhZRbCngf4GDJe2Y+joOA+4CbgSOS3lmA9em9YVpm7T/5xEFz9SPayRmFWvUfSQRcaukq4HfAFuB3wLzgOuBH0n6fEqbnz4yH7hU0lrgUbIRnro4kJhVKKttNO6GtIg4Gzi7X/K9ZNN69M/7DHB8I87rQGJWMd8ib2al9fY6kJhZCUGxu1ZbnQOJWcXqGiZpMQ4kZlVqcGdrVRxIzKrWAVWSQQOJpGsY4hIj4p1NKZFZl+n0Gsm3t1kpzLpYffeStpZBA0lELO1bTxOh7B0Ra7dJqcy6RAREN0z+LOntwB3AkrS9f2r2mFkDNPBZm8oUCYXnAa8neyyZiFgJ7NPMQpl1lSi4tLAiozbPRcTj/SZOavHLMmsX3XND2t2STgBGSJpCNifkLc0tllkX6YA/y0WaNh8CDgR6gWuALcDHmlkos64RjZv8uUrD1kgi4kngU5LOzTbj6eYXy6yLdEONRNIBkn4LrAbWSFoh6YDmF82sS4SKLS2sSB/JD4CPRcSNAJIOTWnTm1gus+7RATWSIoGkty+IAETEMkm9TSyTWfcIWr62UcRQz9q8Nq0uk3QhsIDsst8F/HwblM2sK7T6zWZFDFUjubDf9mtz6x1w6WYtogP+NQ31rM2btmVBzLpWJzdt8iQdAbwGGN2XFhFfbFahzLpGQCf0OA4bSCR9BxgLvJlstObv8J2tZg3S+kO7RRS5s/WvIuLdwKaI+CeyB/j80J5Zo3TAQ3tFAknfnazPSHo58AywZ/OKZNZlOiCQFOkj+YmkscDXgJVAD3BxU0tl1k1aPEgUUeRZm3PS6lWSrgN2AKY0s1BmXaPTb0gbSHpg72lJK4G9m1Mks+6iDqiR1DtZZPuHULNW0cA+EkljJV0t6R5Jd0s6RNJukpZIWpN+7pryStIFktZKur3Mw7j1vtemKTF0zR07ceRfHNyMQ1sycporks2mP9xUW/7G/mv6FnBDRByXJm3fEfg0sDQivixpLjAX+BRwJDA1La8Hvpt+1qye99oIGFfPycxsAA3qI5E0hux+r1MBImILsEXSLODQlO1iYBlZIJkFXBIRAdySajN7RMT6Ws9d73tt/M4bs0aobWh3vKTlue15ETEvtz0FeAT4gaTpwArgo8CEXHB4CJiQ1icCD+Q+vy6lNS6Q5N9rY2ZNVDyQbIyIGUPsHwUcAHw4Im6V9C2yZszzp4oIqfHdu+3/Zh6zNqcothSwDlgXEbem7avJAsvDkvYASD83pP0PApNyn98rpdXMgcSsag0atYmIh4AHJO2bkg4D7gIWArNT2mzg2rS+EDgljd4cDGyup38Eahi1kbR9RDxbz0nMbGBq/NO/HwYuTyM29wKnkVUYrpQ0B7gfOCHlXQQcBawFnkp561Lk6d+DgPnAGGDv1Inzvoj4cL0nNbOcBt7Zmt6EOVA/ymED5A3gjEact0jT5gLgaGBTOvnvgLc04uRmRtc8tDciIu7v98rOniaVx6zrdMIt8kUCyQOpeROSRpK1wVY3t1hmXaRLAskHyJo3ewMPAz9LaWZWVvGh3ZZWZBqBDcCJ26AsZt2pGwKJpO8xwKVGxOlNKZFZl+mKyZ/JmjJ9RgPv4IX355tZlyvStLkivy3pUqC256TNbHDd0LQZwBSef3rQzMrols5WSY/xfMwcATxKvycKzayETg8kyu5Cm87zTwT2pttqzaxROuBf1JC3yKegsSgietLSAZds1jpEQ6cRqEyRZ21WSnpd00ti1o3S079FllY21JytoyJiK/A64DZJfwCeJAuiERF1zzhtZjktXtsoYqg+kl+Tza50zDYqi1l36vBAIoCI+MM2KotZV2r1/o8ihgoku0s6c7CdEfGNJpTHrPt0eCAZCeyM36pn1jxtMGlREUMFkvURcd42K4lZl2r1EZkihu0jMbPm6vQ+khdNFmtmTdDJgSQiHt2WBTHrSl3QR2JmTSY6ow/BgcSsaq6RmFlZnd7ZambbQocP/5pZs7XBFAFFFJlGwMyaqcGv7JQ0UtJvJV2XtqdIulXSWklXpBeMI2n7tL027Z9c7yU4kJhVrAkTG30UuDu3/RXg/IjYB3gMmJPS5wCPpfTzU766OJCYVa2BNRJJewFvB76ftgW8Fbg6ZbkYODatz0rbpP2Hqd9LvotyIDGrWINrJN8E/pHnu3DHAY+nScoA1gET0/pE0juq0v7NKX/NHEjMqlS0NpIFkvGSlueWF7ztUtLRwIaIWLHtLiDjURuzComanv7dGBEzhtj/RuAYSUeRvRXzpcC3gLG5qVP34vm3QjwITALWSRoFjAE21XwRuEZiVr0G9ZFExFkRsVdETAZOBH4eEScDNwLHpWyzgWvT+sK0Tdr/83rfFNG0QCLpIkkbJK1q1jnMOoEiCi0lfAo4U9Jasj6Q+Sl9PjAupZ9JiRffNbNp80Pg28AlTTyHWXtr0tO/EbEMWJbW7wUOGiDPM8DxjThf0wJJRPyyzA0uZt2iE+5srbyzNfU8nw4wWjtVXBqzCjiQlBcR84B5AGNGjOuAr9SsNq6RmFk50fmTP5vZttABNZJmDv8uAG4G9pW0TtKc4T5j1m1EUx7a2+aaOWpzUrOObdZRyt0j0hLctDGrWKvXNopwIDGrkl9HYWaN4FEbMyvNgcTMygnc2Wpm5bmz1czKcyAxszL6bkhrdw4kZlWKcB+JmZXnURszK81NGzMrJ4De9o8kDiRmVWv/OOJAYlY1N23MrDyP2phZWa6RmFkpCpA7W82sNN9HYmZllXwdZ0twIDGrkmdIM7PyOuNZm6a9jsLMimnU6ygkTZJ0o6S7JN0p6aMpfTdJSyStST93TemSdIGktZJul3RAvdfgQGJWtb4ngIdbhrcV+HhETAMOBs6QNA2YCyyNiKnA0rQNcCQwNS2nA9+t9xIcSMyqFKCeKLQMe6iI9RHxm7T+R+BuYCIwC7g4ZbsYODatzwIuicwtwFhJe9RzGQ4kZlWLgksNJE0GXgfcCkyIiPVp10PAhLQ+EXgg97F1Ka1m7mw1q1gNw7/jJS3Pbc+LiHkvOp60M/AfwMci4glJf94XESE1/l5aBxKzqhUPJBsjYsZQGSS9hCyIXB4RP07JD0vaIyLWp6bLhpT+IDAp9/G9UlrN3LQxq1KQ3dlaZBmGsqrHfODuiPhGbtdCYHZanw1cm0s/JY3eHAxszjWBauIaiVmFRDTyztY3Au8B7pC0MqV9GvgycKWkOcD9wAlp3yLgKGAt8BRwWr0ndiAxq1qDAklE3EQ2Mf1ADhsgfwBnNOLcDiRmVQqgwNBuq3MgMauYH9ozs/IcSMysnM54aM+BxKxKgQOJmTWAZ0gzs7Lc2Wpm5QTQ0/5VEgcSs0q5s7XhnohHN/706cvur7ocNRgPbKy6EDW5s+oC1Kz9vmN4RU25HUgaKyJ2r7oMtZC0fLinMa2crviOHUjMrJQA/IIsMysnINzZ2u1eNDuVNVxnf8cdMmrjiY1KGGiauz6SeiStlLRK0lWSdqz3PJIOlXRdWj9G0twh8o6V9ME6znGOpE8UTR/iOH9qxHn7DPUdd4zGzSJfGQeS5nk6IvaPiP2ALcD78zvTrFQ1f/8RsTAivjxElrFAzYHEKuRAYgX9CthH0mRJv5d0CbAKmCTpcEk3S/pNqrnsDCBppqR7JP0GeGffgSSdKunbaX2CpGsk/S4tbyCbDeuVqTb01ZTvk5JuSy9BOjd3rM9IWi3pJmDfWi5I0n9KWpFexHR6v33np/SlknZPaa+UdEP6zK8kvbqO77EDFQwiDiTdTdIoshcR3ZGSpgLfiYjXAE8CnwXeFhEHAMuBMyWNBr4H/C1wIPDyQQ5/AfCLiJgOHEB2l8hc4A+pNvRJSYencx4E7A8cKOnNkg4ETkxpRwF/WeOlvTciDgRmAB+RNC6l7wQsT9f3C+DslD4P+HD6zCeA79R4vs4UQG9vsaWFubO1eXbIzZv5K7JJefcE7k8vI4LsbWjTgP9OrwzYDrgZeDVwX0SsAZB0Gdmb0Pp7K3AKQET0AJv7XseYc3hafpu2dyYLLLsA10TEU+kcC2u8vo9Iekdan5SOuYnsEbQrUvplwI9TLesNwFW5VyNsX+P5OleL1zaKcCBpnqcjYv98QvpH9GQ+CVgSESf1y/eCz5Uk4EsR8W/9zvGxug8oHQq8DTgkIp6StAwYPUj2IKv5Pt7/+7CkAwKJmzbVugV4o6R9ACTtJOlVwD3AZEmvTPlOGuTzS4EPpM+OlDQG+CNZbaPPYuC9ub6XiZJeBvwSOFbSDpJ2IWtGFTUGeCwFkVeT1az6jACOS+vvBm6KiCeA+yQdn8ogSdNrOF/niiB6egotrcyBpEIR8QhwKrBA0u2kZk1EPEPWlLk+dbZuGOQQHwXeIukOYAUwLSI2kTWVVkn6akT8FPh34OaU72pgl/SO2CuA3wE/AW4boqiflbSubwFuAEZJupusc/eWXN4ngYMkrSJrep2X0k8G5kj6HVlfzqyi31PH641iSwtTdEC1yqxdjRm1exyyS7GYuvjx+Sta9bkj95GYVSmi5UdkinAgMataB7QKHEjMKhaukZhZOa1/12oRDiRmVQqgxYd2i3AgMatQANHiQ7tFOJCYVSk8sZGZNUAn1Eh8Q5pZhSTdQDZTfhEbI2JmM8tTLwcSMyvNz9qYWWkOJGZWmgOJmZXmQGJmpTmQmFlpDiRmVpoDiZmV5kBiZqU5kJhZaf8PcvGK7rVMDtsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fac338e1190>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.matshow(confusion_matrix)\n",
    "plt.title('Confusion Marix')\n",
    "plt.colorbar()\n",
    "plt.ylabel('True Label')\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy\n",
    "Accuracy measures a fraction of the classifier's predictions that are correct.\n",
    "scikit-learn provides `accuracy_score` to calculate the accuracy of a set of predictions given the correct labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.977027997128\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "print 'Accuracy: ', accuracy_score(y_test, prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cross_val_score` predicts and scores labels for a test set using accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.91314778153 Scores of 5 folds:  [ 0.89964158  0.90681004  0.91756272  0.92114695  0.92057762]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "scores = cross_val_score(clf, X_test, y_test, cv=5)\n",
    "print scores.mean(), 'Scores of 5 folds: ', scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`LogisticRegression.score()` predicts and scores labels for a test set using accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.97702799712849964"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Accuracy is not an informative metric if the proportions of\n",
    "the classes are skewed in the number**. For example, a classifier that predicts whether or not credit card transactions are fraudulent may be more sensitive to false negatives than to false positives. To promote customer satisfaction, the credit card company may prefer to risk verifying legitimate transactions than risk ignoring a fraudulent transaction. Because most transactions are legitimate, accuracy is not an appropriate metric for this problem. A classifier that always predicts that transactions are legitimate could have a high accuracy score, but would not be useful. For these reasons, classifiers are often evaluated using two additional measures called precision and recall."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision and Recall\n",
    "\n",
    "#### Precision\n",
    "Precision is the ratio of true positives over sum true positives and false positive\n",
    "\n",
    "Precision can be thought of as the accuracy of the actual correct label out of all the classified labels as true.\n",
    "\n",
    "![Image of Precision formula](http://www.murase.m.is.nagoya-u.ac.jp/KSCGR/images/eval1.png)\n",
    "\n",
    "#### Recall\n",
    "Sometimes called sensitivity in medical domains, recall is the fraction of the truly positive instances that the classifier recognizes. A recall score of one indicates that the classifier did not make any false negative predictions.\n",
    "\n",
    "![Image of Recall Formula](http://www.murase.m.is.nagoya-u.ac.jp/KSCGR/images/eval2.png)\n",
    "\n",
    "In scikit-learn cross_val_score has both the scoring methods which can be used by setting attribute `scoring= 'precision'` or `scoring ='recall'`.\n",
    "\n",
    "\n",
    "*To compute the recall and precision, the data has to be indeed binarized.\n",
    "Labels didn't have to binarize the data when you want to calculate the accuracy as the accuracy formula doesn't really need information about which class is considered as positive or negative: (TP + TN) / (TP + TN + FN + FP). We can indeed see that TP and TN are exchangeable, it's not the case for recall, precision and f1*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision:  0.989640182602 Precision of 5-fold:  [ 0.98666667  0.98701299  0.98717949  1.          0.98734177]\n",
      "Recall:  0.651342959168 Recall of 5-fold:  [ 0.64912281  0.66666667  0.6754386   0.57522124  0.69026549]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelBinarizer\n",
    "\n",
    "lb = LabelBinarizer()\n",
    "ytb = lb.fit_transform(y_train)\n",
    "y_train_binary = ytb.reshape(ytb.shape[0],)\n",
    "precision = cross_val_score(clf, X_train, y_train_binary, cv=5, \n",
    "                                    scoring='precision')\n",
    "recall = cross_val_score(clf, X_train, y_train_binary, cv=5, \n",
    "                                    scoring='recall')\n",
    "print 'Precision: ', precision.mean(), 'Precision of 5-fold: ', precision \n",
    "print 'Recall: ', recall.mean(), 'Recall of 5-fold: ', recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our classifier's precision is 0.992; almost all of the messages that it predicted as spam were actually spam. Its recall is lower, indicating that it incorrectly classified approximately 22 percent of the spam messages as ham.\n",
    "\n",
    "### F1 measure\n",
    "\n",
    "The F1 measure is the harmonic mean, or weighted average, of the precision and recall scores. Also called the f-measure or the f-score, the F1 score is calculated using the following formula:\n",
    "\n",
    "![Image of formula](https://hassetukda.files.wordpress.com/2012/08/f12.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score:  0.784760142676\n"
     ]
    }
   ],
   "source": [
    "f1s = cross_val_score(clf, X_train, y_train_binary, cv=5,\n",
    "                     scoring='f1')\n",
    "print 'F1 score: ', f1s.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC AUC\n",
    "\n",
    "A Receiver Operating Characteristic, or ROC curve, visualizes a classifier's performance. Unlike accuracy, **the ROC curve is insensitive to data sets with unbalanced class proportions; unlike precision and recall, the ROC curve illustrates the classifier's performance for all values of the discrimination threshold.** ROC curves plot the classifier's recall against its fall-out. Fall-out, or the false positive rate, is the number of false positives divided by the total number of negatives. It is calculated using the following formula:\n",
    "\n",
    "![image](http://statconsulting.nazwa.pl/instalator/wordpress/wp-content/uploads/2015/06/roc1_en.png)\n",
    "\n",
    "AUC is the area under the ROC curve; it reduces the ROC curve to a single value,\n",
    "which represents the expected performance of the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8VNX9//HXBxRwYVHAjUWouAEiYipiFXEBUVFUFMFqARdqrdUiru23P5faWndr3XCrVitUbVXcbZVFrSjBFVAsgkLYVAQBJUDI5/fHuTFDMplMQiZ3ZvJ+Ph7zYJY7M5+5TO5nzjn3fI65OyIiIlVpFHcAIiKS3ZQoREQkJSUKERFJSYlCRERSUqIQEZGUlChERCQlJQpJycx+amavxB1HNjGzNWb2oxjet5OZuZltUd/vnQlmNsvM+tXiefpO1jMlihxiZp+b2droQLXUzB4ys20z+Z7u/nd3H5DJ90hkZgeZ2WtmttrMvjWzZ82sa329f5J4JpvZ2Yn3ufu27j4vQ++3h5k9YWZfR5//QzO7yMwaZ+L9aitKWF025zXcvZu7T67mfSolx/r+TooSRS46zt23BXoC+wFXxBxPrST7VWxmfYBXgGeAXYDOwAfAm5n4BZ9tv8zNbDfgbWAhsI+7twROAQqA5nX8XrF99mzb75IGd9clRy7A58CRCbdvAJ5PuN0UuAlYACwD7gG2Snh8MPA+sAr4DBgY3d8SeABYAiwCrgUaR4+NBN6Irt8N3FQhpmeAi6LruwD/BL4C5gMXJGx3FfAk8Gj0/mcn+XyvA3cluf9F4G/R9X5AEfAb4Oton/w0nX2Q8NzLgKXAI8B2wHNRzCui6+2j7f8AbASKgTXAHdH9DnSJrj8E3Ak8D6wmHOh3S4hnADAH+Ba4C5iS7LNH2z6a+P+Z5PFO0XuPiD7f18BvEx4/AHgLWBn9X94BNEl43IFfAv8D5kf3/ZmQmFYBM4BDErZvHO3nz6LPNgPoAEyNXuu7aL+cGm0/iPD9Wgn8F+hR4bt7GfAhsA7YgoTvcxR7YRTHMuCW6P4F0XutiS59SPhORtt0A/4NfBM99zepXlOXWhx74g5Alxr8Z236h9Ue+Aj4c8LjtwITge0Jv0CfBa6LHjsgOlj1J7Qk2wF7RY89BYwDtgF2AN4Bfh499sMfJdA3OqhYdHs7YC0hQTSKDiT/D2gC/AiYBxwVbXsVsAE4Idp2qwqfbWvCQfmwJJ97FLAkut4PKAFuISSFQ6MD1p5p7IOy514fPXcroDUwJHr/5sATwNMJ7z2ZCgd2KieK5dH+3QL4OzAheqxNdJA6KXrswmgfVJUolgKjUvz/d4re+74o9n0JB929o8f3Bw6M3qsT8DHw6wpx/zvaN2XJ8/RoH2wBjI1iaBY9dgnhO7YnYNH7ta64D6Lb+wFfAr0JCWYE4fvaNOG7+z4h0WyVcF/Z9/kt4Izo+rbAgRU+8xYJ7zWS8u9kc0JSHAs0i273TvWautTi2BN3ALrU4D8r/GGtIfy6c+BVoFX0mBEOmIm/ZvtQ/stxHHBrktfcMTrYJLY8hgOTouuJf5RG+IXXN7p9DvBadL03sKDCa18B/DW6fhUwNcVnax99pr2SPDYQ2BBd70c42G+T8PjjwO/S2Af9gPVEB8Iq4ugJrEi4PZnqE8X9CY8dA3wSXf8Z8FbCY0ZItFUlig1ErbwqHi87aLZPuO8dYFgV2/8aeKpC3IdX8x1bAewbXZ8DDK5iu4qJ4m7g9xW2mQMcmvDdPTPJ97ksUUwFrgbaVPGZq0oUw4H3qogx6WvqUvOLxihyzwnu3pxw0NuL8KsVoC3hV/EMM1tpZiuBl6L7IfyS+yzJ6+0KbAksSXjeOELLYhMe/vomEP44AU4j/IIue51dyl4jep3fEBJRmYUpPtcKoBTYOcljOxO6WX7Y1t2/S7j9BaFVU90+APjK3YvLbpjZ1mY2zsy+MLNVhINLqxoOHi9NuP494dcrUUw/fOZo/xWleJ3lJP/8ab1fNBD+XHSiwyrgj5R/P8ps8n9gZheb2cfRwPlKQjdk2XOq+s4ksyswtsL/fwfCPkj63hWcBewBfGJm081sUJrvmyrG2r6mVKBEkaPcfQrh1+xN0V1fE7qBurl7q+jS0sPAN4Q/0t2SvNRCQouiTcLzWrh7tyreejxwspntSmhF/DPhdeYnvEYrd2/u7sckhp3i83xH6Co4JcnDQwmtpzLbmdk2Cbc7AovT2AfJYhhL6Frp7e4tCN1rEH79p4w5DUsILaXwgmaWeDuJ/xC6wWrrbuATYPfos/yG8s9R5ofPY2aHAJcS9u927t6K0D1Z9pyqvjPJLAT+UOH/f2t3H5/svSty9/+5+3DCD5TrgSej/+Pq9v9CQjdnTV5TakiJIrfdBvQ3s33dvZTQd32rme0AYGbtzOyoaNsHgFFmdoSZNYoe28vdlxDONLrZzFpEj+1mZocme0N3f49wQL4feNndV0YPvQOsNrPLzGwrM2tsZt3N7Mc1+DyXAyPM7AIza25m25nZtYTuo6srbHu1mTWJDnaDgCfS2AfJNCckl5Vmtj1wZYXHl1HFgSgNzwP7mNkJ0Zk+vwR2SrH9lcBBZnajme0Uxd/FzB41s1ZpvF9zwpjIGjPbC/hFGtuXEAbytzCz/we0SHj8fuD3Zra7BT3MrHX0WMX9ch9wrpn1jrbdxsyONbO0ztYys9PNrG30f1j2nSqNYiul6v+D54CdzezXZtY0+t70ruY1pYaUKHKYu38F/I0wgAzhrJK5wLSo6+E/hF/LuPs7hEHhWwm/GqcQugsg9KU3AWYTuoCeJHUXyGPAkdG/ZbFsJBywexLOeCpLJi1r8HneAI4iDP4uIXQp7Qcc7O7/S9h0aRTnYkLX17nu/kl1+6AKtxEGhr8GphG6qhL9mdCCWmFmt6f7WaLP8zWhhXQDoVupK+EsnHVVbP8ZISl2AmaZ2beEFlshYVyqOhcTugNXEw7c/6hm+5cJn/dTwr4uZtPuoVsI4z+vEBLQA4R9BWHM6eGom2mouxcSxqzuIPzfzCWMJaRrIOEzryHs82HuvtbdvyecffZm9F4HJj7J3VcTTtA4jvC9+B9wWKrXrEFMEik7e0UkJ1iYyfuou6fqwslKZtaIMEbxU3efFHc8IulSi0Ikg8zsKDNrZWZNKR8zmBZzWCI1krFEYWYPmtmXZjazisfNzG43s7kWyhT0ylQsIjHqQzgr52tC98gJ6v6QXJOxricz60s45/9v7t49yePHAL8inHfemzBxrHdGghERkVrLWIvC3acSptRXZTAhibi7TyOcu57OOeQiIlKP4izO1Y5Nz7Aoiu5bUnFDMxsNjAbYZptt9t9rr73qJUARkWzkDhs2wPr15ZeKt0tKwrYd+YJWrORDSr5297apXzm5nKji6O73AvcCFBQUeGFhYcwRiYhkxsaNsHQpLFwYLkVF5dfLLkuXQmmFGSHNm0OHDuHSvp2H6x2NPu/fTRv/kh3uuOqL2sYUZ6JYRJh+X6Z9dJ+ISF4qLYUvv0x+8C+7b/Hi8tZAma22Kk8CAwaUX+/QAdq3D/+2LJuxtGgR/OIXcPip8NOf8sO8yzuuqnXccSaKicD5ZjaBMJj9bTRLWEQk57jD8uXJD/5ll0WLQrdQoqZNyw/2hx666cG/7LLddmAVi7EkC+D+++Hii0M/1LHH1tlny1iiMLPxhMJ1bcysiFCeYEsAd78HeIFwxtNcQmGzUZmKRURkc7jDypVVdwWV3V9cvOnzttwS2rULB/sDD9z04F+WENq2TSMJVOezz+Ccc2DSJDjsMLjvPtgt3TJd1ctYooiKcaV63Am1b0REYrV6ddUH/7Lr33236XMaNYJddgkH/F69YPDgyt1BO+4Ytsu4jz6CGTPg3nvh7LPrIPNsKicGs0VEauv771MPDC9cCKtWbfocM9hpp3Cw79YNBg6s3B20006wRZxH0Jkz4d134Wc/gxNOgHnzoHXr6p9XC0oUIpKziotDv3+q1sA3SWZz7bBDONh36RJ6aip2B+2yCzRpUv+fJy3r18Mf/xguO+4IQ4dCs2YZSxKgRCEiWWrDhspJoGKL4KuvKj9v++3LD/oHHVR5cLhdu3BczUlvvw1nnQWzZsHpp8Ott9bLh1GiEJF6V1ICS5ZU3R1UVBTmClSsMNSyZfkBf//9kw8Ob711PJ8p4xYtgkMOCa2I556r07OaqqNEISJ1qrQUli1LPTC8ZEmYWJZo223LD/b77JN8rkDztJZByjOffgp77BGaQv/4BxxxBLRoUf3z6pAShYikzT1096QaHF60qPKEsWbNyg/6RxyRfK5Ay5Z1frJOblu5Ei69NMyNmDwZ+vaFE0+MJRQlChEBQhL45pvU8wSKimBdhfX5mjQpP+gffHDl7qAOHcK4gZJADUycGGZXL10Kl1wCP67JisJ1T4lCpIH49tvUA8NFReFU0kSNG5dPGPvxj+Gkkyq3Btq2rae5Ag3F2WfDAw+E/rdnnoGCgrgjUqIQyQdr1lQ/a3h1hVW3GzWCnXcOB/t99w1joxVbAjvuGJKFZFjZqL1ZSAy77gqXXZY15+gqUYhkubVrUxeRW7gwdGdXVDZhbK+9oH//5HMFYp0wJsHChXDuuTBsGJxxRrieZfQ1EYnRunWbzhVIlhCWL6/8vDZtwgG/c+cwxplsrkCW/BiVqpSWwrhxoeWwcWNsA9XpUKIQyZANG0LJ6FTdQcuWVX7edtuVH/B7904+VyBnJ4xJ8L//hbGIqVPhyCNDjabOneOOqkpKFCK1UHFxmWQtgmSLy7RoUf7Lf7/9ks8V2GabeD6T1KPZs+HDD+HBB2HkyKw/JUyJQqSCssVlUnUHLV5cecLY1luXH/SPOir5XIF6nicl2eSDD+D992HEiFBqdt680HzMAUoU0qAkW1ymYkKoanGZsoN9v37Ju4PSWlxGGp516+Daa+FPfwqnmZ16aug7zJEkAUoUkkcSF5dJNVcg1eIyffok7w5q00ZJQGrhrbdCEb+PPw7lwG+5JScHmJQoJGesWpW6iFyyxWUaNy5fXGb//TddXKbsssMOmjAmGbBoUVjbdKed4IUX4Oij446o1pQoJCskLi5TVWsg2eIyO+8cfvWXLS5TsTWw886aMCb17OOPYe+9QzP18cdDcascr2aoRCEZV1xcftCv6lTRFSsqP69scZndd4fDD688OLzLLqHbSCQrrFgBY8fCX/8aTns95JCw8lweUKKQzbJ+fTgDKNXgcLLFZVq3Dgf7XXeFn/ykcndQu3ZhAFkkJzz1FJx3XviyX3FF7EX86poShVQpcXGZqrqDli2rvLhMq1blv/wLCip3B+X14jLS8Jx5ZmhF9OwJzz8PvXrFHVGdU6JooEpLw4SwVLOGFy+uPGGsbHGZDh2gR4/K3UHt2+d8d6xI9RKL+B14YOgfvfjivO0LVaLIQxUXl0nWGki2uMxWW1VeXKZia0CLy0iD98UX8POfw2mnhVNeR4+OO6KMU6LIMWWLy6SaNVxUVHnCWOLiMoccUnmegBaXEalGaSncfTdcfnn4QzzllLgjqjdKFFnEPSwuU11J6bVrN33eFluUTxjr3RuGDKncGmjbVklApNbmzAlF/N54AwYMCFVfO3WKO6p6o0RRj8oWl0nVGlizZtPnNGpUPmFs331h0KDK3UFaXEYkw+bMgVmz4KGHQndTA/vVpURRR9aurX6FsYqLy5iFg3yHDmF+zoABlbuDdt5Zi8uIxOK990IRv1Gj4PjjQxG/Vq3ijioWOgSloeLiMslaBMkWl2nbNhzsd9stzOSv2B20yy5aXEYk6xQXwzXXwA03hD7d4cNDfaYGmiRAieKHxWVSdQd9+WXl522/ffkv/wMPrDw4rMVlRHLQm2+GIn5z5oSWxM036w+ZPE8UGzeGCWOpuoOWLKk8YaxFi/IDf69eyecKaHEZkTyzaBEcdlhoRbz8cugLFiCHE0XFxWWStQaSLS6zzTblB/xu3ZLPFdDiMiINyOzZ0LVrSBD//GdIFttuG3dUWcW84s/pLNeiRYG3bl3IokWh2yhRs2aVVxSr2Bpo1arBnbAgIsl88w1cdBE8/DBMmQJ9+8YdUUaZ2Qx3L6jNc3OuRbF6NfTvHxaJqpgQWrdWEhCRNPzzn/DLX4azUH77WzjggLgjymo5lygA7rgjnDYqIlJjI0eGVkSvXvDSS6GYn6SUk4lCRKRGEov4HXRQmLg0dqwmKaUpowtAmtlAM5tjZnPN7PIkj3c0s0lm9p6ZfWhmx2QyHhFpgObPD2cw/e1v4fbo0XDZZUoSNZCxRGFmjYE7gaOBrsBwM+taYbP/Ax539/2AYcBd6b12XUYqInlp40a4/Xbo3h2mTat8HrykLZMtigOAue4+z93XAxOAwRW2caDsZNSWwOIMxiMiDcXHH4cyyRdeGMoizJoVxiakVjKZKNoBCxNuF0X3JboKON3MioAXgF8leyEzG21mhWZWGG7XfbAikkfmzg2zqx95JKw617Fj3BHltIyOUaRhOPCQu7cHjgEeMbNKMbn7ve5eUNtzgEWkAZgxAx58MFw/7rgwNnH66fplWQcymSgWAR0SbreP7kt0FvA4gLu/BTQD2mQwJhHJN2vXhsWEeveG3/8+FPUDlVioQ5lMFNOB3c2ss5k1IQxWT6ywzQLgCAAz25uQKL6q7oX1A0FEAJg6NSzUcv31YQzivfdUxC8DMnZ+mLuXmNn5wMtAY+BBd59lZtcAhe4+ERgL3GdmYwgD2yM912qKiEg8Fi0qX9z9P/8J1yUjcq7Wk1mBf/llIW3bxh2JiMTio49gn33C9eeeC0X8VM65WptT6ynuwWwRkfR8/TWccQb06BG6nCCsDawkkXGamigi2c0dnngCzj8fVqyAK68MA9dSb3IyUWgwW6QBGTEizIcoKIBXXy3vdpJ6k5OJQkTyXGIRv0MPDd1Nv/616jPFJCfHKNSiEMlj8+bBkUfCQw+F22edBRdfrCQRo5xMFCKShzZuhNtuC11L06dDIx2esoVStIjEb/ZsOPNMePttOPZYuOeesIaxZIWcTBTqehLJM/Pnw2efwWOPwbBh+iPPMjmZKEQkD0yfDu+/D+ecE1oR8+ZB8+ZxRyVJ5GQnoH5siOSw778Pg9MHHgjXXVdexE9JImvlZKIQkRw1eXI41fXmm0NLQkX8coK6nkSkfhQVQf/+sOuu8NproUaT5IScbFGo60kkh3zwQfi3fXt45hn48EMliRyTk4lCRHLAV1/BaadBz54wZUq475hjYOut441Laiwnu57UohDJYu4wYQJccAF8+y1cfTX06RN3VLIZcjJRiEgWO+MM+PvfQ4XXBx6Abt3ijkg2kxKFiGy+0tLQ1DcL4w/77x9aFI0bxx2Z1IGcHKNQ15NIFpk7NyxD+te/httnnQVjxihJ5JGcTBQikgVKSuCmm0IRv/fegyZN4o5IMiQnu57UohCJ2cyZMGoUFBbC4MFw112wyy5xRyUZkpOJQkRitmABfPFFOLtp6FD9estzShQikp633w6T50aPDvMh5s2DbbeNOyqpBzk5RqEfLyL16Lvv4KKLwlyIG26AdevC/UoSDUZOJgoRqSevvRaK+N16K5x7Lrz7LjRtGndUUs/U9SQiyRUVwVFHQefOoQRH375xRyQxyckWhbqeRDLovffCv+3bw7PPhnEJJYkGLScThYhkwLJlcOqp0KtXeRG/gQNhq63ijUtil5OJQi0KkTrkDo8+Cl27wtNPw7XXwkEHxR2VZBGNUYg0dKedFuZD9OkTivjtvXfcEUmWUaIQaYgSi/gNGBCSxC9/qfpMkpS6nkQamk8/DRVeH3ww3B41SpVeJaWcTBQiUgslJWHC3L77huVINUgtacrJrie1KERq6MMP4cwzYcYMOPFEuPNO2HnnuKOSHJGTiUJEaqioCBYuhCeegCFD9GtLaiSjXU9mNtDM5pjZXDO7vIpthprZbDObZWaPZTIekQblv/+Fe+4J18uK+J18spKE1FjGEoWZNQbuBI4GugLDzaxrhW12B64AfuLu3YBfp/fadRysSD5ZswYuvBAOPhhuvrm8iN8228Qbl+SsTLYoDgDmuvs8d18PTAAGV9jmHOBOd18B4O5fZjAekfz3yivQvTv85S/hdFcV8ZM6kMlE0Q5YmHC7KLov0R7AHmb2pplNM7OByV7IzEabWaGZFYbbGYlXJLctXAjHHgvNmsHUqSFZNG8ed1SSB+I+PXYLYHegHzAcuM/MWlXcyN3vdfcCdy+o5/hEst+MGeHfDh3ghRfg/fdDt5NIHclkolgEdEi43T66L1ERMNHdN7j7fOBTQuIQkeosXQqnnAIFBeVF/Pr3Dy0KkTqUyUQxHdjdzDqbWRNgGDCxwjZPE1oTmFkbQlfUvOpeWF1P0qC5w8MPhyJ+zz4Lf/yjivhJRmVsHoW7l5jZ+cDLQGPgQXefZWbXAIXuPjF6bICZzQY2Ape4+/JMxSSSF4YNg8cfh5/8BO6/H/baK+6IJM+Zu8cdQ42YFfiGDYVsoamC0pAkFvF7+GFYvRrOOw8axT3MKLnCzGbUdpxX3zKRbPfJJ2GFuQceCLdHjIDzz1eSkHqjb5pIttqwIYw/7LsvzJ4N224bd0TSQOVkB44GsyXvvf9+KP/9/vuh7MZf/gI77RR3VNJA5WSiEMl7S5eGyz//CSedFHc00sClTBRmdlGqx939lroNJz1qUUheeuONUA78vPNg4ED47DPYeuu4oxKpdoyieTUXEdlcq1eHwelDDoHbbisv4qckIVkiZYvC3a+ur0BEGqSXX4bRo0OdpgsvhGuvVRE/yTrVdT3dnupxd7+gbsNJj7qeJC8sXAiDBkGXLqHbSbOrJUtVN5g9o16iEGko3GH6dDjggFDE78UXQwE/1WeSLJaTM7PdC+MOQ6TmliwJa0Q89RRMngyHHhp3RNKAbM7M7LROjzWztsBlhJXqfvjp4+6H1+ZNRRoUd3joIbjoIiguhuuvD3WaRHJEujOz/w58DHQGrgY+J1SHFZHqDB0KZ54J++wDH3wAl16KipVJLkn329ra3R8wswvdfQowxcyUKESqsnFjOOuiUSM47jg4/HD4+c9Vn0lyUrrf2g3Rv0vM7Fgz2w/YPkMxieS2jz8OcyLKivj97Gfwi18oSUjOSrdFca2ZtQTGAn8BWgBjMhaVSC7asCGMP/z+96GAX8uWcUckUifSShTu/lx09VvgsMyFI5Kj3nsPRo4MJThOPRVuvx122CHuqETqRFptYTN72MxaJdzezswezFxYIjlm2TL4+mt4+mmYMEFJQvJKul1PPdx9ZdkNd18RjVOINFxTp8JHH4W5EQMHwty5sNVWcUclUufSHV1rZGbbld0ws+1RiXJpqFatChVeDz00dDGVFfFTkpA8le7B/mbgLTN7Irp9CvCHzIQkksVeeCGc5rp4cZhAd801KuIneS/dwey/mVkhUDYT+yR3n525sESy0MKFMHgw7LknPPkk9O4dd0Qi9aImJ3ZvD3zn7ncAX5lZ5wzFJJI93GHatHC9Qwd45RV4910lCWlQ0j3r6UpCracroru2BB7NVFAiWWHxYjjhBOjTB6ZMCfcddhg0aRJvXCL1LN0WxYnA8cB3AO6+GK1wJ/nKHe6/H7p2DS2Im25SET9p0NIdzF7v7m5mDmBm22QwppS0aJFk3Mknw7/+Fc5quv/+sLCQSAOWbqJ43MzGAa3M7BzgTOD+zIUlUs8Si/idcAIMGADnnKP6TCLUYOEiM+sPDAAMeNnd/53JwKrSqFGBl5Zq4SKpQzNnwtlnw1lnheQgkocyvnARQJQY/h29YSMz+6m7/702byqSFdavh+uugz/8IRTw22676p8j0gClbFebWQszu8LM7jCzARacD8wDhtZPiCIZMGMG7L8/XHUVnHIKzJ4dxiZEpJLqWhSPACuAt4Czgd8Qup5OcPf3MxybSOYsXw4rV8Kzz8KgQXFHI5LVUo5RmNlH7r5PdL0xsATo6O7F9RRfJRqjkFqbNCkU8bvggnC7uBiaNUv9HJE8sTljFNWd0lG2sh3uvhEoijNJiNTKt9+G+kyHHw53311exE9JQiQt1SWKfc1sVXRZDfQou25mq+ojQJHN8uyzYeLc/ffDxReHsQkV8ROpkZRjFO7euL4CEalzCxfCkCGw115hQaEf/zjuiERykmYTSX5xh//+N1wvK+JXWKgkIbIZMpoozGygmc0xs7lmdnmK7YaYmZtZrQZaRAAoKoLjjw91mcqK+PXrpyJ+IpspY4kiOkvqTuBooCsw3My6JtmuOXAh8HamYpE8V1oK48aFsYhXX4VbboGDD447KpG8kckWxQHAXHef5+7rgQnA4CTb/R64HkjrbCoVBZRKhgyBc88N3UszZ8KYMdBYw2sidSWTiaIdsDDhdlF03w/MrBfQwd2fT/VCZjbazArNrDDd2lSS50pKQksCQqK47z74z3/gRz+KNy6RPBTbYLaZNQJuAcZWt6273+vuBe5eYGpSyIcfhsWE7rsv3D799FDUT98NkYzIZKJYBHRIuN0+uq9Mc6A7MNnMPgcOBCZqQFuqtG4dXHllqNH0xRfQtm3cEYk0CGlXj62F6cDu0drai4BhwGllD7r7t0CbsttmNhm42N1Vn0Mqmz4dRo4MxfvOOANuvRVat447KpEGIWOJwt1LokqzLwONgQfdfZaZXQMUuvvETL235KEVK2DNGnjhBTj66LijEWlQ0l64KFs0blzgGzeq0dEgvPZaKOJ34YXh9rp1Kr8hUkuZLAooUv9WrgwrzR1xRJgfUVbET0lCJBZKFJJdnnkmTJx78EG49FIV8RPJApkczBapmQULwmpze+8NEydCgU6AE8kGalFIvNzh9dfD9Y4dw6S56dOVJESyiBKFxGfBAjj2WOjbt7yIX9++KuInkmWUKKT+lZbCXXdBt24wdSrcfruK+IlksZwbo1CVhjxw0klh0Lp/f7j3XujUKe6IRCSFnEsUkqNKSqBRo3A59VQYPDjMtFbmF8l66nqSzPvgA+jdO7QeAIYPh1GjlCREcoQShWROcTH83/+FM5iKimCnneKOSERqQV1PkhnvvAMjRsAnn4R/b7kFtt/IvWsnAAARDElEQVQ+7qhEpBaUKCQzVq2CtWvhpZfgqKPijkZENoMShdSdV16BWbPCUqRHHglz5qj8hkge0BiFbL4VK8Lg9FFHwQMPqIifSJ5RopDN869/hSJ+jzwCV1wBhYVKECJ5Rl1PUnsLFsCwYdC9e1hQaL/94o5IRDJALQqpGffyukwdO4bFhd5+W0lCJI8pUUj6vvgiLEPar195sjj4YNhyy1jDEpHMyrlEocm8MSgthTvuCEX83ngD/vIXOOSQuKMSkXqiMQqp3gknwLPPhrOaxo2DXXeNOyIRqUdKFJLchg3QuHEo4jd8OJx8Mpxxhpp0Ig1QznU9ST1491044AC4555we/hw+NnPlCREGiglCim3dm2YC3HAAbB0KXToEHdEIpIF1PUkwbRpoXjfp5/CmWfCTTfBdtvFHZWIZAElCgm++y6MS/z736FOk4hIRImiIXvppVDEb+xYOOKIUBK8SZO4oxKRLKMxioZo+fLQzXT00fDww7B+fbhfSUJEklCiaEjc4cknQxG/xx4Lq89Nn64EISIpqeupIVmwAE47DXr0CGtH7Ltv3BGJSA5QiyLfuYfCfRBmVE+eHM5wUpIQkTQpUeSz+fNhwIAwUF1WxO+gg2ALNSRFJH05lyg0OTgNGzfCn/8c1ol4+224+24V8RORWtNPy3w0eDA8/zwcc0wow6EZ1iKyGZQo8kViEb8zzgj1mU47TU0wEdlsGe16MrOBZjbHzOaa2eVJHr/IzGab2Ydm9qqZqX51bRQWQkFB6GICOPVU+OlPlSREpE5kLFGYWWPgTuBooCsw3My6VtjsPaDA3XsATwI3ZCqevLR2LVx2GfTuDV99pXUiRCQjMtmiOACY6+7z3H09MAEYnLiBu09y9++jm9OA9hmMJ7+89VY4xfWGG0IRv9mzYdCguKMSkTyUyTGKdsDChNtFQO8U258FvJjsATMbDYwGaNx4v7qKL7etXRuWKP3Pf8LpryIiGZIVg9lmdjpQABya7HF3vxe4F6BJkwKvx9CyywsvhCJ+l1wChx8OH38MW24Zd1Qikucy2fW0CEg8L7N9dN8mzOxI4LfA8e6+LoPx5K6vv4bTT4djj4W//728iJ+ShIjUg0wmiunA7mbW2cyaAMOAiYkbmNl+wDhCkvgyg7HkJneYMAH23hsefxyuvBLeeUdF/ESkXmWs68ndS8zsfOBloDHwoLvPMrNrgEJ3nwjcCGwLPGHhVM4F7n58pmLKOQsWhHLg++4LDzwA++wTd0Qi0gCZe251+TdpUuDr1xfGHUbmuMOrr5avMjdtGvz4x2EynYhILZnZDHcvqM1zc67WU1777LNwBlP//uVF/A48UElCRGKVc4kiLycbb9wIt9wSupZmzIBx41TET0SyRlacHtvgHXccvPhimDB3993QXvMORSR7KFHEZf36sC5Eo0YwcmQo5DdsWJ42mUQkl+Vc11NeeOcd2H9/uOuucHvo0FDtVUlCRLKQEkV9+v57GDsW+vSBFStgt93ijkhEpFrqeqovb7wR5kTMmwc//zlcfz20bBl3VCIi1VKiqC9lCwtNmgT9+sUdjYhI2pQoMunZZ0PhvksvhcMOC6XAt9AuF5HcojGKTPjqq7AM6fHHw/jx5UX8lCREJAcpUdQld3jssVDE78kn4Zpr4O23VcRPRHKafuLWpQULYNQo2G+/UMSvW7e4IxIR2WxqUWyu0lJ4+eVwfddd4fXX4c03lSREJG/kXKLIqjlp//tfWGlu4ECYOjXcd8ABKuInInkl5xJFVigpgRtvhB494P33QzeTiviJSJ7SGEVtDBoUupsGDw5lOHbZJe6IRLLShg0bKCoqori4OO5QGoxmzZrRvn17tqzDpZJzbuGiZs0KvLg4hoWL1q0La1Q3ahTOaCothVNOybK+MJHsMn/+fJo3b07r1q0x/a1knLuzfPlyVq9eTefOnTd5TAsXZdq0adCrF9x5Z7h98smhkJ+++CIpFRcXK0nUIzOjdevWdd6CU6JI5bvvYMwYOOggWL0adt897ohEco6SRP3KxP7WGEVVXn89FPGbPx/OOw+uuw5atIg7KhGReqcWRVVKSsKYxJQpoctJSUIkZz399NOYGZ988skP902ePJlBgwZtst3IkSN58skngTAQf/nll7P77rvTq1cv+vTpw4svvrjZsVx33XV06dKFPffck5fL5mBV8Nprr9GrVy+6d+/OiBEjKCkpAeCTTz6hT58+NG3alJtuummzY0mXEkWip58OLQcIRfxmzYK+feONSUQ22/jx4zn44IMZP3582s/53e9+x5IlS5g5cybvvvsuTz/9NKtXr96sOGbPns2ECROYNWsWL730Eueddx4bN27cZJvS0lJGjBjBhAkTmDlzJrvuuisPP/wwANtvvz233347F1988WbFUVPqegJYtgx+9St44okwaD12bKjPpCJ+InXm178O047qUs+ecNttqbdZs2YNb7zxBpMmTeK4447j6quvrvZ1v//+e+677z7mz59P06ZNAdhxxx0ZOnToZsX7zDPPMGzYMJo2bUrnzp3p0qUL77zzDn369Plhm+XLl9OkSRP22GMPAPr37891113HWWedxQ477MAOO+zA888/v1lx1FTDblG4wyOPQNeu8Mwz8Ic/hDOcVMRPJG8888wzDBw4kD322IPWrVszY8aMap8zd+5cOnbsSIs0upzHjBlDz549K13+9Kc/Vdp20aJFdOjQ4Yfb7du3Z9GiRZts06ZNG0pKSigsDNMAnnzySRYuXFhtHJnUsH8yL1gAZ58NBQVhdvVee8UdkUjequ6Xf6aMHz+eCy+8EIBhw4Yxfvx49t9//yrPDqrpWUO33nrrZsdY8f0nTJjAmDFjWLduHQMGDKBxzGWBGl6iKCvid/TRoYjfm2+Gaq+qzySSd7755htee+01PvroI8yMjRs3YmbceOONtG7dmhUrVlTavk2bNnTp0oUFCxawatWqalsVY8aMYdKkSZXuHzZsGJdffvkm97Vr126T1kFRURHt2rWr9Nw+ffrw+uuvA/DKK6/w6aefpv2ZM8Ldc+rSrNn+Xmtz5rgfcog7uE+eXPvXEZG0zJ49O9b3HzdunI8ePXqT+/r27etTpkzx4uJi79Sp0w8xfv75596xY0dfuXKlu7tfcsklPnLkSF+3bp27u3/55Zf++OOPb1Y8M2fO9B49enhxcbHPmzfPO3fu7CUlJZW2W7Zsmbu7FxcX++GHH+6vvvrqJo9feeWVfuONN1b5Psn2O1DotTzuNowxipISuP76UMTvo4/gr3/V2UwiDcD48eM58cQTN7lvyJAhjB8/nqZNm/Loo48yatQoevbsycknn8z9999Py5YtAbj22mtp27YtXbt2pXv37gwaNCitMYtUunXrxtChQ+natSsDBw7kzjvv/KFb6ZhjjmHx4sUA3Hjjjey999706NGD4447jsMPPxyApUuX0r59e2655RauvfZa2rdvz6pVqzYrpnTkXK2nrbYq8LVra1jr6aij4JVX4KSTwpyInXbKTHAisomPP/6YvffeO+4wGpxk+31zaj3l7xhFcXGYMNe4MYweHS5DhsQdlYhIzsnPrqc33wwnWJcV8RsyRElCRKSW8itRrFkDF1wQFhEqLgY1eUVil2vd27kuE/s7fxLFlCnQvTvccQecfz7MnAn9+8cdlUiD1qxZM5YvX65kUU88Wo+iWbNmdfq6+TVGsfXWoerrT34SdyQiQph5XFRUxFdffRV3KA1G2Qp3dSm3z3r617/gk0/gN78Jtzdu1MQ5EZEksnaFOzMbaGZzzGyumV2e5PGmZvaP6PG3zaxTWi+8dGlYZW7IEHjqKVi/PtyvJCEiUucylijMrDFwJ3A00BUYbmZdK2x2FrDC3bsAtwLXV/e6rTYuD4PUzz0XSoL/978q4icikkGZbFEcAMx193nuvh6YAAyusM1g4OHo+pPAEVZNRa5dNnwRBq0/+AAuvzzMlRARkYzJ5GB2OyCxNm4R0Luqbdy9xMy+BVoDXyduZGajgdHRzXX2xhszVekVgDZU2FcNmPZFOe2LctoX5fas7RNz4qwnd78XuBfAzAprOyCTb7QvymlflNO+KKd9Uc7Malj7qFwmu54WAR0SbreP7ku6jZltAbQElmcwJhERqaFMJorpwO5m1tnMmgDDgIkVtpkIjIiunwy85rl2vq6ISJ7LWNdTNOZwPvAy0Bh40N1nmdk1hLroE4EHgEfMbC7wDSGZVOfeTMWcg7QvymlflNO+KKd9Ua7W+yLnJtyJiEj9yp9aTyIikhFKFCIiklLWJoqMlf/IQWnsi4vMbLaZfWhmr5rZrnHEWR+q2xcJ2w0xMzezvD01Mp19YWZDo+/GLDN7rL5jrC9p/I10NLNJZvZe9HdyTBxxZpqZPWhmX5rZzCoeNzO7PdpPH5pZr7ReuLaLbWfyQhj8/gz4EdAE+ADoWmGb84B7ouvDgH/EHXeM++IwYOvo+i8a8r6ItmsOTAWmAQVxxx3j92J34D1gu+j2DnHHHeO+uBf4RXS9K/B53HFnaF/0BXoBM6t4/BjgRcCAA4G303ndbG1RZKT8R46qdl+4+yR3/z66OY0wZyUfpfO9APg9oW5YcX0GV8/S2RfnAHe6+woAd/+ynmOsL+nsCwdaRNdbAovrMb564+5TCWeQVmUw8DcPpgGtzGzn6l43WxNFsvIf7araxt1LgLLyH/kmnX2R6CzCL4Z8VO2+iJrSHdz9+foMLAbpfC/2APYwszfNbJqZDay36OpXOvviKuB0MysCXgB+VT+hZZ2aHk+AHCnhIekxs9OBAuDQuGOJg5k1Am4BRsYcSrbYgtD91I/QypxqZvu4+8pYo4rHcOAhd7/ZzPoQ5m91d/fSuAPLBdnaolD5j3Lp7AvM7Ejgt8Dx7r6unmKrb9Xti+ZAd2CymX1O6IOdmKcD2ul8L4qAie6+wd3nA58SEke+SWdfnAU8DuDubwHNCAUDG5q0jicVZWuiUPmPctXuCzPbDxhHSBL52g8N1ewLd//W3du4eyd370QYrzne3WtdDC2LpfM38jShNYGZtSF0Rc2rzyDrSTr7YgFwBICZ7U1IFA1xfdaJwM+is58OBL519yXVPSkru548c+U/ck6a++JGYFvgiWg8f4G7Hx9b0BmS5r5oENLcFy8DA8xsNrARuMTd867Vnea+GAvcZ2ZjCAPbI/Pxh6WZjSf8OGgTjcdcCWwJ4O73EMZnjgHmAt8Do9J63TzcVyIiUoeytetJRESyhBKFiIikpEQhIiIpKVGIiEhKShQiIpKSEoUIYGYbzez9hEunFNt2KqvOaWb9zOy5Ooqhn5kdVBevJVKXsnIehUgM1rp7z5hj6AesAf4bcxwim1CLQqQKUcvhdTN7N7rU6Ne+mR0RrX/wUbROQNPo/s+jmdKYWYGZTY5aMOcCY6IWzSF1/XlEakuJQiTYKqHb6anovi+B/u7eCzgVuD3dFzOzZsBDwKnuvg+h9f6LqrZ398+Be4Bb3b2nu79eu48hUvfU9SQSJOt62hK4w8x6Ekpg7FGD19sTmO/un0a3HwZ+Cdy22ZGK1DMlCpGqjQGWAfsSWt8pF0Iys5eBHYFC4C8pNi2hvDXfbPPDFMksJQqRqrUEity91MxGEArOVcndjyq7HnU9dTKzLu4+FzgDmBI9/DmwP2GBqSEJL7Ga8lXYRLKGxihEqnYXMMLMPgD2Ar5L94nuXkyozPmEmX0ElBLGIACuBv5sZoWELq0yzwInajBbso2qx4qISEpqUYiISEpKFCIikpIShYiIpKREISIiKSlRiIhISkoUIiKSkhKFiIik9P8BAikqXOjctscAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fac2fedfdd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "from sklearn.preprocessing import LabelBinarizer\n",
    "\n",
    "lbt = LabelBinarizer()\n",
    "lbp = LabelBinarizer()\n",
    "ytb = lbt.fit_transform(y_test)\n",
    "y_test_binary = ytb.reshape(ytb.shape[0],)\n",
    "pb = lbp.fit_transform(prediction)\n",
    "prediction_binary = pb.reshape(pb.shape[0],)\n",
    "\n",
    "false_positive_rate, recall, threshold = roc_curve(y_test_binary, prediction_binary)\n",
    "roc_auc = auc(false_positive_rate, recall)\n",
    "plt.title('Receiver Operating Characteristics')\n",
    "plt.plot(false_positive_rate, recall, 'b', label='AUC = %0.2f'% roc_auc)\n",
    "plt.legend(loc='lower right')\n",
    "plt.plot([0,1],[0,1], 'r--')\n",
    "plt.axis([0.0, 1.0, 0.0, 1.0])\n",
    "plt.xlabel('Fall-out')\n",
    "plt.ylabel('Recall')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning models with grid search\n",
    "\n",
    "Hyperparameters are parameters of the model that are not learned. For example, hyperparameters of our logistic regression SMS classifier include the value of the regularization term and thresholds used to remove words that appear too frequently or infrequently. In scikit-learn models use default which are good for start but can certainly be optimized. \n",
    "\n",
    "Grid search is a common method to select the hyperparameter values\n",
    "that produce the best model. Grid search takes a set of possible values for each hyperparameter that should be tuned, and evaluates a model trained on each element of the Cartesian product of the sets. That is, grid search is an exhaustive search that trains and evaluates a model for each possible combination of the hyperparameter values supplied by the developer.\n",
    "\n",
    "scikit-learn has a feature `GridSearchCV()`to find better parameter values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 384 candidates, totalling 1152 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  52 tasks      | elapsed:    2.1s\n",
      "[Parallel(n_jobs=-1)]: Done 352 tasks      | elapsed:   13.4s\n",
      "[Parallel(n_jobs=-1)]: Done 658 tasks      | elapsed:   27.2s\n",
      "[Parallel(n_jobs=-1)]: Done 1008 tasks      | elapsed:   44.1s\n",
      "[Parallel(n_jobs=-1)]: Done 1152 out of 1152 | elapsed:   49.9s finished\n"
     ]
    }
   ],
   "source": [
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('vect', TfidfVectorizer(stop_words='english')),\n",
    "    ('clf', LogisticRegression())\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'vect__max_df' : (0.25, 0.5, 0.75),\n",
    "    'vect__stop_words': ('english', None),\n",
    "    'vect__max_features': (2500, 5000, 10000, None),\n",
    "    'vect__ngram_range': ((1,1), (1,2)),\n",
    "    'vect__use_idf': (True, False),\n",
    "    'vect__norm': ('l1', 'l2'),\n",
    "    'clf__penalty': ('l1', 'l2')\n",
    "}\n",
    "\n",
    "\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1,\n",
    "                          scoring= 'accuracy', cv=3)\n",
    "scores = grid_search.fit(X_train_raw, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Score: 0.967\n",
      "Accuracy:  0.979181622398\n",
      "Precision:  0.987012987013\n",
      "Recall:  0.849162011173\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score\n",
    "\n",
    "prediction_grid = grid_search.predict(X_test_raw)\n",
    "glb = LabelBinarizer()\n",
    "pgb  = glb.fit_transform(prediction_grid)\n",
    "prediction_grid_binary = pgb.reshape(pgb.shape[0],)\n",
    "\n",
    "print 'Best Score: %0.3f' % grid_search.best_score_\n",
    "print 'Accuracy: ', accuracy_score(y_test_binary, prediction_grid_binary)\n",
    "print 'Precision: ', precision_score(y_test_binary, prediction_grid_binary)\n",
    "print 'Recall: ', recall_score(y_test_binary, prediction_grid_binary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-class Classification\n",
    "\n",
    "In many classification problems, however, there are more than two classes that are of interest. We might wish to predict the genres of songs from samples of audio, or classify images of galaxies by their types. The goal of multi-class classification\n",
    "is to assign an instance to one of the set of classes. scikit-learn uses a strategy called one-vs.-all, or one-vs.-the-rest, to support multi-class classification. Onevs.-all\n",
    "classification uses one binary classifier for each of the possible classes. The class that is predicted with the greatest confidence is assigned to the instance.\n",
    "LogisticRegression supports multi-class classification using the one-versus-all strategy out of the box. Let's use `LogisticRegression` for a multi-class classification problem.\n",
    "\n",
    "In this example, we will classify the sentiments of phrases taken from movie reviews in the Rotten Tomatoes data set. Each phrase can be classified as one of the following sentiments: negative, somewhat negative, neutral, somewhat positive, or positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    A series of escapades demonstrating the adage ...\n",
      "1    A series of escapades demonstrating the adage ...\n",
      "2                                             A series\n",
      "3                                                    A\n",
      "4                                               series\n",
      "Name: Phrase, dtype: object\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df_train = pd.read_csv('train.tsv', delimiter='\\t',header=0)\n",
    "print df_train['Phrase'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Analysing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PhraseId</th>\n",
       "      <th>SentenceId</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>156060.000000</td>\n",
       "      <td>156060.000000</td>\n",
       "      <td>156060.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>78030.500000</td>\n",
       "      <td>4079.732744</td>\n",
       "      <td>2.063578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>45050.785842</td>\n",
       "      <td>2502.764394</td>\n",
       "      <td>0.893832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>39015.750000</td>\n",
       "      <td>1861.750000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>78030.500000</td>\n",
       "      <td>4017.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>117045.250000</td>\n",
       "      <td>6244.000000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>156060.000000</td>\n",
       "      <td>8544.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            PhraseId     SentenceId      Sentiment\n",
       "count  156060.000000  156060.000000  156060.000000\n",
       "mean    78030.500000    4079.732744       2.063578\n",
       "std     45050.785842    2502.764394       0.893832\n",
       "min         1.000000       1.000000       0.000000\n",
       "25%     39015.750000    1861.750000       2.000000\n",
       "50%     78030.500000    4017.000000       2.000000\n",
       "75%    117045.250000    6244.000000       3.000000\n",
       "max    156060.000000    8544.000000       4.000000"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    156060.000000\n",
       "mean          2.063578\n",
       "std           0.893832\n",
       "min           0.000000\n",
       "25%           2.000000\n",
       "50%           2.000000\n",
       "75%           3.000000\n",
       "max           4.000000\n",
       "Name: Sentiment, dtype: float64"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['Sentiment'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    79582\n",
       "3    32927\n",
       "1    27273\n",
       "4     9206\n",
       "0     7072\n",
       "Name: Sentiment, dtype: int64"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['Sentiment'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    0.509945\n",
       "3    0.210989\n",
       "1    0.174760\n",
       "4    0.058990\n",
       "0    0.045316\n",
       "Name: Sentiment, dtype: float64"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['Sentiment'].value_counts()/df_train['Sentiment'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most common class, Neutral, includes more than 50 percent of the instances. Accuracy will not be an informative performance measure for this problem, as a degenerate classifier that predicts only Neutral can obtain an accuracy near 0.5.\n",
    "Approximately one quarter of the reviews are positive or somewhat positive, and approximately one fifth of the reviews are negative or somewhat negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 24 candidates, totalling 72 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   54.3s\n",
      "[Parallel(n_jobs=-1)]: Done  72 out of  72 | elapsed:  2.9min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=Pipeline(memory=None,\n",
       "     steps=[('vect', TfidfVectorizer(analyzer=u'word', binary=False, decode_error=u'strict',\n",
       "        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), norm=u'l2', preprocessor=None, smooth_idf=True...ty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False))]),\n",
       "       fit_params={}, iid=True, n_jobs=-1,\n",
       "       param_grid={'vect__ngram_range': ((1, 1), (1, 2)), 'vect__use_idf': (True, False), 'clf__C': (0.1, 1, 10), 'vect__max_df': (0.25, 0.5)},\n",
       "       pre_dispatch='2*n_jobs', refit=True, scoring='accuracy', verbose=1)"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_train['Phrase'],\n",
    "                                                   df_train['Sentiment'])\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('vect', TfidfVectorizer(stop_words='english')),\n",
    "    ('clf', LogisticRegression())\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'vect__max_df': (0.25, 0.5),\n",
    "    'vect__ngram_range': ((1,1), (1,2)),\n",
    "    'vect__use_idf': (True, False),\n",
    "    'clf__C': (0.1, 1, 10)\n",
    "}\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='accuracy')\n",
    "grid_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Score:  0.636814900252\n",
      "Best Parameter set:\n",
      "clf__C : 10\n",
      "vect__max_df : 0.25\n",
      "vect__ngram_range : (1, 2)\n",
      "vect__use_idf : False\n"
     ]
    }
   ],
   "source": [
    "print 'Best Score: ', grid_search.best_score_\n",
    "print 'Best Parameter set:'\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print param_name, ':', best_parameters[param_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-class Perfoemance Metrics\n",
    "\n",
    "As with binary classification, confusion matrices are useful for visualizing the types of errors made by the classifier. Precision, recall, and F1 score can be computed for each of the classes, and accuracy for all of the predictions can also be calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "prediction_multi = grid_search.predict(X_test)\n",
    "mets= precision_recall_fscore_support(y_test, prediction_multi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precison: 0.58\n",
      "Recall: 0.51\n",
      "F1 score: 0.54\n"
     ]
    }
   ],
   "source": [
    "print 'Precison: %.2f' %mets[0].mean()\n",
    "print 'Recall: %.2f' %mets[1].mean()\n",
    "print 'F1 score: %.2f' %mets[2].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use `precision_recall_fscore_support` as a method for evaluating all three metrics in a single command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  624   905   260    34     3]\n",
      " [  466  3284  2807   211     6]\n",
      " [  108  1550 16420  1682    72]\n",
      " [    8   179  3133  4346   602]\n",
      " [    3    16   213  1252   831]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARkAAAD0CAYAAACvvVL/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XmcXWWd5/HPN3vCElYxbB0aAw4wLAEBYWQQJCxtE7sHFUVBxM4oSKu4EXUGRbGl7RGlVeggyDoERBgZQWNEaIUhEJawhDVs3aHDGnYwJFXf+eN5bnJT1L11quqeOnVv/d6+zou65z73nKfK1K+e7Tw/2SaEEMoyquoKhBA6WwSZEEKpIsiEEEoVQSaEUKoIMiGEUkWQCSGUKoJMCKFUEWQqJmmipP8r6SVJvxjEdY6S9LtW1q0qkt4j6cGq6xFaQ7EYrxhJHwVOAt4JvAIsAk6zfeMgr/tx4ERgH9urBl3RYU6SgWm2l1RdlzA0xlRdgXYg6STgZODTwDzgTeAQYCYwqCAD/AXw0EgIMEVIGjOSfhYHv3cdP7+8q1DZ2+9eMc/2ISVXqfVsx9HkACYDrwIfbFJmPPBD4D/y8UNgfH5vf2Ap8EXgGWAZcGx+71ukgLUy3+M44JvAxXXXngoYGJNffwJ4lNSaegw4qu78jXWf2wdYCLyU/7tP3Xs3AN8GbsrX+R2wSYPvrVb/r9TV/wPAYcBDwHLga3Xl9wRuBl7MZX8MjMvv/TF/L6/l7/fDddf/KvAUcFHtXP7Mtvke0/PrzYFngf2r/rfRimP6zuO9ctm2hQ7gtqrrO5AjxmT69m5gAnBVkzJfB/YGdgV2If2ifaPu/beTgtUWpEDyE0kb2j4F+C5wme11bZ/brCKS1gHOBA61vR4pkCzqpdxGwDW57MbAD4BrJG1cV+yjwLHA24BxwJea3PrtpJ/BFsD/BM4BPgbsDrwH+B+Stsllu4AvAJuQfnYHAscD2N4vl9klf7+X1V1/I1Krblb9jW0/QgpAF0uaBPwcuMD2DU3q20ZMl7sLHUVIOk/SM5Lu7XH+REkPSFos6R/rzs+WtETSg5IOrjt/SD63RNLJdee3kXRLPn+ZpHF91SmCTN82Bp5z8yb8UcCptp+x/SyphfLxuvdX5vdX2r6W9Fd8+wHWpxvYSdJE28tsL+6lzF8BD9u+yPYq25cCDwB/XVfm57Yfsv0GcDkpQDaykjT+tBKYSwogP7L9Sr7/faTgiu3bbS/I930c+Bfgvxb4nk6xvSLXZy22zwGWALcAU0hBvSMY6MaFjoLOJ3XlV5P0XlLXfhfbOwL/lM/vABwJ7Jg/81NJoyWNBn4CHArsAHwklwU4HTjD9juAF0h/NJuKINO354FNJDUbv9oceKLu9RP53Opr9AhSrwPr9rcitl8jdTE+DSyTdI2kdxaoT61OW9S9fqof9Xnedm3goBYEnq57/43a5yVtJ+nXkp6S9DKppbZJk2sDPGv7z32UOQfYCfhn2yv6KNtWugv+rwjbfyR1L+t9Bvhe7edm+5l8fiYwNwf3x0iBfM98LLH9qO03SX9YZkoScABwRf78BaSuc1MRZPp2M7CC5j/M/yA19Wu2zucG4jVgUt3rt9e/aXue7YNIf9EfIP3y9VWfWp2eHGCd+uMsUr2m2V4f+BqgPj7T9M+0pHVJ41znAt/M3cGOYEyXix2DsB3wntzN+VdJ78rntwD+va7c0nyu0fmNgRfr/mDWzjfVUUGmUT9yMGy/RBqH+ImkD0iaJGmspEPr+raXAt+QtKmkTXL5FyU9QxpD6I9FwH6StpY0GZhde0PSZpJm5rGZFaRuV29/4q4FtpP0UUljJB0PvAuYLWkxBf5hDMJ6wMvAq7mV9Zke7z8NbC/pVkl3kX4+6/dxzR+RBj0/RRprOrvFdQYgdxXulPTrMq7fSD+6S5tIuq3umNXXtbMxpDGvvYEvA5fnVsmQ6Jgp7Lp+5EGkCLtQ0tW27xvstW3/L0lPkQZzLyHNyNwOnJaLfIf0i3J3fv0L4GpSs/WX/bzXfEmX5Ws9R+oDH57fHkVaq3Mh6a//It76S4zt5yW9n/TLeRbwOPDfbZ8naT3SL/qU/tSrH74EzCHNRt0JXEZqYtd8E/gZMJE0yPs8cK2kvW0v6HkxSTNJ4wX/OZ86CVgk6Sjbl7S47p8D7qfvoNcyBrqKj7c8Z3uPAdxmKXClbQO3SuomdWGfBLaqK7cla1q7vZ1/HtigbplBffnGqp7eatVBmsmYV/d6NjC74jpNBe6t+mfTS71+BRxUdT1yXSYBdwB7VVyPLYHrSAHx10N13513HutlS6cUOig4hd3z3x1pDO/U/PV2pK6QSAO+d5GWYGxDWhoxmtT4eDSfG5fL7Jg//wvgyPz12cDxfdWnk7pLjfqRoY6kqcBupJmaKusxWtIi0tqb+bYrrQ9pzOcr9N79LFV3waMISZeSxhG3l7RU0nHAecBf5mntucAxThaTZhbvA34LnGC7y6mV8lnSwtP7gcu9Zhbzq8BJkpaQxmiaLruADuouhb7lAdRfAp+3/XKVdXGardpV0gbAVZJ2sn1vX58rQ+5aPmP7dkn7D+W9jfvTXer7evZHGrz1sQblT2NNt7/+/LWksb2e5x8lzT4V1kktmWb9yxFP0lhSgLnE9pVV16fG9ovA9fRY2zHE9gUOl/Q46S/9AZIuHpI7G7oKHu2qk4LMQmBaXpE4jrTI6OqK6zQs5JmEc4H7bf9gGNRn09yCQdJE0mD9A1XVx/Zs21vankr6d/MH273+5W/5vWltd2k46pgg00c/csg16BtXZV/SCuQDJC3Kx2EV1mcKcL2ku0l/HObbHtJp4+FDdBU82lVs9RBChXbaeZx/eU1fC6KTd2697HYPbAq7UjHwG0KFDLzZOR2KXkWQCaFi3W7frlAREWRCqFBa8RtBJoRQEiO6Ory71HHfXT8eGhsSUZ/mhlN9qqpLt1XoaFcdF2TosbPaMBD1aW441WfI61LrLnXyFHZ0l0KolOhyJ/6tX2NYBZlxmuCJo/q9YdxaJmgdJo/epDWLf0YN/q/HhFHrMnnspi2pj7sGv+5zApNYf9TGg69Pi9ZXTWAS62ujYbFYq1V1+TOv8aZXFPrHY2Alowd7y2FtWAWZiaPWZe91D++74BDR+D73SB5S3S+9UnUVVvPKN6uuwrB1i68rXNaOlkwIoWTdbTzeUkQEmRAqlAZ+oyUTQihNdJdCCCVKWz1EkAkhlMSINx2zSyGEEnV3eHeps7+7EIa52sBvkaOIRrmw83tflOScGwwlZ+Y8ZXdLml5X9hhJD+fjmLrzu0u6J3/mzCL5myLIhFAhI7pc7CjofHrZL1nSVsAM4N/qTh8KTMvHLFKOLnKGzlOAvUibhp8iacP8mbOAv6v7XJ97M0eQCaFi3YwqdBTh3nNhA5xBSvlSv6J5JnBhTo+ygJS4bQpwMGlL1OW2XwDmA4fk99a3vcBpS80LKZALO8ZkQqiQTX+msDeRdFvd6zm25/T1oZyF80nbd/Xo3fQ3F/YW+eue55uKIBNCpdSfFb/9TlMraRLwNVJXqRIRZEKokIE3Xeqv4bakdLO1VsyWwB2S9qRxrrIngf17nL8hn9+yl/JNxZhMCBUyxTasGuimVbbvsf0221NzXqmlwHTbT5Hykh2dZ5n2Bl6yvYyUVmiGpA3zgO8MUp75ZcDLkvbOs0pHk/KqN1VqkJF0iKQH83TXyWXeK4R21eIp7P7k+7oWeBRYApwDHA9geznwbVJOrIXAqfkcuczP8mceAX7TV51Ka6dJGg38hJQdcCmwUNLVtu8r654htBvT2sV4TXJh196fWve1gRMalDsPOK+X87cBO/WnTmW2ZPYElth+1PabpBzDM0u8XwhtqPMzSJY54tTbNNhePQvlzZtnQdrVLoSRpNUtmeGo8tmlPM8/B2jdtpkhtJF2bqUUUWaQaTQ9FkLIbLGyu/K/9aUqs522EJgmaRtJ44AjSVNmIYQs7SejQke7Ki2E2l4l6bOkOffRwHm2F5d1vxDaU+yMNyi2ryXNxYcQepEGftu3lVJEZ3cGQ2gDsZF4CKE0tccKOlkEmRAqFhuJhxBKY8PK7ggyIYSSpO5SBJkQQolixW8IoTQxhR1CKFl0l0IIJWvnRwaKiCATQoVStoIIMiGEkhixqruzc2F3dmcwhDbQyqewe0tTK+n7kh7IqWivkrRB3Xuz8x7cD0o6uO58r/tz510VbsnnL8s7LDQ1/FoyXV1V12C1FTtPrboKaxn9xqqqq7DaqIXD64F6rxo+P5v+KGF26Xzgx6TsjjXzgdl5Z4TTgdnAVyXtQNqCZUdgc+D3krbLn2m0P/fpwBm250o6GziOnN62kWjJhFCxbo8qdBTRW5pa27+zXYvCC1iTO2kmMNf2CtuPkTIQ7EmD/blzGpQDgCvy5y8g0tSGMMz1L6fSgNLU9vBJ4LL89RakoFNTn3a2t/25NwZerAtYkaY2hOGutjNeQf1OU1tP0teBVcAlA73GQESQCaFiQ7HiV9IngPcDB+Z8S9B8H+7ezj8PbCBpTG7NRJraEIY7A6u6RxU6BkrSIcBXgMNtv1731tXAkZLGS9oGmAbcSoP9uXNwuh44In/+GAqkqY2WTAgVavWmVTlN7f6k8ZulwCmk2aTxwPw0dssC25+2vVjS5cB9pG7UCba78nUa7c/9VWCupO8AdwLn9lWnCDIhVKyVjxU0SFPbMBDYPg04rZfzve7PbftR0uxTYRFkQqiS4ynsEEKJYquHEELpIsiEEEpjRFfs8RtCKFPsJxNCKI1j4DeEUDZ3eJAprTPY274WIYSe0mK8Ike7KnPE6XzgkBKvH0JHsFXoaFeldZds/1HS1LKuH0IniHUyQ0DSLGAWwAStU3FtQhhisZF4+fKmO3MAJo/exH0UD6GjmM4f+K08yIQwsrX3oG4REWRCqJg7vP1e5hT2pcDNwPaSlko6rqx7hdDOYnZpgBrsaxFCqGPHmEwIoWQxJhNCKFV3d2cHmc5+xjyEYc4UG48p2qVqkKZ2I0nzJT2c/7thPi9JZ+aUs3dLml73mWNy+YclHVN3fndJ9+TPnJkTvjUVQSaEirngUdD5vPVxnpOB62xPA67LrwEOJWUomEZaEHsWpKBE2oB8L9J+vqfUAlMu83d1n+vz0aEIMiFUya2dXeotTS0pHe0F+ev61LIzgQudLCDlVJoCHAzMt73c9gukXNqH5PfWt70gp0e5kEhTG0IbKN5MGWia2s1sL8tfPwVslr/egremo92ij/NLeznfVMMgI2n9Zh+0/XJfFw8h9K0fU9iDSlOb7mVLGtLlf81aMotJMbb+J1B7bWDrEusVwogxBCt+n5Y0xfay3OV5Jp9vlKb2SVKCuPrzN+TzW/ZSvqmGYzK2t7K9df7vVj1eR4AJoQVscPeoQscgXE1KKQtrp5a9Gjg6zzLtDbyUu1XzgBmSNswDvjOAefm9lyXtnWeVjqZVaWolHQn8pe3vStqS1Me7vR/fZAihgVa2ZBqkqf0ecHl+tOcJ4EO5+LXAYcAS4HXg2FQfL5f0bVJObIBTbdcGk48nzWBNBH6Tj6b6DDKSfgyMBfYDvpsrczbwrr4+G0IooIVBpsnjPAf2UtbACQ2ucx5wXi/nbwN26k+dirRk9rE9XdKd+SbLJY3rz01CCI2098OPRRQJMisljSLHW0kbA91lVMbd3XS/8UYZlx6QCY89V3UV1nLNTX12f4fMYfv9TdVVWEvXI49XXYU1+tsyia0e+AnwS2BTSd8CbgROL7VWIYwULV6MNxz12ZKxfaGk24H35VMftB1pTkJolQ5vyRRd8TsaWEn6ccSjCCG0Uhu3UoroM2BI+jpwKbA5afHN/5Y0u+yKhTBitPgJyeGmSEvmaGA3268DSDoNuBP4hzIrFsKIYDq+JVMkyCzrUW5MPhdCaIFO30i82QOSZ5Di7HJgsaR5+fUM1qwEDCEM1kgNMkBtBmkxcE3d+QXlVSeEEWikdpdsnzuUFQlhRDKolKWtw0eRZ5e2BU4DdgAm1M7b3q7EeoUwQqjjWzJF1rycD/yctI/MocDlwGUl1imEkaXDp7CLBJlJtucB2H7E9jdIwSaE0AodHmSKTGGvyA9IPiLp06SdsNYrt1ohjCBtHECKKNKS+QKwDvD3wL6kdAif7OtDkraSdL2k+yQtlvS5wVU1hA5UW4xX5GhTRR6QvCV/+Qrw8X5cexXwRdt3SFoPuF3SfNv3DaCeIXSsod3We+g1W4x3FU0acrb/ttmF836gy/LXr0i6n5Q+IYJMCPVGapABftyqm0iaCuwG3NK8ZAgjTytbMpK+AHyKFLruIe3bOwWYC2wM3A583PabksaTErTtDjwPfNj24/k6s4HjgC7g72uTPwPRbDHedQO9aD1J65I2vfp8b7maJM0ipchkApNaccsQ2kuLxlskbUEaO93B9huSLgeOJG0WfobtuZLOJgWPs/J/X7D9jpws4HTgw5J2yJ/bkbT7wu8lbWe7ayD1KnVvGEljSQHmEttX9lbG9hzbe9jeYyzjy6xOCMNP0enr4q2dMcBESWOASaQhiwOAK/L7PdPU1tLXXgEcmFOdzATm2l5h+zFSNoM9B/YNlhhkcmXPBe63/YOy7hNC22tRkLH9JPBPwL+RgstLpO7Ri7ZX5WL1qWVXp6PN779E6lI1SlM7IIWDTO6/9ce+pNmoAyQtysdh/bxGCB1PLnaQc2HXHbPWuk5KxDYT2IbUzVkHOGSov5+eijy7tCepRTIZ2FrSLsCnbJ/Y7HO2b2TtFLchhN4U7wr1lQv7fcBjtp8FkHQl6Y/9BpLG5NZKfWrZWprapbl7NZk0ANwofe2AFGnJnAm8P98c23cB7x3oDUMIayg/hV3kKODfgL0lTcrDFQeSloxcDxyRy/RMU1tLX3sE8Iec8O1q4EhJ4yVtA0wDbh3o91jksYJRtp9IdV5tQKPMIYRetGh2yfYtkq4A7iAthr0TmEPaD2qupO/kc7VtXM4FLpK0hLQ53ZH5OovzzNR9+TonDHRmCYoFmX/PXSZLGg2cCDw00BuGEHpobZraU0j5r+s9Si+zQ7b/DHywwXVOI23xMmhFgsxnSF2mrYGngd/ncyGEFhixjxXU2H6G3IwKIZRgpAcZSefQy4/B9qxeiocQ+sPRkoHUPaqZAPwNay/UCSEMxkgPMrbX2mpT0kXAjaXVKIQRptM3Eh/IYwXbAJu1uiIhhM5UZEzmBdY06EaR5tNPLrNSIYwoI7m7lFcN7sKaJcXdeUVgCKEVRsDAb9PuUg4o19ruykeH/zhCqEBkK2CRpN1s31l6bWBYZR9f9cTSqquwloM+cmzVVVjtjV3HVV2Ftaz39HNVV2E1vdrPoc7h80++FM32+K09tbkbsFDSI8BrpCerbXv6ENUxhI4lOr+71KwlcyswHTh8iOoSwsgzwnNhC1LWyCGqSwgj0whuyWwq6aRGb8aWmiG0yAgOMqOBdYnd7UIo1Ugek1lm+9Qhq0kII9UIDjLRggmhbG2+BqaIZkHmwCGrRQgjWKfPLjVcNWR7+VBWJISRqh8pUfq+lrSBpCskPSDpfknvlrSRpPmSHs7/3TCXlaQzJS2RdLek6XXXOSaXf1jSMY3v2LdSM0iGEApo7WMFPwJ+a/udpOcO7yc90Hyd7WnAdax5wPlQUiaCaaRU0WcBSNqItE/wXqS9gU+pBaaBiCATQpVamKZW0mRgP3I2Attv2n6RtdPR9kxTe6GTBaT8TFOAg4H5tpfbfgGYzyCSxEWQCaFC6sdRwDbAs8DPJd0p6WeS1gE2s70sl3mKNftBNUpHW02a2hBCSYq3ZJqmqSVN5EwHzrK9G+lZw7X2fso7KQzpfFaRp7BDCCXqx2K8vtLULgWW2r4lv76CFGSeljTF9rLcHXomv98oHe2TwP49zt9QuJY9lNaSkTRB0q2S7pK0WNK3yrpXCG2tu+DRB9tPkZIxbp9P1dLU1qej7Zmm9ug8y7Q38FLuVs0DZkjaMA/4zsjnBqTMlswK4ADbr0oaC9wo6Td5gCmEAGXsjHcicImkcaTMkceSGhOXSzoOeAL4UC57LXAYsAR4PZfF9nJJ3wYW5nKnDmZJS2lBJvf9Xs0vx+ajw9c2hjAArU1TuwjorUv1lsW1+Xf0hAbXOQ84rxV1KnXgV9JoSYtIfcD5dX3F+jKzagNZK1lRZnVCGJZauRhvOCo1yOR9gXclDRztKWmnXsrMsb2H7T3GMr7M6oQwPHX4Hr9DMoWdFwRdzyAW9ITQqaIlM0CSNpW0Qf56InAQ8EBZ9wuhLbVwxe9wVebs0hTgAkmjyaPbtn9d4v1CaDui85/CLnN26W5SpoMQQjNt3EopIlb8hlAxDaNcY2WIIBNCldp8vKWICDIhVKydZ46KiCATQtUiyIQQyhQtmRBCeUZ4mtoQwlCIlkwIoSwiukshhLLFOpkQQpmiJRNCKE8sxgshlC1ml0ay7q6qa7CW0TfdU3UVVpu82duqrsJaPGYY/VNWwSxJteIdHmQi71IIVTJp4LfIUVDe9vZOSb/Or7eRdEvOeX1Z3mQcSePz6yX5/al115idzz8o6eDBfIsRZEKoWAk7432OlAO75nTgDNvvAF4AjsvnjwNeyOfPyOWQtANwJLAjaTfLn+Z9oQYkgkwIVWvhzniStgT+CvhZfi3gAFKiN3hrLuxajuwrgANz+ZnAXNsrbD9GSpmy50C/vQgyIVSothivYEumrzS1AD8EvsKadHAbAy/aXpVf1+e1Xp3zOr//Ui7f0lzYw2i0LIQRqH/jLU3T1Ep6P/CM7dsl7d+K6rVCBJkQKtbC2aV9gcMlHQZMANYHfgRsIGlMbq3U8l3DmlzYSyWNASYDz9M4R/aARHcphIq1auDX9mzbW9qeShq4/YPto0jpiI7IxXrmwq7lyD4il3c+f2SefdoGmAbcOtDvL1oyIVTJQHfpS36/CsyV9B3gTuDcfP5c4CJJS4DlpMCE7cWSLgfuA1YBJ9ge8KKxCDIhVK2EGGP7BuCG/PWj9DI7ZPvPwAcbfP404LRW1CWCTAgViwckQwjliq0eQghlipZMCKE0Mqj8gd9KRZAJoWrxFPbg9HwiNISwNtmFjnY1FIvxej4RGkKoKfpwZPvGmHKDTM8nQkMIPRXcS6aNWzJlj8nUnghdr1GB/CTpLIAJTCq5OiEMP50+u1RaS6b+idBm5WzPsb2H7T3GMr6s6oQwfEVLZsDe8kSopIttf6zEe4bQXgzqat8AUkRpLZkGT4RGgAmhpw4f+I11MiFUrJ2np4sYkiBT/0RoCKGHCDIhhNKYjl/xG0EmhAqJ9l7NW0QEmRCqFkEmhFAaAzGFHUIoU6sekJS0laTrJd0nabGkz+XzG0maL+nh/N8N83lJOjOno71b0vS6ax2Tyz8s6ZhG9ywigkwIVWvdit9VwBdt7wDsDZyQU86eDFxnexpwXX4NcCgpE8E00qM9Z0EKSsApwF6kvYFPqQWmgYggE0KlWveApO1ltu/IX79C2v1gC9ZOR9szTe2FThaQ8jNNAQ4G5ttebvsFYD4pJ/aAxJhMCFUy/Rn43UTSbXWv59ie01tBSVOB3YBbgM1sL8tvPQVslr9ulI420tSG0FGKr5Npmqa2RtK6wC+Bz9t+WdLq92xbGtrnvqO7FELFWrkznqSxpABzie0r8+mnczeI/N9n8vlG6WgjTW0IHcNAV3exow9KTZZzgftt/6Durfp0tD3T1B6dZ5n2Bl7K3ap5wAxJG+YB3xn53IBEdymESrV0r5h9gY8D90halM99DfgecLmk44AngA/l964FDgOWAK8DxwLYXi7p28DCXO5U28sHWil5GK02lPQs6YcwGJsAz7WgOq0S9WluONWnVXX5C9ubFik4ecLbvc9WRxe66G+XfP/2ImMyw82waskU/T+mGUm3Daf/I6I+zQ2n+lRWl2H0h74MwyrIhDDiGIjkbiGE8hjc2Xs9dGKQ6XVxUoWiPs0Np/oMfV1qs0sdrOOmsButgBwKkrokLZJ0r6RfSJo00PpI2r+WdVPS4ZJOblJ2A0nHF7lufX0kfVPSlwZSvx73nyrpowP5bK0+kj4hafPB1mUwKvu30+HZCjouyFTsDdu72t4JeBP4dP2beT1Cv3/mtq+2/b0mRTYACgWZkkwFBhRk6nwCqDTIVCaCTBigPwHvyH/lH5R0IXAvsJWkGZJulnRHbvGsCyDpEEkPSLoD+NvahfJf+R/nrzeTdJWku/KxD2kdxLa5FfX9XO7LkhbmR/i/VXetr0t6SNKNwPa9VTzX+Q/5s9dJ2jqfP1/SEXXlXs1ffg94T77/F3J9fyXphrxVwCl117237vNfyq2pI4A9gEvyNSYO8mffRjo/g2QEmRJIGkN6jP6efGoa8FPbOwKvAd8A3md7OnAbcJKkCcA5wF8DuwNvb3D5M4F/tb0LMB1YTHp0/5HcivqypBn5nnsCuwK7S9pP0u6k9DS7khZhvavBPf4ZuMD2zsAl+Z7NnAz8Kd//jHxuT+C/ATsDH5TUcGrY9hX553BUvsYbfdyvcxjo7i52tKlOHPit0sS6lZZ/Ii3x3hx4Ij9KD2mfjx2Am/KDa+OAm4F3Ao/ZfhhA0sXk9L09HAAcDWC7C3ipl70+ZuTjzvx6XVLQWQ+4yvbr+R5XN/g+3s2altRFwD/2+Z2/1Xzbz+f7XAn8F+D/DOA6na+NWylFRJBprTds71p/IgeS1+pPkX4BP9Kj3FqfGyQB/2D7X3rc4/ODvO4qcus3jy2Na1K252+O6z+fTRhkfTpDhweZ6C4NvQXAvpLeASBpHUnbAQ8AUyVtm8t9pMHnrwM+kz87WtJk4BVSK6VmHvDJurGeLSS9Dfgj8AFJEyWtR+qa9eb/kbpVAEeRWmUAj5O6cgCHA2Pz1z3vD3CQ0raPE0mbJN0EPA28TdLGksYD768r39s1Op+Nu7oKHe0qWjJDzPazkj4BXJp/0QC+YfshSbOAayS9TvrF7u2X7nPAnPywWxfwGds3S7opD6r+Jo/L/Cfg5tySehX4mO07JF0G3EV63H9hL9cHOBH4uaQvA8+SH5wjjRn9StJdwG9Z00K7G+jK588HXgCed518AAAA3klEQVRuJW05sCVwse3bACSdmt97khRYa84Hzpb0BvDuETUu0+ErfofVA5KhM+Qguoftz1Zdl+Fu8phN/e71ZhYqO+/Fc+MByRBCP9ltPXNURASZ0HK2zyd1f0IRHd6biCATQsUcLZkQQnnaezVvERFkQqiSgTaeni4igkwIFTLgDp/CjiATQpUcm1aFEErW6S2ZWIwXQoUk/ZaUJaGI52wPOCd1VSLIhBBKFQ9IhhBKFUEmhFCqCDIhhFJFkAkhlCqCTAihVBFkQgiliiATQihVBJkQQqkiyIQQSvX/AVDxh3aoZ4b7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fac4e5bfdd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "confusion_mat = confusion_matrix(y_test, prediction_multi)\n",
    "print confusion_mat\n",
    "plt.matshow(confusion_mat)\n",
    "plt.title('Confusion matrix')\n",
    "plt.xlabel('Predicted output')\n",
    "plt.ylabel('True label')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Important References"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scikit",
   "language": "python",
   "name": "scikit"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}