Assignment2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2: Support Vector Machines\n",
    "\n",
    "In this assignment you will:\n",
    "* Implement a linear SVM via stochastic gradient descent\n",
    "* Do multi-class handwritten digit classification with off-the-shelf SVM solvers (with kernel functions)\n",
    "* Tune model hyper-parameters to achieve good generalisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Implement a linear SVM\n",
    "\n",
    "## Setup\n",
    "\n",
    "First import the required packages and do some setup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.io import loadmat\n",
    "from sklearn.model_selection import train_test_split\n",
    "from assignment2 import svm_loss, svm_gradient, svm_solver\n",
    "from sklearn import preprocessing \n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Set default parameters for plots\n",
    "plt.rcParams['figure.figsize'] = (12.0, 6.0)\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now load the face dataset. The labels are set to **-1 for non-face and 1 for face**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "data = loadmat('faces.mat')\n",
    "labels = np.squeeze(data['Labels'])\n",
    "labels[labels == -1] = -1    # Want labels in {0, 1}\n",
    "data = data['Data']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Divide the dataset into train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data = StandardScaler().fit_transform(data)\n",
    "X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.3)\n",
    "num_train = X_train.shape[0]\n",
    "num_test = X_test.shape[0]\n",
    "\n",
    "# As a sanity check, we print out the size of the training and test data.\n",
    "print('Training data shape: ', X_train.shape)\n",
    "print('Training labels shape: ', y_train.shape)\n",
    "print('Test data shape: ', X_test.shape)\n",
    "print('Test labels shape: ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize some examples to check that the data is fine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize some examples from the dataset.\n",
    "samples_per_class = 10\n",
    "classes = [-1, 1]\n",
    "train_imgs = np.reshape(X_train, [-1, 24, 24], order='F')\n",
    "\n",
    "for y, cls in enumerate(classes):\n",
    "    idxs = np.flatnonzero(np.equal(y_train, cls))\n",
    "    idxs = np.random.choice(idxs, samples_per_class, replace=False)\n",
    "    for i, idx in enumerate(idxs):\n",
    "        plt_idx = y * samples_per_class + i + 1\n",
    "        plt.subplot(len(classes), samples_per_class, plt_idx)\n",
    "        plt.imshow(train_imgs[idx])\n",
    "        plt.axis('off')\n",
    "        plt.title(cls)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1: The SVM loss function [15 Points]\n",
    "\n",
    "Remember the formulation of the SVM optimization problem as:\n",
    "\n",
    "\\begin{aligned}\n",
    "& \\min_{w, b}\n",
    "& & \\frac{1}{2}||w||^2 + C\\sum_{i=1}^m \\xi_i \\\\\n",
    "& \\ \\text{ s.t.}\n",
    "& & y^{(i)}(w^Tx^{(i)}+b) \\geq 1-\\xi_i, \\; i = 1, \\ldots, m \\\\\n",
    "& & & \\xi_i \\geq 0, \\; i = 1, \\ldots, m \\\\\n",
    "\\end{aligned}\n",
    "\n",
    "Let $f(x)=w^Tx+b$. The constraints can then be written as $y^{(i)}f(x^{(i)})\\geq 1-\\xi_i$. Together with the constraints $\\xi_i \\geq 0$ this leads to $\\xi_i=\\max(0, 1-y^{(i)}f(x^{(i)}))$. The above constraint optimization problem is therefore equivalent to the following **unconstraint** problem:\n",
    "\n",
    "\\begin{equation}\n",
    "\\min_{w, b} \\frac{\\lambda}{2}||w||^2 + \\frac{1}{m}\\sum_{i=1}^m \\max(0, 1-y^{(i)}f(x^{(i)}))\n",
    "\\end{equation}\n",
    "\n",
    "The first term in this objective is a regularization term (prevents overfitting) and the second term measures the classification loss. Here the parameter $\\lambda=1/C$ is a **hyper-parameter** that controls the relative weight of  both losses.\n",
    "\n",
    "\n",
    "**TODO**: Implement the **unconstrained** objective function for SVM in *svm_loss.py* according to the specifications.\n",
    "\n",
    "***HINT***: Consider again what you would expect with the provided values of w, b and C."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Test your cost-function\n",
    "w_0 = np.zeros(X_train.shape[1])\n",
    "b_0 = 0.\n",
    "l_0 = svm_loss(w_0, b_0, X_train, y_train, 1.)\n",
    "print('SVM-loss with initial parameters: ', l_0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: The SVM Gradient [15 Points]\n",
    "\n",
    "**TODO**: Implement the gradient of **w** and **b** w.r.t. the above unconstrained objective. The gradient will be computed on a mini-batch (i.e., a random subset of the training set).\n",
    "\n",
    "**Hint**: Don't worry about the fact that $\\max(0, 1-y^{(i)}f(x^{(i)}))$ is not differentiable at $1-y^{(i)}f(x^{(i)})=0$. Just pick a one-sided gradient (this is called a subgradient for convex functions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Test your implementation\n",
    "x_ = np.ones([2, 10]) #Part of training set? \n",
    "y_ = np.array([1, -1])\n",
    "w_0 = np.zeros(10)\n",
    "b_0 = 0.\n",
    "grad_w, grad_b = svm_gradient(w_0, b_0, x_, y_, 1.)\n",
    "print(grad_w)\n",
    "print(grad_b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3: The SVM Solver [20 Points]\n",
    "\n",
    "You will implement the **Pegasos** algorithm - a variant of SGD - to solve for the parameters **w** and **b**. \n",
    "\n",
    "The algorithm was introduced in the following [Paper](http://ttic.uchicago.edu/~nati/Publications/PegasosMPB.pdf) (see Figure 2). It is essentially Stochastic Gradient Descent on mini-batches + a specific choice for the learning rate giving convergence guarantees. The required steps are outlined in **svm_solver.py**. For more details, please refer to the [Paper](http://ttic.uchicago.edu/~nati/Publications/PegasosMPB.pdf).\n",
    "\n",
    "**TODO**: Implement the Pegasos algorithm in **svm_solver.py** according to specs. \n",
    "\n",
    "***HINT***: You know what performance to expect from the previous assignment (maybe you need to tune the hyper-parameter **C**)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# We'll meausure the execution time\n",
    "start = time.time()\n",
    "\n",
    "C = 1.0\n",
    "#w, b = svm_solver(X_train, y_train, C, num_iter=30000, num_per_batch=64)\n",
    "w, b = svm_solver(X_train, y_train, C, num_iter=50000, num_per_batch=64)\n",
    "\n",
    "exec_time = time.time()-start\n",
    "print('Total exection time: {}s'.format(exec_time))\n",
    "\n",
    "# We can have a look at what theta has learned to recognise as \"face\"\n",
    "plt.imshow(np.reshape(w, [24, 24], order='F'))\n",
    "plt.title('Learned w')\n",
    "plt.show()\n",
    "\n",
    "# Make predictions\n",
    "preds_train = np.ones_like(y_train) - 2 * (np.dot(X_train, w) + b < 0)\n",
    "preds_test = np.ones_like(y_test) - 2 * (np.dot(X_test, w) + b < 0)\n",
    "\n",
    "print('Accuracy train: {}'.format(np.mean(preds_train == y_train)))\n",
    "print('Accuracy test: {}'.format(np.mean(preds_test == y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Multi-Class SVM\n",
    "\n",
    "In this second part of the assignment you will train a multi-class SVM on a dataset of handwritten digits.\n",
    "\n",
    "A seperate (withheld) test set will be used for the evaluation of your classifiers. It is therefore important that you take good care not to overfit and ensure that your models generalize to unseen data.\n",
    "\n",
    "\n",
    "## Setup\n",
    "\n",
    "First load and visualize some of the data:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import preprocessing\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "classes = range(10)\n",
    "\n",
    "data = np.load('data.npy', encoding = 'latin1')\n",
    "X = data[()]['X']\n",
    "y = data[()]['y']\n",
    "\n",
    "# Visualize some examples from the dataset.\n",
    "samples_per_class = 5\n",
    "imgs = np.reshape(X, [-1, 16, 16])\n",
    "labels = y\n",
    "\n",
    "for j, cls in enumerate(classes):\n",
    "    idxs = np.flatnonzero(np.equal(labels, cls))\n",
    "    idxs = np.random.choice(idxs, samples_per_class, replace=False)\n",
    "    for i, idx in enumerate(idxs):\n",
    "        plt_idx = j * samples_per_class + i + 1\n",
    "        plt.subplot(samples_per_class, len(classes), plt_idx)\n",
    "        plt.imshow(imgs[idx])\n",
    "        plt.axis('off')\n",
    "        plt.title(cls)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4: Preparing the data [10 Points]\n",
    "\n",
    "In this exercise you should:\n",
    "* Prepare the data for cross validation (train/test splits)\n",
    "* Preprocess the data for use with the SVM (this can have a **big** effect on the performance!)\n",
    "\n",
    "**NOTE:** You are allowed to use any functions in sklearn for this exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = y_train = X_test = y_test = None\n",
    "\n",
    "#######################################################################\n",
    "# TODO:                                                               #\n",
    "# Arrange the data into train and test sets                           #\n",
    "# Be careful about how you split the data:                            #\n",
    "# - If train and test distribution are very different your test       #\n",
    "#   performance will be poor                                          #\n",
    "# - Think about the sizes of the splits: What are good values and how #\n",
    "#   does this affect your train/test performance?                     #\n",
    "#######################################################################\n",
    "\n",
    "#Splits the code into 30% test set and 70% training set.\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,  random_state=0) \n",
    "\n",
    "#######################################################################\n",
    "#                         END OF YOUR CODE                            #\n",
    "#######################################################################\n",
    "\n",
    "\n",
    "def pre_process(x):\n",
    "    #######################################################################\n",
    "    # TODO:                                                               #\n",
    "    # Implement preprocessing of the data before feeding to the SVM.      #\n",
    "    # NOTE: This function will be used to grade the performance on the    #\n",
    "    # held-out test set                                                   #\n",
    "    #######################################################################\n",
    "\n",
    "    x = preprocessing.scale(x)\n",
    "    #scaler = StandardScaler().fit(x)\n",
    "    #x = scaler.transform(x)\n",
    "    #scaler.transform(x)\n",
    "    \n",
    "    #######################################################################\n",
    "    #                         END OF YOUR CODE                            #\n",
    "    #######################################################################\n",
    "    return x\n",
    "\n",
    "X_train = pre_process(X_train)\n",
    "X_test  = pre_process(X_test)\n",
    "\n",
    "print('Training data shape: ', X_train.shape)\n",
    "print('Training labels shape: ', y_train.shape)\n",
    "print('Test data shape: ', X_test.shape)\n",
    "print('Test labels shape: ', y_test.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5: Training and evaluating a linear SMV [10 Points]\n",
    "\n",
    "Now its time to train a linear SVM on your training data and evaluate its performance on the test set.\n",
    "\n",
    "You should use [sklearn.svm.LinearSVC](http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC) for training the model. See the documentation for usage, arguments, return values and methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def train_linear_SVM(X, y, C, max_iter=100):\n",
    "    \"\"\"\n",
    "    Linear multi-class SVM solver.\n",
    "\n",
    "    Args:\n",
    "        X: Data matrix of shape [num_train, num_features]\n",
    "        y: Labels corresponding to X of size [num_train]\n",
    "        C: Hyper-parameter for SVM\n",
    "        max_iter: Maximum number of iterations\n",
    "\n",
    "    Returns:\n",
    "        lin_clf: The learnt classifier (LinearSVC instance)\n",
    "\n",
    "    \"\"\"\n",
    "    lin_clf = None\n",
    "    print('Solving linear-SVM...')\n",
    "\n",
    "    #######################################################################\n",
    "    # TODO:                                                               #\n",
    "    # Train the SVM using LinearSVC and return the learnt classifier      #\n",
    "    #######################################################################\n",
    "    lin_clf = LinearSVC(random_state=0, C=1.0, max_iter=100)\n",
    "    lin_clf.fit(X, y)\n",
    "\n",
    "\n",
    "    #######################################################################\n",
    "    #                         END OF YOUR CODE                            #\n",
    "    #######################################################################\n",
    "    return lin_clf\n",
    "\n",
    "C = 1.0\n",
    "lin_clf = train_linear_SVM(X_train, y_train, C)\n",
    "\n",
    "#######################################################################\n",
    "# TODO:                                                               #\n",
    "# Visualize the learnt weights (lin_clf.coef_) for all the classes:   #\n",
    "# - Make a plot with ten figures showing the respective weights for   #\n",
    "#   each of the classes                                               #\n",
    "#######################################################################\n",
    "for j, cls in enumerate(classes):\n",
    "    idxs = np.flatnonzero(np.equal(labels, cls))\n",
    "    idxs = np.random.choice(idxs, samples_per_class, replace=False)\n",
    "    for i, idx in enumerate(idxs):\n",
    "        plt_idx = j * samples_per_class + i + 1\n",
    "        plt.subplot(samples_per_class, len(classes), plt_idx)\n",
    "        plt.imshow(imgs[idx])\n",
    "        plt.axis('off')\n",
    "        plt.title(cls)\n",
    "plt.show()\n",
    "\n",
    "#######################################################################\n",
    "#                         END OF YOUR CODE                            #\n",
    "#######################################################################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 7: Evaluating a multi-class classifier [10 Points]\n",
    "\n",
    "You should now evaluate you learnt classifier. For this you have to:\n",
    "* Compute predictions on the train and test sets\n",
    "* Compute the accuracy of the predictions\n",
    "* Compute the confusion matrix (see [here](https://en.wikipedia.org/wiki/Confusion_matrix) for info)\n",
    "* Answer the question below\n",
    "\n",
    "**Note:** You can make use of sklearn.metrics\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "\n",
    "def eval_clf(y_pred_train, y_pred_test):\n",
    "    acc_test = acc_train = cm_test = None\n",
    "    #######################################################################\n",
    "    # TODO:                                                               #\n",
    "    # Use the learnt classifier to make predictions on the test set.(done)#\n",
    "    # Compute the accuracy on train and test sets.(done)                  #\n",
    "    # Compute the confusion matrix on the test set.(done)                 #\n",
    "    #######################################################################    \n",
    "    _test = train_linear_SVM(X_test, y_test, C = 1)\n",
    "    preds_test = _test.predict(X_test)\n",
    "    \n",
    "    acc_train = accuracy_score(y_train, y_pred_train)\n",
    "    acc_test = accuracy_score(y_test, preds_test)\n",
    "    \n",
    "    #cm_train =\n",
    "    cm_test = confusion_matrix(y_test, preds_test)\n",
    "    \n",
    "    \n",
    "    #######################################################################\n",
    "    #                         END OF YOUR CODE                            #\n",
    "    #######################################################################\n",
    "    return acc_train, acc_test, cm_test\n",
    "\n",
    "acc_train, acc_test, cm_test = eval_clf(lin_clf.predict(X_train), lin_clf.predict(X_test))\n",
    "print(\"Linear SVM accuracy train: {}\".format(acc_train))\n",
    "print(\"Linear SVM accuracy test: {}\".format(acc_test))\n",
    "print(\"Confusion matrix:\\n%s\" % cm_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question: \n",
    "* Which pair of digits does the classifier confuse most often?\n",
    "\n",
    "***Your Answer:***\n",
    "\n",
    "The pair of digits that most misclassified are: 8 and 5 with the classify confusing 92 samples in the test set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 7: Training and evaluating a SMV with Gaussian kernel [10 Points]\n",
    "\n",
    "Now you will train a SVM with Gaussian kernel (also called RBF kernel).\n",
    "\n",
    "You should use [sklearn.svm.SVC](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) for training the model. See the documentation for usage, arguments and return values.\n",
    "Experiment with values for the hyper-parameters $C$ and $\\gamma$ and answer the question below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "def train_gaussian_SVM(X, y, C, gamma, max_iter=100):\n",
    "    \"\"\"\n",
    "    Multi-class SVM solver with Gaussian kernel.\n",
    "\n",
    "    Args:\n",
    "        X: Data matrix of shape [num_train, num_features]\n",
    "        y: Labels corresponding to X of size [num_train]\n",
    "        C: Hyper-parameter for SVM\n",
    "        max_iter: Maximum number of iterations\n",
    "\n",
    "    Returns:\n",
    "        w: The value of the parameters after logistic regression\n",
    "\n",
    "    \"\"\"\n",
    "    clf_rbf = None\n",
    "    print('Solving RBF-SVM: This can take a while...')\n",
    "    \n",
    "    #######################################################################\n",
    "    # TODO:                                                               #\n",
    "    # Train the SVM using LinearSVC and return the learnt classifier      #\n",
    "    #######################################################################\n",
    "    clf_rbf = SVC(random_state=0, C=C, max_iter=max_iter, kernel='rbf', gamma=gamma)\n",
    "    clf_rbf.fit(X, y)\n",
    "\n",
    "\n",
    "    #######################################################################\n",
    "    #                         END OF YOUR CODE                            #\n",
    "    #######################################################################\n",
    "    return clf_rbf\n",
    "\n",
    "C = 1.5\n",
    "gamma = 0.01\n",
    "\n",
    "clf_rbf = train_gaussian_SVM(X_train, y_train, C, gamma)\n",
    "\n",
    "acc_train, acc_test, cm_test = eval_clf(clf_rbf.predict(X_train), clf_rbf.predict(X_test))\n",
    "print(\"RBF SVM accuracy train: {}\".format(acc_train))\n",
    "print(\"RBF SVM accuracy test: {}\".format(acc_test))\n",
    "print(\"Confusion matrix:\\n%s\" % cm_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question:\n",
    "* How do the hyper-parameters influence the classifier? What happens for extreme values of the hyper-parameters?\n",
    "\n",
    "***Your Answer:***\n",
    "\n",
    "The ***gamma*** parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close'. The parameter can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.\n",
    "\n",
    "The ***C*** parameter trades off misclassification of training examples against simplicity of the decision surface. A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly by giving the model freedom to select more samples as support vectors.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8: Questions [10 Points]\n",
    "\n",
    "Answer the following questions and justify your answers:\n",
    "\n",
    "* How does your implementation of the linear SVM (Pegasos) compare to the classifiers of the previous assignment? Which method is the most accurate? Which one the fastest to train? Which one is the most versatile?\n",
    "\n",
    "\t***Your Answer:***\n",
    "    \n",
    "\n",
    "* Linear SVM vs. Gaussian Kernel SVM: Give advantages and disadvantages of both approaches. \n",
    "\n",
    "\t***Your Answer:***\n",
    "    \n",
    "    \n",
    "* Linear SVM vs. Gaussian Kernel SVM: In what setting would you pick one method over the other? Answer in terms of number of training examples $m_{train}$ and feature dimension $d$\n",
    "\n",
    "\t***Your Answer:***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus: Tune those Hyper-Parameters! [15 Points]\n",
    "\n",
    "Perform hyper-parameter tuning for the two multi-class SVMs. Your performance will be measured on a held-out test set and the **10% best scoring submissions get 5 bonus points** (5 points for linear-SVM and 5 points for RBF-SVM). \n",
    "\n",
    "Set the best parameter values above. The performance will be **tested with max_iter=100**!\n",
    "\n",
    "Additional 5 points can be gained by demonstrating a principled approach for hyper-parameter search below. For ideas refer to:\n",
    "http://scikit-learn.org/stable/modules/grid_search.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your optional hyper-parameter tuning code goes here...\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn import svm\n",
    "\n",
    "param_grid = [\n",
    "  {'C': [1, 10], 'kernel': ['linear']},\n",
    "  {'C': [1, 10], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},\n",
    " ]\n",
    "\n",
    "#RBF Kernel Hyper-Parameter tuning\n",
    "#svc = svm.SVC() -- Not needed, classifier already defined.\n",
    "    \n",
    "GridSearch = GridSearchCV(clf_rbf, param_grid)\n",
    "GridSearch.fit(X_test, y_test)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}