3. Scikit Learn - Parameter Search via Randomized Parameter Search.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:56dd5c49a6f7cf7d1f81f7eb1ebb8a8f4a8c6d6054c8b6d6a6e6114b405112ed"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Randomized Parameter Search"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you cannot do a parameter search because the permutation of a number of parameters grow _really_ fast. Then, you need to be _smarter_ about grid search. You need to decrease your parameter search space without (hopefully) giving up too much performance. The performance of the classifier may not match to the grid search, and it is quite okay. Do not forget the loss function that we are optimizing for\n",
      "is not the loss function that will decide how successful the classifier or regressor is (this is the test loss and we do not have chance to look at it). So, even second or third best parameter set for a classifier or pipeline(we will come to this in the next notebook) may work pretty well in the real settings. That is to say, sub-optimal is acceptable if we are not losing too much with the trade-off gaining quite a bit performance. \n",
      "Luckily, RandomizedParameterSearch in scikit-learn does the hard work for us and without losing too much performance, we have similar results to the GridSearch in a much more efficient and faster way."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline \n",
      "\n",
      "import matplotlib as mlp\n",
      "import matplotlib.pyplot as plt\n",
      "import numpy as np\n",
      "from operator import itemgetter\n",
      "import os\n",
      "import pandas as pd\n",
      "from scipy.stats import randint as sp_randint\n",
      "import sklearn\n",
      "from sklearn import cross_validation\n",
      "from sklearn import datasets\n",
      "from sklearn import ensemble\n",
      "from sklearn import grid_search\n",
      "from sklearn import metrics\n",
      "from sklearn import preprocessing\n",
      "import time\n",
      "\n",
      "plt.style.use('fivethirtyeight')\n",
      "\n",
      "_DATA_DIR = 'data'\n",
      "_DATA_PATH = os.path.join(_DATA_DIR, 'titanic.csv')\n",
      "\n",
      "# To encode categorical variables\n",
      "label_encoder = preprocessing.LabelEncoder()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df = pd.read_csv(_DATA_PATH)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Preprocessing Variables"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "del df['row.names']\n",
      "df['pclass'] = label_encoder.fit_transform(df.pclass)\n",
      "df['embarked'] = label_encoder.fit_transform(df.embarked)\n",
      "df['sex'] = label_encoder.fit_transform(df.sex)\n",
      "def convert_age(number):\n",
      "    try:\n",
      "        number = int(number)\n",
      "    except ValueError:\n",
      "        number = 0\n",
      "    return number\n",
      "\n",
      "def extract_home_destination(address):\n",
      "    try:\n",
      "        address = address.split(',')[-1]\n",
      "    except AttributeError:\n",
      "        address = ''\n",
      "    return address\n",
      "\n",
      "df.age = df.age.apply(convert_age)\n",
      "# Preprocess first\n",
      "df['home.dest'] = df['home.dest'].apply(extract_home_destination)\n",
      "df['destination'] = label_encoder.fit_transform(df['home.dest'])\n",
      "del df['home.dest']\n",
      "del df['boat']\n",
      "del df['ticket']\n",
      "del df['name']\n",
      "del df['room']\n",
      "y = np.array(df.survived.tolist())\n",
      "del df['survived']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "feature_names = ['pclass', 'age', 'embarked', 'sex', 'destination']\n",
      "X = df[feature_names].as_matrix()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.2, random_state=0)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "param_grid = {\n",
      "              'learning_rate': [0.1, 0.05, 0.01],\n",
      "              'max_depth': [4, 6],\n",
      "              'min_samples_leaf': [1, 2, 3, 4, 9, 15],\n",
      "              'n_estimators': [1000, 2000, 3000],\n",
      "              }\n",
      "\n",
      "est = ensemble.GradientBoostingClassifier()\n",
      "\n",
      "start_time = time.time()\n",
      "# run randomized search\n",
      "n_iter_search = 20\n",
      "randomized_search = grid_search.RandomizedSearchCV(est, param_distributions=param_grid,\n",
      "    n_iter=n_iter_search, n_jobs=4).fit(X_train, y_train)\n",
      "\n",
      "gs_cv = grid_search.GridSearchCV(est, param_grid, n_jobs=4).fit(X_train, y_train)\n",
      "end_time = time.time()\n",
      "\n",
      "print('It took {} seconds'.format(end_time - start_time))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "It took 995.277484894 seconds\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# best hyperparameter setting\n",
      "randomized_search.best_params_"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "{'learning_rate': 0.01,\n",
        " 'max_depth': 4,\n",
        " 'min_samples_leaf': 1,\n",
        " 'n_estimators': 2000}"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "randomized_search.best_score_"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "0.80476190476190479"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "randomized_search.grid_scores_"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "[mean: 0.77905, std: 0.01553, params: {'n_estimators': 3000, 'learning_rate': 0.1, 'max_depth': 6, 'min_samples_leaf': 3},\n",
        " mean: 0.78000, std: 0.02100, params: {'n_estimators': 3000, 'learning_rate': 0.1, 'max_depth': 4, 'min_samples_leaf': 9},\n",
        " mean: 0.78286, std: 0.00841, params: {'n_estimators': 3000, 'learning_rate': 0.05, 'max_depth': 4, 'min_samples_leaf': 2},\n",
        " mean: 0.80286, std: 0.00233, params: {'n_estimators': 1000, 'learning_rate': 0.01, 'max_depth': 4, 'min_samples_leaf': 3},\n",
        " mean: 0.79238, std: 0.00943, params: {'n_estimators': 1000, 'learning_rate': 0.05, 'max_depth': 6, 'min_samples_leaf': 2},\n",
        " mean: 0.78571, std: 0.00404, params: {'n_estimators': 3000, 'learning_rate': 0.05, 'max_depth': 6, 'min_samples_leaf': 2},\n",
        " mean: 0.77714, std: 0.01234, params: {'n_estimators': 3000, 'learning_rate': 0.1, 'max_depth': 4, 'min_samples_leaf': 2},\n",
        " mean: 0.80286, std: 0.00467, params: {'n_estimators': 1000, 'learning_rate': 0.01, 'max_depth': 4, 'min_samples_leaf': 1},\n",
        " mean: 0.78476, std: 0.01720, params: {'n_estimators': 2000, 'learning_rate': 0.1, 'max_depth': 4, 'min_samples_leaf': 9},\n",
        " mean: 0.77238, std: 0.00883, params: {'n_estimators': 2000, 'learning_rate': 0.1, 'max_depth': 4, 'min_samples_leaf': 3},\n",
        " mean: 0.78381, std: 0.00943, params: {'n_estimators': 2000, 'learning_rate': 0.05, 'max_depth': 6, 'min_samples_leaf': 4},\n",
        " mean: 0.78190, std: 0.01655, params: {'n_estimators': 1000, 'learning_rate': 0.1, 'max_depth': 4, 'min_samples_leaf': 9},\n",
        " mean: 0.78762, std: 0.01197, params: {'n_estimators': 1000, 'learning_rate': 0.05, 'max_depth': 4, 'min_samples_leaf': 4},\n",
        " mean: 0.80381, std: 0.00943, params: {'n_estimators': 1000, 'learning_rate': 0.01, 'max_depth': 4, 'min_samples_leaf': 9},\n",
        " mean: 0.79619, std: 0.00713, params: {'n_estimators': 1000, 'learning_rate': 0.01, 'max_depth': 6, 'min_samples_leaf': 3},\n",
        " mean: 0.80286, std: 0.01234, params: {'n_estimators': 2000, 'learning_rate': 0.01, 'max_depth': 4, 'min_samples_leaf': 3},\n",
        " mean: 0.79143, std: 0.01400, params: {'n_estimators': 2000, 'learning_rate': 0.01, 'max_depth': 4, 'min_samples_leaf': 9},\n",
        " mean: 0.77810, std: 0.02117, params: {'n_estimators': 1000, 'learning_rate': 0.05, 'max_depth': 4, 'min_samples_leaf': 9},\n",
        " mean: 0.78762, std: 0.01984, params: {'n_estimators': 1000, 'learning_rate': 0.1, 'max_depth': 6, 'min_samples_leaf': 3},\n",
        " mean: 0.80476, std: 0.01197, params: {'n_estimators': 2000, 'learning_rate': 0.01, 'max_depth': 4, 'min_samples_leaf': 1}]"
       ]
      }
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Comparison between Grid Search and Randomized Parameter Search"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf = ensemble.RandomForestClassifier()\n",
      "\n",
      "def report(grid_scores, n_top=3):\n",
      "    top_scores = sorted(grid_scores, key=itemgetter(1), reverse=True)[:n_top]\n",
      "    for ii, score in enumerate(top_scores):\n",
      "        print(\"Model with rank: {}\".format(ii + 1))\n",
      "        print(\"Mean validation score: {0:.3f} (std: {1:.3f})\".format(\n",
      "              score.mean_validation_score,\n",
      "              np.std(score.cv_validation_scores)))\n",
      "        print(\"Parameters: {}\\n\".format(score.parameters))\n",
      "\n",
      "param_grid = {\"max_depth\": [3, 2, 1, None],\n",
      "              \"max_features\": [1, 2, 3, 4],\n",
      "              \"min_samples_split\": [1, 2, 3, 4],\n",
      "              \"min_samples_leaf\": [1, 2, 3, 4],\n",
      "              \"bootstrap\": [True, False],\n",
      "              \"criterion\": [\"gini\", \"entropy\"],\n",
      "              'n_estimators': [50, 100, 1000, 2000, 3000],\n",
      "}\n",
      "\n",
      "n_iter_search = 20\n",
      "random_search = grid_search.RandomizedSearchCV(clf, param_distributions=param_grid,\n",
      "                                   n_iter=n_iter_search)\n",
      "\n",
      "start = time.time()\n",
      "random_search.fit(X, y)\n",
      "end = time.time()\n",
      "print(\"RandomizedSearchCV took {0:.2f} seconds for {1} candidates\"\n",
      "      \" parameter settings.\".format((end - start), n_iter_search))\n",
      "report(random_search.grid_scores_)s\n",
      "\n",
      "\n",
      "param_grid = {\"max_depth\": [3, 2, 1, None],\n",
      "              \"max_features\": [1, 2, 3, 4],\n",
      "              \"min_samples_split\": [1, 2, 3, 4],\n",
      "              \"min_samples_leaf\": [1, 2, 3, 4],\n",
      "              \"bootstrap\": [True, False],\n",
      "              \"criterion\": [\"gini\", \"entropy\"],\n",
      "              'n_estimators': [50, 100, 1000, 2000, 3000],\n",
      "}\n",
      "\n",
      "gs = grid_search.GridSearchCV(clf, param_grid=param_grid)\n",
      "start = time.time()\n",
      "gs.fit(X, y)\n",
      "end = time.time()\n",
      "\n",
      "print(\"GridSearchCV took {0:.2f} seconds for {1} candidate parameter settings.\".format(\n",
      "      (end - start, len(gs.grid_scores_))))\n",
      "report(gs.grid_scores_)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}