Ad_click_prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ad Click Prediction\n",
    "\n",
    "\n",
    "## Part 2: Advertisement click prediction.\n",
    "\n",
    "The aim of this project is prediction of the advertisement click using the machine learning models. Based on the historical data of advertisement clicks (user behaviour, user profile, etc.) I made a model to predict who is going to click ad on a website in the future\n",
    "\n",
    "The internet marketing is an important part of marketing strategies. The companies prefer to advertise their products on websites and social media platforms. It is important to targeting the right audience because it spending millions to display the advertisements to the audience who is not want to buy ours products can be costly.\n",
    "The Ad click is important because it allows to determine whether spending their money on digital advertising is worth or not. Nowadays, different types of advertisers and search engines rely on modeling to predict ad CTR (i.e. click-through rate) accurately.\n",
    "\n",
    "**Dataset**\n",
    "\n",
    "The dataset comes from Kaggle: https://www.kaggle.com/datasets/arashnic/ctr-in-advertisement\n",
    "\n",
    "Attributes:\n",
    "\n",
    "- session_id - unique session id \n",
    "- DateTime - date time of log \n",
    "- user_id - unique user id \n",
    "- product - unique product\n",
    "- campaign_id - unique campaign id\n",
    "- webpage_id - unique webpage id \n",
    "- product_category_1 - unique product 1\n",
    "- product_category_2 - unique product 2\n",
    "- user_group_id - id of user group\n",
    "- gender - gender of user\n",
    "- age_level - age level of user\n",
    "- user_depth \n",
    "- city_development_index \n",
    "- var_1 - unknown variable\n",
    "- is_click - target\n",
    "\n",
    "**Overview:**\n",
    "\n",
    "The whole project is divided into 7 steps:\n",
    "\n",
    "- Importing libraries and loading data set\n",
    "- Basic data analysis\n",
    "- Data cleaning  and preparation\n",
    "- Spliting data into train and test set\n",
    "- Building and training the model\n",
    "- Testing the model accuracy\n",
    "- Predictions on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from imblearn.over_sampling import SMOTE\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "from imblearn.pipeline import Pipeline as imbpipeline\n",
    "\n",
    "# Model packages\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score, classification_report\n",
    "\n",
    "import pickle\n",
    "import warnings\n",
    "warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libriaries and data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>session_id</th>\n",
       "      <th>DateTime</th>\n",
       "      <th>user_id</th>\n",
       "      <th>product</th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>webpage_id</th>\n",
       "      <th>product_category_1</th>\n",
       "      <th>product_category_2</th>\n",
       "      <th>user_group_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_level</th>\n",
       "      <th>user_depth</th>\n",
       "      <th>city_development_index</th>\n",
       "      <th>var_1</th>\n",
       "      <th>is_click</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>140690</td>\n",
       "      <td>2017-07-02 00:00</td>\n",
       "      <td>858557</td>\n",
       "      <td>C</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>333291</td>\n",
       "      <td>2017-07-02 00:00</td>\n",
       "      <td>243253</td>\n",
       "      <td>C</td>\n",
       "      <td>105960</td>\n",
       "      <td>11085</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>129781</td>\n",
       "      <td>2017-07-02 00:00</td>\n",
       "      <td>243253</td>\n",
       "      <td>C</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>464848</td>\n",
       "      <td>2017-07-02 00:00</td>\n",
       "      <td>1097446</td>\n",
       "      <td>I</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>90569</td>\n",
       "      <td>2017-07-02 00:01</td>\n",
       "      <td>663656</td>\n",
       "      <td>C</td>\n",
       "      <td>405490</td>\n",
       "      <td>60305</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   session_id          DateTime  user_id product  campaign_id  webpage_id  \\\n",
       "0      140690  2017-07-02 00:00   858557       C       359520       13787   \n",
       "1      333291  2017-07-02 00:00   243253       C       105960       11085   \n",
       "2      129781  2017-07-02 00:00   243253       C       359520       13787   \n",
       "3      464848  2017-07-02 00:00  1097446       I       359520       13787   \n",
       "4       90569  2017-07-02 00:01   663656       C       405490       60305   \n",
       "\n",
       "   product_category_1  product_category_2  user_group_id  gender  age_level  \\\n",
       "0                   4                 NaN           10.0  Female        4.0   \n",
       "1                   5                 NaN            8.0  Female        2.0   \n",
       "2                   4                 NaN            8.0  Female        2.0   \n",
       "3                   3                 NaN            3.0    Male        3.0   \n",
       "4                   3                 NaN            2.0    Male        2.0   \n",
       "\n",
       "   user_depth  city_development_index  var_1  is_click  \n",
       "0         3.0                     3.0      0         0  \n",
       "1         2.0                     NaN      0         0  \n",
       "2         2.0                     NaN      0         0  \n",
       "3         3.0                     2.0      1         0  \n",
       "4         3.0                     2.0      1         0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(r'C:\\Python Scripts\\Datasets\\CTR advertisement\\Ad_click_prediction_train.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(463291, 15)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  First observations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 463291 entries, 0 to 463290\n",
      "Data columns (total 15 columns):\n",
      " #   Column                  Non-Null Count   Dtype  \n",
      "---  ------                  --------------   -----  \n",
      " 0   session_id              463291 non-null  int64  \n",
      " 1   DateTime                463291 non-null  object \n",
      " 2   user_id                 463291 non-null  int64  \n",
      " 3   product                 463291 non-null  object \n",
      " 4   campaign_id             463291 non-null  int64  \n",
      " 5   webpage_id              463291 non-null  int64  \n",
      " 6   product_category_1      463291 non-null  int64  \n",
      " 7   product_category_2      97437 non-null   float64\n",
      " 8   user_group_id           445048 non-null  float64\n",
      " 9   gender                  445048 non-null  object \n",
      " 10  age_level               445048 non-null  float64\n",
      " 11  user_depth              445048 non-null  float64\n",
      " 12  city_development_index  338162 non-null  float64\n",
      " 13  var_1                   463291 non-null  int64  \n",
      " 14  is_click                463291 non-null  int64  \n",
      "dtypes: float64(5), int64(7), object(3)\n",
      "memory usage: 53.0+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['session_id', 'DateTime', 'user_id', 'product', 'campaign_id',\n",
      "       'webpage_id', 'product_category_1', 'product_category_2',\n",
      "       'user_group_id', 'gender', 'age_level', 'user_depth',\n",
      "       'city_development_index', 'var_1', 'is_click'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking unique values in each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of unique values in session_id: 463291\n",
      "Number of unique values in DateTime: 8610\n",
      "Number of unique values in user_id: 150347\n",
      "Number of unique values in product: 10\n",
      "Number of unique values in campaign_id: 10\n",
      "Number of unique values in webpage_id: 9\n",
      "Number of unique values in product_category_1: 5\n",
      "Number of unique values in product_category_2: 29\n",
      "Number of unique values in user_group_id: 13\n",
      "Number of unique values in gender: 2\n",
      "Number of unique values in age_level: 7\n",
      "Number of unique values in user_depth: 3\n",
      "Number of unique values in city_development_index: 4\n",
      "Number of unique values in var_1: 2\n",
      "Number of unique values in is_click: 2\n"
     ]
    }
   ],
   "source": [
    "for column in df:\n",
    "    print(f'Number of unique values in {column}:', df[column].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking the missing values in data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "session_id                     0\n",
       "DateTime                       0\n",
       "user_id                        0\n",
       "product                        0\n",
       "campaign_id                    0\n",
       "webpage_id                     0\n",
       "product_category_1             0\n",
       "product_category_2        365854\n",
       "user_group_id              18243\n",
       "gender                     18243\n",
       "age_level                  18243\n",
       "user_depth                 18243\n",
       "city_development_index    125129\n",
       "var_1                          0\n",
       "is_click                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem statement\n",
    "\n",
    "1. Defining the problem statement:\n",
    "\n",
    "The aim of this is to create a predict model which can shows whether an user will click on a given product advertisement or not?\n",
    "\n",
    "- **Target Variable**: is click\n",
    "- **Predictors**: DateTime, product, age_level, gender, campaign_id, webpage_id, user_depth... etc.\n",
    " \n",
    "- is click = 1 means the ad was clicked.\n",
    "- is click = 0 means the ad was not clicked.\n",
    "\n",
    "\n",
    "2. Determining the type of Machine Learning\n",
    "\n",
    "Based on the problem statement I will create a supervised classification model because the target variable is categorical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic data analysis\n",
    "\n",
    "In the data ad click is marked as 'is click' where 1 = is click and 0 = no click. \n",
    "\n",
    "#### Ad click analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    431960\n",
       "1     31331\n",
       "Name: is_click, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['is_click'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEHCAYAAABiAAtOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAQnElEQVR4nO3dbaxlVX3H8e+PGR40FUEYlc4Qh+qkcTStD1OYYGoJNDBodYiKgVaZWpKpBqvGNhX7Bqul0RctFaTYsYw8xIjEJ6YGSyhqSQMCF0EeQ5hCK1MoMziAWANm6L8v7hp7vJx75ziuc+7Mne8nOTl7//fae62T3OR3997r7JOqQpKknvab7wFIkhYew0WS1J3hIknqznCRJHVnuEiSuls83wPYUxx++OG1fPny+R6GJO1Vbr311seqasnMuuHSLF++nKmpqfkehiTtVZL857C6l8UkSd0ZLpKk7gwXSVJ3hoskqTvDRZLUneEiSerOcJEkdWe4SJK6M1wkSd35Df2Opj7w3vkegvYwq87/7HwPQZoXnrlIkrozXCRJ3RkukqTuDBdJUneGiySpO8NFktSd4SJJ6s5wkSR1Z7hIkrozXCRJ3RkukqTuxh4uSRYluS3JN9r6UUluSnJ/ki8lOaDVD2zrm9v25QPH+Gir35fkpIH6mlbbnOTsgfrQPiRJkzGJM5cPAvcOrH8KOK+qVgCPA2e2+pnA41X1CuC81o4kK4HTgFcBa4C/b4G1CLgQOBlYCZze2s7VhyRpAsYaLkmWAW8G/rGtBzge+HJrcilwSlte29Zp209o7dcCV1TVM1X1ILAZOLq9NlfVA1X1U+AKYO0u+pAkTcC4z1z+Dvhz4H/b+mHAE1W1o61vAZa25aXAQwBt+5Ot/c/qM/aZrT5XHz8nyfokU0mmtm3btrufUZI0w9jCJcnvAVur6tbB8pCmtYttverPLVZtqKpVVbVqyZIlw5pIknbDOH8s7A3AW5O8CTgIOJjpM5lDkixuZxbLgIdb+y3AkcCWJIuBFwLbB+o7De4zrP7YHH1IkiZgbGcuVfXRqlpWVcuZviH/rar6A+DbwDtas3XAVW15U1unbf9WVVWrn9Zmkx0FrABuBm4BVrSZYQe0Pja1fWbrQ5I0AfPxPZePAB9Ospnp+yMXt/rFwGGt/mHgbICquhu4ErgH+GfgrKp6tp2VvB+4hunZaFe2tnP1IUmagHFeFvuZqvoO8J22/ADTM71mtnkaOHWW/c8Fzh1Svxq4ekh9aB+SpMnwG/qSpO4MF0lSd4aLJKk7w0WS1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1J3hIknqznCRJHVnuEiSuhtbuCQ5KMnNSb6f5O4kf9nqRyW5Kcn9Sb6U5IBWP7Ctb27blw8c66Otfl+Skwbqa1ptc5KzB+pD+5AkTcY4z1yeAY6vqt8EXgOsSbIa+BRwXlWtAB4HzmztzwQer6pXAOe1diRZCZwGvApYA/x9kkVJFgEXAicDK4HTW1vm6EOSNAFjC5ea9uO2un97FXA88OVWvxQ4pS2vbeu07SckSatfUVXPVNWDwGbg6PbaXFUPVNVPgSuAtW2f2fqQJE3AWO+5tDOM24GtwLXAvwNPVNWO1mQLsLQtLwUeAmjbnwQOG6zP2Ge2+mFz9DFzfOuTTCWZ2rZt2y/zUSVJA8YaLlX1bFW9BljG9JnGK4c1a++ZZVuv+rDxbaiqVVW1asmSJcOaSJJ2w0Rmi1XVE8B3gNXAIUkWt03LgIfb8hbgSIC2/YXA9sH6jH1mqz82Rx+SpAkY52yxJUkOacvPA34XuBf4NvCO1mwdcFVb3tTWadu/VVXV6qe12WRHASuAm4FbgBVtZtgBTN/039T2ma0PSdIELN51k912BHBpm9W1H3BlVX0jyT3AFUn+CrgNuLi1vxi4PMlmps9YTgOoqruTXAncA+wAzqqqZwGSvB+4BlgEbKyqu9uxPjJLH5KkCRhbuFTVHcBrh9QfYPr+y8z608CpsxzrXODcIfWrgatH7UOSNBl+Q1+S1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1J3hIknqbqRwSXLdKDVJkgAWz7UxyUHA84HDkxwKpG06GPjVMY9NkrSXmjNcgD8GPsR0kNzK/4fLj4ALxzguSdJebM5wqapPA59O8idVdcGExiRJ2svt6swFgKq6IMmxwPLBfarqsjGNS5K0FxspXJJcDrwcuB14tpULMFwkSc8xUrgAq4CVVVXjHIwkaWEY9XsudwEvHedAJEkLx6hnLocD9yS5GXhmZ7Gq3jqWUUmS9mqjhsvHxjkISdLCMupssX8d90AkSQvHqLPFnmJ6dhjAAcD+wP9U1cHjGpgkae816pnLCwbXk5wCHD2WEUmS9nq79VTkqvo6cHznsUiSFohRL4u9bWB1P6a/9+J3XiRJQ406W+wtA8s7gP8A1nYfjSRpQRj1nst7xj0QSdLCMeqPhS1L8rUkW5M8muQrSZaNe3CSpL3TqDf0Pw9sYvp3XZYC/9RqkiQ9x6jhsqSqPl9VO9rrEmDJGMclSdqLjRoujyV5V5JF7fUu4IfjHJgkae81arj8EfBO4L+BR4B3AHPe5E9yZJJvJ7k3yd1JPtjqL0pybZL72/uhrZ4k5yfZnOSOJK8bONa61v7+JOsG6q9Pcmfb5/wkmasPSdJkjBounwDWVdWSqnox02HzsV3sswP406p6JbAaOCvJSuBs4LqqWgFc19YBTgZWtNd64CKYDgrgHOAYpp8KcM5AWFzU2u7cb02rz9aHJGkCRg2X36iqx3euVNV24LVz7VBVj1TV99ryU8C9TE8GWAtc2ppdCpzSltcCl9W07wKHJDkCOAm4tqq2tzFcC6xp2w6uqhvbj5hdNuNYw/qQJE3AqOGy3+ClpXY2MeoXMEmynOkwugl4SVU9AtMBBLy4NVsKPDSw25ZWm6u+ZUidOfqYOa71SaaSTG3btm3UjyNJ2oVRA+JvgBuSfJnpx768Ezh3lB2T/ArwFeBDVfWjdltkaNMhtdqN+siqagOwAWDVqlU+zkaSOhnpzKWqLgPeDjwKbAPeVlWX72q/JPszHSxfqKqvtvKj7ZIW7X1rq28BjhzYfRnw8C7qy4bU5+pDkjQBIz8VuaruqarPVNUFVXXPrtq3mVsXA/dW1d8ObNoE7JzxtQ64aqB+Rps1thp4sl3SugY4Mcmh7dLcicA1bdtTSVa3vs6YcaxhfUiSJmDk+ya74Q3Au4E7k9zean8BfBK4MsmZwA+AU9u2q4E3AZuBn9CmOlfV9iSfAG5p7T7eJhQAvA+4BHge8M32Yo4+JEkTMLZwqap/Y/h9EYAThrQv4KxZjrUR2DikPgW8ekj9h8P6kCRNxm79WJgkSXMxXCRJ3RkukqTuDBdJUneGiySpO8NFktSd4SJJ6s5wkSR1Z7hIkrozXCRJ3RkukqTuDBdJUneGiySpO8NFktSd4SJJ6s5wkSR1Z7hIkrozXCRJ3RkukqTuDBdJUneGiySpO8NFktSd4SJJ6s5wkSR1Z7hIkrozXCRJ3RkukqTuDBdJUneGiySpO8NFktSd4SJJ6s5wkSR1Z7hIkrozXCRJ3RkukqTuDBdJUneGiySpO8NFktSd4SJJ6m5s4ZJkY5KtSe4aqL0oybVJ7m/vh7Z6kpyfZHOSO5K8bmCfda39/UnWDdRfn+TOts/5STJXH5KkyRnnmcslwJoZtbOB66pqBXBdWwc4GVjRXuuBi2A6KIBzgGOAo4FzBsLiotZ2535rdtGHJGlCxhYuVXU9sH1GeS1waVu+FDhloH5ZTfsucEiSI4CTgGurantVPQ5cC6xp2w6uqhurqoDLZhxrWB+SpAmZ9D2Xl1TVIwDt/cWtvhR4aKDdllabq75lSH2uPp4jyfokU0mmtm3bttsfSpL08/aUG/oZUqvdqP9CqmpDVa2qqlVLliz5RXeXJM1i0uHyaLukRXvf2upbgCMH2i0DHt5FfdmQ+lx9SJImZNLhsgnYOeNrHXDVQP2MNmtsNfBku6R1DXBikkPbjfwTgWvatqeSrG6zxM6YcaxhfUiSJmTxuA6c5IvAccDhSbYwPevrk8CVSc4EfgCc2ppfDbwJ2Az8BHgPQFVtT/IJ4JbW7uNVtXOSwPuYnpH2POCb7cUcfUiSJmRs4VJVp8+y6YQhbQs4a5bjbAQ2DqlPAa8eUv/hsD4kSZOzp9zQlyQtIIaLJKk7w0WS1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1J3hIknqznCRJHVnuEiSujNcJEndGS6SpO4MF0lSd4aLJKk7w0WS1N3i+R6ApPF77w1T8z0E7YE+e+yqsR3bMxdJUneGiySpO8NFktSd4SJJ6s5wkSR1Z7hIkrozXCRJ3RkukqTuDBdJUneGiySpO8NFktTdgg2XJGuS3Jdkc5Kz53s8krQvWZDhkmQRcCFwMrASOD3JyvkdlSTtOxZkuABHA5ur6oGq+ilwBbB2nsckSfuMhfrI/aXAQwPrW4BjZjZKsh5Y31Z/nOS+CYxtX3E48Nh8D2LeXfAP8z0CPZd/m02nv86XDSsu1HDJkFo9p1C1Adgw/uHse5JMVdX4fixC2k3+bU7GQr0stgU4cmB9GfDwPI1FkvY5CzVcbgFWJDkqyQHAacCmeR6TJO0zFuRlsarakeT9wDXAImBjVd09z8Pa13i5UXsq/zYnIFXPuRUhSdIvZaFeFpMkzSPDRZLUneGirnzsjvZUSTYm2Zrkrvkey77AcFE3PnZHe7hLgDXzPYh9heGinnzsjvZYVXU9sH2+x7GvMFzU07DH7iydp7FImkeGi3oa6bE7khY+w0U9+dgdSYDhor587I4kwHBRR1W1A9j52J17gSt97I72FEm+CNwI/HqSLUnOnO8xLWQ+/kWS1J1nLpKk7gwXSVJ3hoskqTvDRZLUneEiSerOcJEkdWe4SGOU5IZOx/nDJJ9py+9NcsYcbY9L8o0e/Uq7a/F8D0BayKrq2DEc87O9jyn15pmLNEZJftzej0hyfZLbk9yV5Lfn2GdNku8l+X6S64Zs/1iSP2vLr0jyL63t95K8fEbb30pyW5Jf6/3ZpLl45iJNxu8D11TVue1H1Z4/rFGSJcDngDdW1YNJXrSL434B+GRVfS3JQUz/w3hkO9axwAXA2qr6Qa8PIo3CcJEm4xZgY5L9ga9X1e2ztFsNXF9VDwJU1aw/bpXkBcDSqvpaa/t0qwO8EtgAnFhVPplaE+dlMWkC2q8gvhH4L+DyOW7Ih9F/A2fY7+fs9AjwNPDakQcpdWS4SBOQ5GXA1qr6HHAx8LpZmt4I/E6So9p+s14Wq6ofAVuSnNLaHphk5+W2J4A3A3+d5Lg+n0IaneEiTcZxwO1JbgPeDnx6WKOq2gasB76a5PvAl3Zx3HcDH0hyB3AD8NKBYz0KvAW4MMkxv/QnkH4BPnJfktSdZy6SpO6cLSbNkyQ3AQfOKL+7qu6cj/FIPXlZTJLUnZfFJEndGS6SpO4MF0lSd4aLJKm7/wP9vqo84LOb1gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x='is_click',data=df, palette='hls')\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above analysis showed that data are imbalanced, more user is not ad click than is click. When building models and their evaluation it will need to account for this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculating of click through rate (CTR):**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.932373\n",
       "1    0.067627\n",
       "Name: is_click, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['is_click'].value_counts()/len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From above one can see the overall click through rate (CTR) is around 7% and the 93% ads is not clicked."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data cleaning and preparation\n",
    "\n",
    "In the data there are rows that contain a null result so I delete missing values in gender, age_level, user_group_id, user_depth variables beacause hard to fill them another data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.dropna(subset=['gender','age_level', 'user_group_id', 'user_depth'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "session_id                     0\n",
       "DateTime                       0\n",
       "user_id                        0\n",
       "product                        0\n",
       "campaign_id                    0\n",
       "webpage_id                     0\n",
       "product_category_1             0\n",
       "product_category_2        351379\n",
       "user_group_id                  0\n",
       "gender                         0\n",
       "age_level                      0\n",
       "user_depth                     0\n",
       "city_development_index    106886\n",
       "var_1                          0\n",
       "is_click                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(445048, 15)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Feature engineering for date time features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['DateTime'] = pd.to_datetime(df['DateTime'], errors='coerce')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract hour from DateTime feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['hour'] = df['DateTime'].dt.hour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract Day of week from DateTime feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['day_of_week'] = df['DateTime'].apply(lambda x: x.dayofweek)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>session_id</th>\n",
       "      <th>DateTime</th>\n",
       "      <th>user_id</th>\n",
       "      <th>product</th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>webpage_id</th>\n",
       "      <th>product_category_1</th>\n",
       "      <th>product_category_2</th>\n",
       "      <th>user_group_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_level</th>\n",
       "      <th>user_depth</th>\n",
       "      <th>city_development_index</th>\n",
       "      <th>var_1</th>\n",
       "      <th>is_click</th>\n",
       "      <th>hour</th>\n",
       "      <th>day_of_week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>140690</td>\n",
       "      <td>2017-07-02 00:00:00</td>\n",
       "      <td>858557</td>\n",
       "      <td>C</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>333291</td>\n",
       "      <td>2017-07-02 00:00:00</td>\n",
       "      <td>243253</td>\n",
       "      <td>C</td>\n",
       "      <td>105960</td>\n",
       "      <td>11085</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>129781</td>\n",
       "      <td>2017-07-02 00:00:00</td>\n",
       "      <td>243253</td>\n",
       "      <td>C</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>464848</td>\n",
       "      <td>2017-07-02 00:00:00</td>\n",
       "      <td>1097446</td>\n",
       "      <td>I</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>90569</td>\n",
       "      <td>2017-07-02 00:01:00</td>\n",
       "      <td>663656</td>\n",
       "      <td>C</td>\n",
       "      <td>405490</td>\n",
       "      <td>60305</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   session_id            DateTime  user_id product  campaign_id  webpage_id  \\\n",
       "0      140690 2017-07-02 00:00:00   858557       C       359520       13787   \n",
       "1      333291 2017-07-02 00:00:00   243253       C       105960       11085   \n",
       "2      129781 2017-07-02 00:00:00   243253       C       359520       13787   \n",
       "3      464848 2017-07-02 00:00:00  1097446       I       359520       13787   \n",
       "4       90569 2017-07-02 00:01:00   663656       C       405490       60305   \n",
       "\n",
       "   product_category_1  product_category_2  user_group_id  gender  age_level  \\\n",
       "0                   4                 NaN           10.0  Female        4.0   \n",
       "1                   5                 NaN            8.0  Female        2.0   \n",
       "2                   4                 NaN            8.0  Female        2.0   \n",
       "3                   3                 NaN            3.0    Male        3.0   \n",
       "4                   3                 NaN            2.0    Male        2.0   \n",
       "\n",
       "   user_depth  city_development_index  var_1  is_click  hour  day_of_week  \n",
       "0         3.0                     3.0      0         0     0            6  \n",
       "1         2.0                     NaN      0         0     0            6  \n",
       "2         2.0                     NaN      0         0     0            6  \n",
       "3         3.0                     2.0      1         0     0            6  \n",
       "4         3.0                     2.0      1         0     0            6  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filling missing values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"city_development_index\"] = data[\"city_development_index\"].fillna('0')\n",
    "data[\"product_category_2\"] = data[\"product_category_2\"].fillna(\"0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "session_id                0\n",
       "DateTime                  0\n",
       "user_id                   0\n",
       "product                   0\n",
       "campaign_id               0\n",
       "webpage_id                0\n",
       "product_category_1        0\n",
       "product_category_2        0\n",
       "user_group_id             0\n",
       "gender                    0\n",
       "age_level                 0\n",
       "user_depth                0\n",
       "city_development_index    0\n",
       "var_1                     0\n",
       "is_click                  0\n",
       "hour                      0\n",
       "day_of_week               0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I convert non numeric features into numeric features (gender and product):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['gender'] = data['gender'].map({'Male' : 0, 'Female' : 1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 'product' variable will convert by the LabelEncoder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "C    157822\n",
       "H    104976\n",
       "I     61075\n",
       "D     39524\n",
       "B     21262\n",
       "E     20639\n",
       "A     14684\n",
       "J      9271\n",
       "G      9144\n",
       "F      6651\n",
       "Name: product, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['product'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_encoder = LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['product'] = label_encoder.fit_transform(data['product'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally I will dropping unnecessary values which not chosen to models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data.drop(['session_id','user_id', 'DateTime'],axis=1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product</th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>webpage_id</th>\n",
       "      <th>product_category_1</th>\n",
       "      <th>product_category_2</th>\n",
       "      <th>user_group_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_level</th>\n",
       "      <th>user_depth</th>\n",
       "      <th>city_development_index</th>\n",
       "      <th>var_1</th>\n",
       "      <th>is_click</th>\n",
       "      <th>hour</th>\n",
       "      <th>day_of_week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>105960</td>\n",
       "      <td>11085</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>359520</td>\n",
       "      <td>13787</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>405490</td>\n",
       "      <td>60305</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   product  campaign_id  webpage_id  product_category_1 product_category_2  \\\n",
       "0        2       359520       13787                   4                  0   \n",
       "1        2       105960       11085                   5                  0   \n",
       "2        2       359520       13787                   4                  0   \n",
       "3        8       359520       13787                   3                  0   \n",
       "4        2       405490       60305                   3                  0   \n",
       "\n",
       "   user_group_id  gender  age_level  user_depth city_development_index  var_1  \\\n",
       "0           10.0       1        4.0         3.0                      3      0   \n",
       "1            8.0       1        2.0         2.0                      0      0   \n",
       "2            8.0       1        2.0         2.0                      0      0   \n",
       "3            3.0       0        3.0         3.0                      2      1   \n",
       "4            2.0       0        2.0         3.0                      2      1   \n",
       "\n",
       "   is_click  hour  day_of_week  \n",
       "0         0     0            6  \n",
       "1         0     0            6  \n",
       "2         0     0            6  \n",
       "3         0     0            6  \n",
       "4         0     0            6  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spliting data into train and test set\n",
    "\n",
    "In this stage need to split dataset into train and test set. The data are exhibit a large imbalance in the distribution of the target classes hence I have to maintain the class proportion in the train-test sets. For this purpose I will use stratified sampling to ensure relative class frequencies is approximately preserved in each train and validation fold.\n",
    "\n",
    "**Stratified Sampling**\n",
    "\n",
    "Stratification makes even distribution of the target(label) in the train and test set - just as it is distributed in the original dataset. For example  the target column for the training set has 80% of 'yes' and 20% of 'no', and also, the target column for the test set has 80% of 'yes' and 20% of 'no' respectively.\n",
    "\n",
    "In this case I will use StratifiedShuffleSplit method from sci-kit learn:\n",
    "\n",
    "The method is combination of StratifiedKFold and ShuffleSplit. \n",
    "It generates the indices for the training and testing sets. It shuffles each time before breaking down into a training and testing set. Therefore, folds can be repetitive. But it maintains the proportion of all the classes in the testing and training sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array(data.drop(['is_click'],1))\n",
    "y = np.array(data['is_click'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "skf = StratifiedShuffleSplit(n_splits=5, test_size=.25, random_state=0)\n",
    "\n",
    "for train_index, test_index in skf.split(X, y):\n",
    "    X_train, y_train = X[train_index], y[train_index]\n",
    "    X_test, y_test = X[test_index], y[test_index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper functions to modelling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_conf_matrix(pred_set):\n",
    "    \"\"\"The function to plot confusion matrix\"\"\"\n",
    "    plt.figure(figsize=(6,4))\n",
    "    ax = sns.heatmap(confusion_matrix(y_test, pred_set),\n",
    "                annot=True,fmt = \"0.1f\",linecolor=\"k\",linewidths=3)\n",
    "    ax.set_ylim(sorted(ax.get_xlim(), reverse=True))   \n",
    "\n",
    "    plt.title(\"Confusion Matrix\",fontsize=14)\n",
    "    plt.show()\n",
    "\n",
    "    \n",
    "def f_score(prediction):\n",
    "    \"\"\"F1 score calculation\"\"\"\n",
    "    f1_score = metrics.f1_score(y_test, prediction, average='weighted')\n",
    "    score = round(f1_score,3)\n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build models\n",
    "\n",
    "To build models I will use the following classification models:\n",
    "\n",
    "- Logistic Regression,\n",
    "- Linear SVC (Support Vector Classifier),\n",
    "- Decision Tree,\n",
    "- Random Forest,\n",
    "- Ada Boosting.\n",
    "\n",
    "To evaluate the models I will use accuracy score, F1 score and confusion matrix.\n",
    "\n",
    "**SMOTE**\n",
    "\n",
    "The target class variable is imbalanced, where \"not click\" values are more dominating then \"is click\". The simplest way to improve imbalanced dataset is balancing them by oversampling instances of the minority class or undersampling instances of the majority class. I will try to balancing classes by using one of the advanced techniques like the SMOTE method. The aim of the SMOTE method is to balance class distribution by randomly increasing minority class examples by replicating them.\n",
    "\n",
    "**Standardization/Normalization of data**\n",
    "\n",
    "In this step I scale the data so that data points are on the same level and I will use this in some algorithms (e.g. Logistic regression, SVC).\n",
    "\n",
    "**Pipeline**\n",
    "\n",
    "To make the scaler => SMOTE => classifier easier to work with I will use Pipeline class in scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Logistic Regression:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('scaler', MinMaxScaler()), ('smote', SMOTE()),\n",
       "                ('classifier', LogisticRegression(C=0.01, random_state=0))])"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = imbpipeline(steps=[('scaler', MinMaxScaler()),\n",
    "                           ('smote', SMOTE()),\n",
    "                           ('classifier', LogisticRegression(penalty='l2', C=0.01, random_state=0))])\n",
    "\n",
    "                        \n",
    "model.fit(X_train, y_train)                   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)\n",
    "log_acc = accuracy_score(y_pred, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measuring accuracy on Testing Data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on testing data: 0.5373982132264385\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.54      0.68    103748\n",
      "           1       0.08      0.54      0.14      7514\n",
      "\n",
      "    accuracy                           0.54    111262\n",
      "   macro avg       0.51      0.54      0.41    111262\n",
      "weighted avg       0.88      0.54      0.65    111262\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy on testing data: %s' % log_acc)\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.647\n"
     ]
    }
   ],
   "source": [
    "print('F1 score: %s' % f_score(y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Linear SVC (Support Vector Classifier)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('scaler', StandardScaler()), ('smote', SMOTE()),\n",
       "                ('classifier', LinearSVC(random_state=0))])"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svc_model = imbpipeline(steps=[('scaler', StandardScaler()),\n",
    "                               ('smote', SMOTE()),\n",
    "                               ('classifier', LinearSVC(random_state=0))])\n",
    "\n",
    "svc_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = svc_model.predict(X_test)\n",
    "svc_acc = accuracy_score(preds, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.5374251766101634\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.54      0.68    103748\n",
      "           1       0.08      0.54      0.14      7514\n",
      "\n",
      "    accuracy                           0.54    111262\n",
      "   macro avg       0.51      0.54      0.41    111262\n",
      "weighted avg       0.88      0.54      0.65    111262\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('accuracy: %s' % svc_acc)\n",
    "print(classification_report(y_test, preds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.647\n"
     ]
    }
   ],
   "source": [
    "print('F1 score: %s' % f_score(preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Decision Tree**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('smote', SMOTE()),\n",
       "                ('classifier',\n",
       "                 DecisionTreeClassifier(criterion='entropy', max_depth=2,\n",
       "                                        random_state=20))])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_tree = imbpipeline(steps=[('smote', SMOTE()),\n",
    "                                ('classifier', DecisionTreeClassifier(max_depth=2, criterion='entropy', random_state=20))])\n",
    "\n",
    "model_tree.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model_tree.predict(X_test)\n",
    "tree_acc = accuracy_score(pred, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7760151713972425\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.82      0.87    103748\n",
      "           1       0.08      0.24      0.12      7514\n",
      "\n",
      "    accuracy                           0.78    111262\n",
      "   macro avg       0.51      0.53      0.50    111262\n",
      "weighted avg       0.88      0.78      0.82    111262\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('accuracy: %s' % tree_acc)\n",
    "print(classification_report(y_test, pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.821\n"
     ]
    }
   ],
   "source": [
    "print('F1 score: %s' % f_score(pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Random Forest Classifier:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('smote', SMOTE()),\n",
       "                ('classifier',\n",
       "                 RandomForestClassifier(criterion='entropy', max_depth=5,\n",
       "                                        n_estimators=200, random_state=0))])"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_rf = imbpipeline(steps=[('smote', SMOTE()),\n",
    "                              ('classifier', RandomForestClassifier(max_depth=5, n_estimators=200, criterion='entropy', random_state=0))])\n",
    "\n",
    "model_rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_y = model_rf.predict(X_test)\n",
    "rf_acc = accuracy_score(pred_y, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7636749294458126\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.80      0.86    103748\n",
      "           1       0.09      0.28      0.14      7514\n",
      "\n",
      "    accuracy                           0.76    111262\n",
      "   macro avg       0.51      0.54      0.50    111262\n",
      "weighted avg       0.88      0.76      0.81    111262\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('accuracy: %s' % rf_acc)\n",
    "print(classification_report(y_test, pred_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.814\n"
     ]
    }
   ],
   "source": [
    "print('F1 score: %s' % f_score(pred_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**AdaBoost Classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('smote', SMOTE()),\n",
       "                ('classifier',\n",
       "                 AdaBoostClassifier(n_estimators=200, random_state=0))])"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ada_model = imbpipeline(steps=[('smote', SMOTE()),\n",
    "                               ('classifier', AdaBoostClassifier(n_estimators=200 ,random_state=0))])\n",
    "\n",
    "ada_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict = ada_model.predict(X_test)\n",
    "ada_acc = accuracy_score(predict, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.9047293775053478\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.97      0.95    103748\n",
      "           1       0.12      0.06      0.08      7514\n",
      "\n",
      "    accuracy                           0.90    111262\n",
      "   macro avg       0.53      0.51      0.52    111262\n",
      "weighted avg       0.88      0.90      0.89    111262\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('accuracy: %s' % ada_acc)\n",
    "print(classification_report(y_test, predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.891\n"
     ]
    }
   ],
   "source": [
    "print('F1 score: %s' % f_score(predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEJCAYAAABmA8c1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5xV1bn/8c/DOAho6FXAS6L8IkSDIuCIDUSQYijXEjAqKl6MPSa5iTUYNZYbowkaCxEUG0gsQAREpBiREhA7aMASGEE6iBVn5vn9cdYMB6ad6TNrvm9f+3XOXnvttdc+js9ZPnvtfczdERGRmq1OVXdARETKTsFcRCQCCuYiIhFQMBcRiYCCuYhIBBTMRUQioGBey5nZTWa20czczM4vh/Y6hLa6lUP3qi0z6xXOs3lV90UEFMyrJTNrZWZ/MbMPzexbM/vUzGaZ2cByPs7hwBjg50Ab4OlyaHZdaOvNcmirUEnBdKeZNdhnW6ewrUTB1sweNbMXUqy+iMR5bi1Bt0UqzH5V3QHZm5l1AF4DdgHXAm+R+NLtAzwIHFyOhzs0vE71crp7zN2zgc/Ko60U7QTOBCYmlY0C1lK+n1UeM0t3991U7nmKFEkj8+rnfsCAbu4+xd0/cPdV7n4f0CW3kpkdbGbPm9musDxnZu2Stt9kZu+a2fAwwt9lZlNzR6pmdhPwfKieY2YeyvONTnPbSlo/wszmmtnnod23zKx32JYvzWJmJ5rZUjP7JqR07jGzuknbF5jZ/WZ2m5ltMbNNZnaXmaXy9/kocGFSW+nAuaE8+RzSzGy8mX1sZl+b2Woz+03uMcLnMRIYlDSq75V0PiPMbJ6ZfQ1cvG+aJbT9npnVTzrewhKM9EXKRMG8GjGzpkB/4D53/2Lf7e6+PdQzYCrQCjgZ6A0cBEwN23J1AH4KDAP6AUcBfwjb7gL+J7xvE5ZUPQVsAHqENm8CvinknNoCs4A3Qt1RwAjg9n2q/gzIAnoClwO/CH0vzhNADzM7JKyfBnwBLNinXh3gU+AsoBNwPXAdcEHYfhcwBXiZPZ/HoqT9byfxRduZxGe/ryuB9NAOof1DSfqiEalISrNUL4eSGJWvKqbeKSRG6Ye4+ycAZnY2sIZEOublUG8/4Hx33xnqjCMEL3f/wsx2hPclTRf8F3CXu78f1tcUUfdSEoH/UnfPAVaZ2TXAQ2Z2o7t/FeqtdPffhff/NrP/CecyqZi+bAOmkwia15P4sngE2Ctt5O7fAb9LKvrEzLqS+GIZHz6Pr4Fvkz+PpO/Ge939maTyQ5Pawt2/DP8OFpnZVhIpssHuvqmY/ouUC43MqxcrvgqQGFmuzw3kAO7+EbCexMgx139yA3mwHmhZ1k4CdwMPh7TD9WZ2WDF9XRwCea6FQF325OwB3t5nv5L0dTww0szaA33ZJ8WSy8x+bmbLzWyzmX0BXE3qefXlxVVw9+Uk/s/nRmCcu89KsW2RMlMwr15WkxhRdiqmnrHPyDNJcvl3BWwr7t95Dvm/VNL3asT9JvakG3oCb5tZYemEiuxrrpeBbOAxYJ67Z+brhNlPgT+TCPSnAkeSSJvU3bduIb4srkJIcR0f+nLIPikvkQqlYF6NuPs2YDZwuZkduO92M2sc3q4E2oaZL7nbfkAib76yjN3YTP78+ZEF9HW1u49190EkRsYXFdLeSuDYfS5mHg/sBj4sY19z+5JDIkj3Cn0pyPHAUne/z91XuPsa4JB96uwG0srQlV8CXYETgQzgijK0JVIiCubVz6UkRrPLzexMM/uhmR1mZpewJxXxMokpi0+a2dFh5siTwApgXhmPPw84yswuNLNDzew3wHG5G82svpn9NWmmxzEkAmVhXyL3k/iSuT/M/x4E3EHiIu9XhexTGrcCLYDnCtn+b6CrmQ0ws45mdiNw0j51PgEOD5958zAzJiVm1oVEimW0uy8CLgHutMRcfpEKp2Bezbj7xyRGd3OAO0kE8HnAYODiUMeBoSRG0QuA+STmPA8t63xxd58N/J5EYHqdxIyY+5OqZANNSMzr/oDE9MbFJEalBbX3KTCAxEyWN4EJJC5qXleWfhZwnO/cfcs+uflkD5GYrfIUsIzEef1pnzp/I3HxeTmJz/Y4UmBm9Uh8mT7l7s+G/kwCniHxhbt/yc5GpORMvzQkIlLzaWQuIhIBBXMRkQgomIuIFMLMJoTHSyQ/zqKpmc0Jj4SYY2ZNQrmZ2VgzW2Nmb4eb0nL3GRnqrzazkUnlR5vZO2GfsbnTWQs7RlEUzEVECvcoiUdsJLsGmOvuHYG5YR0SF/o7hmU08ADkPaZjDHAMiUdgjEkKzg+Eurn79S/mGIWq8AugFh7gJCJSHHcv841W3235KOWYk978B8UeL9zP8YK7Hx7WPwB6ufsGM2sDLHD3H5rZQ+H9pOR6uYu7XxzKHyIxC20BMN/dDwvlI3LrFXaMovqpkbmI1FpmNjo84iF3GZ3Cbq3cfQNAeM197ERbEs/zz5UZyooqzyygvKhjFKrSHrS1e3O53OwnNVzdFnvfdHlA/Q5V0xGpVr78+pPyaywnO+Wq7j4OGFdORy5olO+lKC8VjcxFJC7ZWakvpbMxpD4Ir7lPxswE2ifVa0figXFFlbcroLyoYxRKwVxEouKek/JSStNJ/JAJ4XVaUvl5YVZLBrAzpEhmA/3MrEm48NkPmB227TKzjDCL5bx92iroGIXS88xFJC45pQ7S+ZjZJBIXMJubWSaJWSl3AFPMLPfnCc8M1WcCA0k83/8r9vx2wDYzu4XEYyQAbg4P1YPEM3weBeqT+BGX3McmF3aMwvtaWbNZlDMXUM5cCpabMy+P2Sy7172VclCr275LNI8p1shcROJSggugMVEwF5G4lD4XXqMpmItIVLz0s1RqNAVzEYlLOV4ArUkUzEUkLkqziIhEQBdARUQioJG5iEgEdAFURCQCugAqIlLzuStnLiJS8ylnLiISAaVZREQioJG5iEgEsr+r6h5UCQVzEYmL0iwiIhFQmkVEJAIamYuIREDBXESk5nNdABURiYBy5iIiEVCaRUQkAhqZi4hEQCNzEZEIaGQuIhKBLP04hYhIzaeRuYhIBJQzFxGJgEbmIiIR0MhcRCQCGpmLiERAs1lERCLgXtU9qBIK5iISF+XMRUQiUEuDeZ2q7oCISLnynNSXYpjZ1Wb2npm9a2aTzKyemX3fzJaa2Woze9rM6oa6+4f1NWF7h6R2rg3lH5jZqUnl/UPZGjO7piynrWAuInHJzk59KYKZtQWuBLq5++FAGjAcuBO4x907AtuBUWGXUcB2dz8UuCfUw8w6h/1+BPQH7jezNDNLA/4KDAA6AyNC3VJRMBeRuOTkpL4Ubz+gvpntBzQANgAnA8+E7ROBoeH9kLBO2N7HzCyUT3b3b939Y2AN0CMsa9z9I3ffDUwOdUtFwVxE4lKCYG5mo81sedIyOrcZd/8UuAtYSyKI7wReB3a4e+78x0ygbXjfFlgX9s0K9Zsll++zT2HlpaILoCISlxLcNOTu44BxBW0zsyYkRsrfB3YAfyeREsnXTO4uhWwrrLygwXSp51UqmItIVDyn3OaZnwJ87O6bAczsOaAn0NjM9guj73bA+lA/E2gPZIa0TCNgW1J5ruR9CisvMaVZRCQu5ZczXwtkmFmDkPvuA6wE5gNnhDojgWnh/fSwTtg+z909lA8Ps12+D3QE/gUsAzqG2TF1SVwknV7a09bIXETiUswslVS5+1IzewZYAWQBb5BIycwAJpvZraFsfNhlPPC4ma0hMSIfHtp5z8ymkPgiyAIuc/dsADO7HJhNYqbMBHd/r7T9Na/gW1/NzAF2b/6wQo8jNUPdFofstX5A/Q5V0xGpVr78+hMA3L2g/HKJfHXvpSkHtQZX3F/m41UXGpmLSFx0B6jkuuG2uzlx0HCGnvPzvLKdn+/ioquuY+BPR3HRVdex8/NdALg7t93zAAPOupBh513Cyg/W5O1z8S9v4NhTz+DS/x2zV/tLX3+TMy+4nKHn/JzrbrmLrKzsYttK9t77qxl27iUMOOtCbrvnASr6/66kcPvvX5cF/5zK4iUzWbZ8Ntff8AsAHnzoj7y78p8sWjKDRUtmcMSPOwFwwgnH8OmGt/LKr7n2iry2Lrv8QpYtn82/lr3II4/+hf33r5vveHXr1mXiY/fy1jvzmf/K8xx8cKlnssXLPfUlIgrmBRg6sC8P3n3rXmUPPz6FjG5HMvPp8WR0O5LxT0wB4NXFy1ibuZ6ZT4/npt9cyS133Ze3zwVnn87tN/56r3ZycnK47tY/8cffX8PUJx7koNYtmTbr5WLbSnbLXfcx5rdXMvPp8azNXM/CJcvL8/SlBL79djeDBpzNsRkDOTZjEKf0PYnu3Y8E4IbrbqdnxiB6ZgzinbdX5e2zaNGyvPI7br8XgDYHteKSS8/nhOMH06N7f9LS0jjjzJ/kO97I889ix46ddDmiN3+9dzy33FqmO8DjVL43DdUYCuYF6HbkETRq+L29yua/upghA04BYMiAU5j3z8WJ8oVLGNy/D2ZGl8M7sWvXF2zesg2AjG5H0aBBg73a2bHzc+qmp9Ph4HYAHNu9Ky8vWFhsW7k2b9nGl19+xZGHd8LMGNy/D/NeXVz+H4Kk7MsvvwIgPX0/0tP3K/VE4f32S6N+/XqkpaVRv0E9NmzYlK/OoEF9efKJZwF4/vlZ9OrVs7TdjleOp75EpNhgbmaHmdlvzWysmf0lvO9UGZ2rTrZu30GL5k0BaNG8Kdt27ARg4+attG7ZPK9eq5bN2bh5S6HtNGnciKysbN5d9W8AXlqwkM82bUm5rY2bt9AquU6L5mzcvLWMZydlUadOHRYtmcHH/1nOvLkLWb7sTQB+d9OvWbJ0FnfceQN16+5JmfTo0ZXFS2by3NRH6NSpIwAb1m9k7J//xqoPXuPDj5by+c5dzJv7ar5jHXRQKzI/3QBAdnY2Oz/fRbNmTSrhLGuQcno2S01TZDA3s9+SeF6AsWdepAGTinrCV/ItsuXZ2eqooHx1YkpqwcyMP958Df83dhzDL7qKAxrUJy2tTspteQHjviIOJ5UgJyeHnhmD+GHHY+nWrQudO/8/xoz5P7oe2YcTTxhCkyaN+eWvLgbgzTffo/Nhx3NsxkAefGAik55+CIDGjRsy6LS+HN75RA49JIMGBzTgp8OH5jtWQX9bumayN8/JSXmJSXEj81FAd3e/w92fCMsdJB4QM6qwndx9nLt3c/du5dnZqtSsSeO8lMfmLdto2rgRAK1bNs8bWQNs3LSFls2bFdnWkYd34rEH7mLyw3/h6C6H81/t26bcVusWLdiYXGdz8ceTyrFz5y5efXUJp/Q9iY2fbQZg9+7dPPH43zm6WxcAdu36Ii8t89LsBaSnp9OsWRN69z6eT/6zji1btpGVlcX0abPJyOia7xiffvoZ7dq2ASAtLY1GDb/Htm07KukMawilWQqUAxxUQHmbsK3W6HV8Rt6FymmzXqb3CcfmlU9/cS7uzlvvruLAAw/IS8cUZuv2xH98u3fvZsKTf+esoQNTbqtF86Y0aFCft95dhbsz/cW59D4+o7xPV1LUvHlTGjVKXF+pV29/evc+nn//+0NatW6RV+e0n/Rj5XuJtFrLVntSZEd360KdOsbWrdtZl7meHt2Pon79egD06tWTD97Pf2/GzJkv87NzTgdg2LABvPKKrpfkU47PM69Jiptn/gtgrpmtZs/TvQ4GDgUur8iOVaX/HXMHy954mx07PqfP0HO4dNS5XHTuWfzqxtt47oXZtGnVgrtvvR6AE4/tzquLlzHgrAupX68et1x3dV47513yaz5eu46vvvqGPkPP4eZrr+a4Y47mkSef4ZVF/8JzcvjpsEEcc/SRxbZ1+sjLeHbiXwG48deXc8Mf7uabb7/lhIzunHBs90r8dCRZq9YtGfe3u0irk0adOsZzz83gxVnzmDHzSZo3b4qZ8fbbq7jqysTfy7BhA7noop+RlZXN1998w/nnXQnA8mVvMnXqLF5b9AJZWVm89dZKJkyYBMANN17NihXvMHPGy0x89GkeHn8Pb70zn+3bd3L+eVcU2rdaK7IRd6qKvQPUzOqQSKu0JZEvzwSW5d6OWuwBdAeoJNEdoFKQ8rwD9MvfDU85mh9w8+RorjgVeweou+cASyqhLyIiZRdZ+iRVup1fROJSS9MsCuYiEpXYphymSsFcROKikbmISAQUzEVEIhDZbfqpUjAXkaiU42+A1igK5iISFwVzEZEIaDaLiEgENDIXEYmAgrmISM3n2UqziIjUfBqZi4jUfJqaKCISAwVzEZEI1M6UuYK5iMTFs2pnNFcwF5G41M5YrmAuInHRBVARkRhoZC4iUvNpZC4iEgONzEVEaj7PquoeVA0FcxGJimtkLiISgVoazOtUdQdERMqT56S+FMfMGpvZM2b2vpmtMrNjzaypmc0xs9XhtUmoa2Y21szWmNnbZtY1qZ2Rof5qMxuZVH60mb0T9hlrZlba81YwF5GolGcwB/4CvOjuhwFdgFXANcBcd+8IzA3rAAOAjmEZDTwAYGZNgTHAMUAPYEzuF0CoMzppv/6lPW8FcxGJimdbyktRzKwhcCIwHsDdd7v7DmAIMDFUmwgMDe+HAI95whKgsZm1AU4F5rj7NnffDswB+odtDd19sbs78FhSWyWmYC4iUSnJyNzMRpvZ8qRldFJTPwA2A4+Y2Rtm9rCZHQC0cvcNAOG1ZajfFliXtH9mKCuqPLOA8lLRBVARiYrnpJ52dvdxwLhCNu8HdAWucPelZvYX9qRUClLQgb0U5aWikbmIRKUcc+aZQKa7Lw3rz5AI7htDioTwuimpfvuk/dsB64spb1dAeakomItIVNwt5aXodvwzYJ2Z/TAU9QFWAtOB3BkpI4Fp4f104LwwqyUD2BnSMLOBfmbWJFz47AfMDtt2mVlGmMVyXlJbJaY0i4hEpZxvGroCeNLM6gIfAReQGARPMbNRwFrgzFB3JjAQWAN8Feri7tvM7BZgWah3s7tvC+8vAR4F6gOzwlIqlriIWnHMzAF2b/6wQo8jNUPdFofstX5A/Q5V0xGpVr78+hMAvLjhcgrWduuTclA7ePncMh+vutDIXESiUpILoDFRMBeRqCiYi4hEoIIzx9WWgrmIREUjcxGRCJTDNdQaScFcRKKSXcwzV2KlYC4iUdHIXEQkAsqZi4hEQLNZREQioJG5iEgEsnNq5/MDFcxFJCpKs4iIRCBHs1kq1r5PyxOBPU/LEykvmpooIhIBpVlERCKgNIuISAQ0m6WCtW7UqbIOJdXYZztX7bWeln5QFfVEqpPs70r9O8b51NIsi0bmIhIXpVlERCKg2SwiIhHIqeoOVBEFcxGJiqORuYhIjZelNIuISM2nkbmISASUMxcRiYBG5iIiEdDIXEQkAtkamYuI1Hy19FfjFMxFJC45GpmLiNR8etCWiEgEdAFURCQCOaY0i4hIjZdd1R2oIrXzJzlEJFo5lvqSCjNLM7M3zOyFsP59M1tqZqvN7GkzqxvK9w/ra8L2DkltXBvKPzCzU5PK+4eyNWZ2TVnOW8FcRKKSg6W8pOgqIPknsu4E7nH3jsB2YFQoHwVsd/dDgXtCPcysMzAc+BHQH7g/fEGkAX8FBgCdgRGhbqkomItIVLwES3HMrB0wCHg4rBtwMvBMqDIRGBreDwnrhO19Qv0hwGR3/9bdPwbWAD3CssbdP3L33cDkULdUFMxFJCrlnGb5M/Ab9kySaQbscPessJ4JtA3v2wLrAML2naF+Xvk++xRWXioK5iISlZwSLGY22syWJy2jc9sxs9OATe7+elLzBX0FeDHbSlpeKprNIiJRyS7BzER3HweMK2TzccBgMxsI1AMakhipNzaz/cLoux2wPtTPBNoDmWa2H9AI2JZUnit5n8LKS0wjcxGJSklG5kVx92vdvZ27dyBxAXOeu/8MmA+cEaqNBKaF99PDOmH7PHf3UD48zHb5PtAR+BewDOgYZsfUDceYXtrz1shcRKJSCXeA/haYbGa3Am8A40P5eOBxM1tDYkQ+HMDd3zOzKcBKIAu4zN2zAczscmA2kAZMcPf3StspS3xxVBwzc4DWjTpV6HGkZvhs56q91tPSD6qinkh1kv1dIrvgXvYf8Hyw/TkpB7Wfr3simttFNTIXkajo2SwiIhGorbfzK5iLSFT04xQiIhFQmkVEJAIK5iIiEdAvDYmIREA5cxGRCGg2i4hIBHJqaaJFwVxEoqILoCIiEaid43IFcxGJjEbmIiIRyLLaOTZXMBeRqNTOUK5gLiKRUZpFRCQCmpooIhKB2hnKFcxFJDJKs4iIRCC7lo7NFcxFJCoamYuIRMA1MhcRqflq68i8TlV3oCY55NAOzHn1ubzl32v/xf9cci6NGzdi8vMP89rrs5j8/MM0atRwr/26HHU4mVvfYdDgfgD86IjD+MdLT7Fg8XTmvvY8g4f1L/B4deum8+CEP7FoxYvMeHky7Q4+qMLPUcquTp06LPvXbKY9PxGABfOeY/myl1i+7CXWfvI6zz4zHoARI4ax4vU5rHh9Dq++Mo0f/7hzge116NCeRQv/war3FvLUkw+Qnp5eaedSE+XgKS8xUTAvgQ/XfELfE/6bvif8N6eedAZff/0Ns16Yy+VXX8TCV5Zw3NEDWPjKEi6/+qK8ferUqcMNv/8lC+a+llf29Vdfc+XPr6XXsYM5+/TR3Hz7tTRs9L18xxtx7uns3PE5Pbv2Z9z9E7nhpl9VynlK2Vx5xUW8//7qvPVeJ/833br3o1v3fixZ+jrPT50FwCcfr+PkPmfQ9ei+/OG2P/Pg/XcW2N7tt13Pn8f+jU4/Op7t23dy4QUjKuU8aiovwRITBfNSOuGkDD75eC2Z69Zz6sCTmTJpKgBTJk2l/6A+efVGXfwzZkyfw5YtW/PKPvrwP3z80X8A2PjZZrZs2UqzZk3zHaN/UrsvTHuJE07KqMhTknLQtm0bBg7ow4QJk/JtO/DAA+jd6zimTXsRgMVLlrNjx04AlixdQdu2bQpss3ev43j22RkAPP743xky+NQK6n0csvCUl5gomJfSkNMHMvXZmQC0aNmMTRu3ALBp4xaat0gE5tZtWjLgtFN4bMLThbZzZNcjqJuezicfr823rXWbVqz/9DMAsrOz+fzzXTRt2ri8T0XK0d1/+j3XXHsrOTn5M7dDhw5g3vzX2LXri3zbLrxgOC/Onp+vvFmzJuzYsZPs7MTv52R+uoGD2rYu/45HxEvwT0xKHczN7IIito02s+Vmtry07Vdn6enpnDqgN/+YOrvIejfffi23jvlTgf9hA7Rs1Zx7H7qDX1x2Pe75/7DM8v+YYUH1pHoYNPAUNm3awoo33ilw+/CzhjD56an5ynud1JMLLhjBtdfdlm+b/gZKLqcES0zKMpvl98AjBW1w93HAOACz+J5HeXLfE3jnrZVs2ZxInWzetJWWrZqzaeMWWrZqzpbN2wDoctSPeHDCnwBo2rQJffqeSHZ2Ni/OmMuB3zuAJ6Y8yJ23jmXF8rcLPM6G9Z9xUNvWbFi/kbS0NBo2/B7bt++snJOUEuvZsxs/Oa0fA/qfTL16+9Ow4feY+OhYRp5/JU2bNqF796M4/cyL9trniCM68dCDf+S0weeybdv2fG1u2bKNxo0bkZaWRnZ2Nu3atmHD+o2VdUo1Umwj7lQVOTI3s7cLWd4BWlVSH6udoacP5PmQYgF4adZ8zhoxFICzRgxl9sx5ABzTpR89ftyXHj/uywvTZ3PNr27hxRlzSU9PZ8IT9/L3ydN4YVrho/vZSe2eNqQfC/+5tALPSsrq+hvuoMMPunHo/8vgZ+dcyvz5rzHy/CsBOOP005gx82W+/fbbvPrt2x/E35/+G+dfcBWrV39UaLsLXlnE6acPAuDcc89k+j9eqtgTqeFq68i8uDRLK+A84CcFLFuL2C9a9evX48TePZn5jzl5Zffd8zdO7N2T116fxYm9e3LfPQ8X2cbgYf3J6Hk0Z509LG+a44+OOAyA/73ucvoN6A3ApMefpUnTxixa8SIXXzaSP9x0d8WdmFSon541mKf3SbHccP3VNGvWhHvvvY3ly15iyeI9A4R/THuMNm0S46Vrr/sDV181mvdXLqRZsyZMeCT/xVXZI9s95SUmVlT+zczGA4+4+8ICtj3l7mcXe4CQZmndqFNZ+imR+Gznqr3W09I1d14g+7v1ALh7/osEJXT2fw1LOUo/9Z/ny3y86qLInLm7jypiW7GBXESkstXWnLlu5xeRqMSWC0+VgrmIRCW22/RTpWAuIlGprWkW3QEqIlEpr9ksZtbezOab2Soze8/MrgrlTc1sjpmtDq9NQrmZ2VgzWxOmcHdNamtkqL/azEYmlR9tZu+EfcZaQXeJpUjBXESiUo5PTcwCfuXunYAM4DIz6wxcA8x1947A3LAOMADoGJbRwAOQCP7AGOAYoAcwJvcLINQZnbRfwY9QTYGCuYhEpbxuGnL3De6+IrzfBawC2gJDgImh2kRgaHg/BHjME5YAjc2sDXAqMMfdt7n7dmAO0D9sa+juiz0xR/yxpLZKTMFcRKJSkgdtJT9HKiyjC2rTzDoARwFLgVbuvgESAR9oGaq1BdYl7ZYZyooqzyygvFR0AVREolKS2SzJz5EqjJkdCDwL/MLdPy8irV3QBi9FealoZC4iUXH3lJfimFk6iUD+pLs/F4o3hhQJ4XVTKM8E2ift3g5YX0x5uwLKS0XBXESiko2nvBQlzCwZD6xy9+QHI00HcmekjASmJZWfF2a1ZAA7QxpmNtDPzJqEC5/9gNlh2y4zywjHOi+prRJTmkVEolKONw0dB5wLvGNmb4ay64A7gClmNgpYC5wZts0EBgJrgK+ACwDcfZuZ3QIsC/Vudvdt4f0lwKNAfWBWWEpFwVxEolJeP94RHjBYWIK8z74FYUbKZYW0NQGYUED5cuDwMnQzj4K5iERFt/OLiESgtt7Or2AuIlGJ7UcnUqVgLiJRUZpFRCQCCuYiIhEor9ksNY2CuYhERSNzEZEIaDaLiEgEsr12/gqogrmIREU5cxGRCChnLiISAeXMRUQikKM0i4hIzaeRuYhIBDSbRXJzWlYAAAHlSURBVEQkAkqziIhEQGkWEZEIaGQuIhIBjcxFRCKQ7dlV3YUqoWAuIlHR7fwiIhHQ7fwiIhHQyFxEJAKazSIiEgHNZhERiYBu5xcRiYBy5iIiEVDOXEQkAhqZi4hEQPPMRUQioJG5iEgENJtFRCQCugAqIhIBpVlERCJQW+8AtYr+FjOz2vnJikiJubuVtY26+7dLOebs/jazzMerLjQyF5Go1NaceYWPzGUPMxvt7uOquh9SvejvQspDnaruQC0zuqo7INWS/i6kzBTMRUQioGAuIhIBBfPKpbyoFER/F1JmugAqIhIBjcxFRCKgYC4iEgEF80piZv3N7AMzW2Nm11R1f6TqmdkEM9tkZu9WdV+k5lMwrwRmlgb8FRgAdAZGmFnnqu2VVAOPAv2ruhMSBwXzytEDWOPuH7n7bmAyMKSK+yRVzN3/CWyr6n5IHBTMK0dbYF3SemYoExEpFwrmlaOgJ7NpTqiIlBsF88qRCbRPWm8HrK+ivohIhBTMK8cyoKOZfd/M6gLDgelV3CcRiYiCeSVw9yzgcmA2sAqY4u7vVW2vpKqZ2SRgMfBDM8s0s1FV3SepuXQ7v4hIBDQyFxGJgIK5iEgEFMxFRCKgYC4iEgEFcxGRCCiYi4hEQMFcRCQC/x8sCkz3B/niPAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conf_matrix(predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the model\n",
    "with open(\"adaboost_model.pkl\", \"wb\") as model_file:\n",
    "    pickle.dump(ada_model, model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature Evaluation\n",
    "\n",
    "After modelling I will check which features play the most important role in the advertisement click prediction by the use an attribute named feature_importance. It contains information about the most important features for a given classification. The analysis is based on Random Forest model \n",
    "\n",
    "The following code creates a plot of the top 10 features for predicting ad click:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x26102fd0898>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd0AAAD4CAYAAACpKzxfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhcZZ328e9NwGAEwhaZjCM2ooKswbRgWAPiMoOKCIwoIgFnMsyouAwvk3lhkHGZCx1HXNDB6LAjYMAgEBXZEtYAnZCkCRJUaF8NDCpgZBfC/f5RT5tKU91d3V051SH357rq6lPPeZZfnVyVXz3nPFVHtomIiIg1b712BxAREbGuSNKNiIioSJJuRERERZJ0IyIiKpKkGxERUZH12x1AjG5bbrmlOzo62h1GRMRaY8GCBb+3PaHRviTdGFBHRwddXV3tDiMiYq0h6Vf97cvp5YiIiIok6UZERFQkSTciIqIiSboREREVyUKqGFD38hV0zJgzaL2e0w6qIJqIiLVbZroREREVSdJdS0nqkHT3MNtOlbRnq2OKiIiBJemOMpLGVDDMVCBJNyKiYkm6FSqz03slnStpiaRLJY2T1CPpFEk3A4dLmiRpfqkzW9Jmpf1kSYsl3QZ8tK7faZLOqHt+laSpZfudkhaWdtdJ6gCOAz4laZGkfSo8BBER67Qk3eptB8y0vQvwR+CfSvkztve2fTFwHvAvpU438JlS52zgeNtTmhlI0gTgO8ChtncFDrfdA5wJnG57ku2bGrSbLqlLUtfKp1YM/5VGRMRqknSr92vbt5TtC4C9y/YlAJLGA5vanlfKzwX2bVB+fhNjvQW40fYDALYfbSZA2zNtd9ruHDNufDNNIiKiCUm61XM/z58cpJ0atO31PKv/W27YRJuIiKhYkm71tpbUe3r4A8DN9TttrwAeq7vWehQwz/YfgBWSemfGR9Y16wEmSVpP0quB3Uv5bcB+krYBkLR5KX8c2LiFrykiIpqQpFu9nwFHS1oCbA78d4M6RwP/WepMAj5byo8BvlkWUj1dV/8W4AFq13+/DCwEsP07YDrwA0mLKaewgSuBQ7KQKiKiWrJz9rEqZeXwVbZ3anMoTRs78fWeePRXB62XX6SKiKiRtMB2Z6N9+RnIGNDOrxpPVxJqRERLJOlWqHxdZ62Z5UZERGvlmm5ERERFknQjIiIqkqQbERFRkSTdiIiIiiTpRkREVCRJNyIioiJJuhERERVJ0o2IiKhIkm5ERERF8otUMaDu5SvomDFnxP3kt5kjIjLTjYiIqEySbkREREWSdEcRSe+RNKPFfX5X0g4NyqdJOqOVY0VExMByTXcUsX0FcEWL+/y7VvYXERHDl5luHUkflrRE0mJJ50t6t6TbJd0l6VpJW5V6p0o6V9JPJfVIep+kL0nqlvQTSRuUej2SvijpjvJ4XSnvr98/zz4lbStpvqQ7JX1W0hOlfKqkuZIulXSvpAslaYDXNFdSZ9k+RtJ9kuYBew3QZrqkLkldK59a0aKjGxERSbqFpB2Bk4ADbO8KfAK4GXiL7d2Ai4ET65psCxwEHAxcANxge2fg6VLe64+2dwfOAL5aygbqt9fXgK/ZfjPwYJ99uwGfBHYAXssACbTu9U0E/r3UfVtp25DtmbY7bXeOGTd+sK4jIqJJSbqrHABcavv3ALYfBf4KuFpSN/B/gB3r6v/Y9nNANzAG+Ekp7wY66updVPd3StkeqN9eU4BZZft7ffbdYfs3tl8AFvUZrz97AHNt/872n4BLmmgTEREtlKS7igD3KfsGcEaZwf4DsGHdvmcBSuJ7znZv2xdY/Vq5G2wP1G8znq3bXknz1+b7vr6IiKhQku4q1wF/K2kLAEmbA+OB5WX/0cPs9/11f28r2830Ox84tGwfMcyx690OTJW0RbnmfHgL+oyIiCHI6uXC9lJJXwDmSVoJ3AWcCsyStJxaEtxmGF2PlXQ7tQ84HyhlzfT7SeACSf8MzAFGtKLJ9kOSTqWW+B8CFlI7LT6gnV81nq78mlREREto1VnRaDVJPUBn73XiIbYdBzxt25KOAD5g++BWxziYzs5Od3V1VT1sRMRaS9IC252N9mWmO3pNBs4oXwf6A3Bsm+OJiIgRStJdg2x3jKDtTcCuzdaXNJsXn6b+F9tXDzeGiIhorSTdlwjbh7Q7hoiIGFhWL0dERFQkSTciIqIiSboREREVSdKNiIioSJJuRERERZJ0IyIiKpKvDMWAupevoGPGnJb325OfloyIdVBmuhERERVJ0o2IiKjISybpSuqQdPcw206VtGeL4/m/reyvFSTtK2mhpOclHdbueCIi1jWjPulKGvT2cy0wFWhp0gXWeNKVNNRr8v8PmAZ8r/XRRETEYNqadMvs9F5J50paIulSSeMk9Ug6RdLNwOGSJkmaX+rMlrRZaT9Z0mJJtwEfret3mqQz6p5fJWlq2X5nme0tlnSdpA7gOOBTkhZJ2qefWLcqYy8ujz1L+eWSFkhaKml6KTsNeHnp78JS9iFJd5Syb/d+mJD0EUn3SZor6Tu9cUt6TYlvSfm7dSk/R9JXJN0A/Kekn0uaUPatJ+kXkrZs9Bps99heArwwvH+xiIgYidEw090OmGl7F+CPwD+V8mds7237YuA8anfM2QXoBj5T6pwNHG97SjMDleT0HeBQ27sCh9vuAc4ETrc9qdzdp5GvA/NKuzcBS0v5sbYnA53A8ZK2sD2D2r1wJ9k+UtIbgfcDe9meBKwEjpT0l8C/AW8B3gZsXzfeGcB55TVfWMbv9QbgQNufAi4AjizlBwKLh3P/3nqSpkvqktS18qkVI+kqIiLqjIak+2vbt5TtC4C9y/YlAJLGA5vanlfKzwX2bVB+fhNjvQW40fYDALYfHUKcBwD/XdqttN2bjY6XtBiYD7waeH2Dtm+ldn/cOyUtKs9fC+xOLZE/avs5YFZdmymsOg18PquOC8As2yvL9lnAh8v2sdQ+iIyI7Zm2O213jhk3fqTdRUREMRq+p+t+nj85SDs1aNvreVb/QLFhE22GrJyyPhCYYvspSXPrxlqtKnCu7X/t034ot+Orj/vPx8b2ryU9LOkAYA9WzXojImKUGQ0z3a0l9Z4e/gBwc/3OMqN8rO5a61HUZod/AFZI6p0B1iebHmBSucb5amozSoDbgP0kbQMgafNS/jiw8SBxXgf8Y2k3RtImwHjgsZJwt6c2k+71nKQN6toeJumVveNKeg1wR4lns7Io6tC69rcCR9S9ttWOSx/fpXaW4Pt1M+CIiBhlRkPS/RlwtKQlwOaUU7h9HE1t0dASYBLw2VJ+DPDNspDq6br6twAPULv++2VgIYDt3wHTgR+UU8KXlPpXAocMtJAK+ASwv6RuYAGwI/ATYP0S1+eonWLuNRNYIulC2/cAJwM/LXWvASbaXg78B3A7cC1wD/Dn09bAMaX+UWX8/lwBbMQgp5YlvVnSb4DDgW9LWjpQ/YiIaC3ZLTvbOvTBayuHr7K9U9uCaDNJG9l+osx0ZwNn2Z49xD46qS0E6+8Dw7B1dna6q6ur1d1GRLxkSVpgu7PRvtEw013XnVoWV91NbXZ++VAaS5oBXAb862B1IyKivdo60x2NJJ1E7fRrvVm2v9COeIajla8hM92IiKEZaKabpBsDStKNiBianF6OiIgYBZJ0IyIiKpKkGxERUZEk3YiIiIok6UZERFQkSTciIqIiSboREREVGQ13GYpRrHv5CjpmzGnL2D2nHdSWcSMi1pTMdCMiIiqSpBsREVGRJN0WktQjact2x9FL0mclHdigfKqkq9oRU0TEuizXdF/CbJ/S7hgiImKVzHQbkHSipOPL9umSri/bb5V0gaS3S7pN0kJJsyRtVNf8/0i6ozxeV9qdI+lMSTdJuk/Su0p5RylbWB57lvL1JH1L0lJJV0n6kaTDyr7JkuZJWiDpakkTB3gd59S1e6ekeyXdDLxvkNc/XVKXpK6VT60Y/oGMiIjVJOk2diPQe0P4TmAjSRsAewPdwMnAgbbfBHQBn65r+0fbuwNnAF+tK+8A9gMOAs6UtCHwW+BtpZ/3A18vdd9X6u8M/B0wBaDE8A3gMNuTgbOAQW/XV8b6DvDu8rr+YqD6tmfa7rTdOWbc+MG6j4iIJuX0cmMLgMmSNgaeBRZSS777AFcAOwC3SAJ4GXBbXduL6v6eXlf+fdsvAD+XdD+wPbWb1p8haRKwEnhDqbs3tfvfvgD8r6QbSvl2wE7ANWXsMcBDTbye7YEHbP8cQNIFwPQm2kVERAsl6TZg+zlJPcAxwK3AEmB/YFtqifIa2x/or3kT273PPwU8DOxK7azDM2Wf+ulbwFLbU5p7Jf3GFRERbZDTy/27ETih/L0JOA5YBMwH9qq7XjtO0hvq2r2/7m/9DPjwcq12W+C1wDJgPPBQmdEeRW3mCnAzcGipvxUwtZQvAyZI+vPpZkk7NvFa7gW2KWMD9PeBISIi1qAk3f7dBEwEbrP9MLVZ6E22fwdMAy6StIRaEt6+rt1YSbcDn6A2k+21DJgH/Bg4zvYzwLeAoyXNp3Zq+clS9zLgN8DdwLeB24EVtv8EHAZ8UdJiah8C9hzshZSxpgNzykKqXw3xWERERAvIzlnHNU3SOcBVti8dQpuNbD8haQvgDmAv2/+7pmLsT2dnp7u6uqoeNiJirSVpge3ORvtyTXf0ukrSptQWan2uHQk3IiJaK0m3AranDaPN1GbrSvomsFef4q/ZPnuo40ZExJqTpPsSYPuj7Y4hIiIGl4VUERERFUnSjYiIqEiSbkREREWSdCMiIiqSpBsREVGRJN2IiIiKJOlGRERUJN/TjQF1L19Bx4w57Q4DgJ7TDmp3CBERI5KZbkREREWSdCMiIiqSpPsSJ+nWfsrPkXRY1fFERKzLknRHQNKIrolLGjN4rZGxPej9diMiohrrVNKV1CHp7rrnJ0g6VdLxku6RtETSxWXfKySdJelOSXdJOriUT5M0S9KVwE/7GWc9Sd+StFTSVZJ+1DurlNQj6ZRyM/nDJU2SNL+MPVvSZqXeXEmdZXtLST114/9Q0k8kLZP0mUFe8xPlrySdUV7nHOCVA7SZLqlLUtfKp1Y0e3gjImIQWb1cMwPYxvaz5R62ACcB19s+tpTdIenasm8KsIvtR/vp731AB7AzteT2M+Csuv3P2N4bQNIS4OO250n6LPAZ4JODxLs7sBPwFHCnpDm2B7vT/CHAdiWmrYB7+sT0Z7ZnAjMBxk58vQfpNyIimrROzXQHsAS4UNKHgOdL2duBGZIWAXOBDYGty75rBki4AHsDs2y/UG4+f0Of/ZcASBoPbGp7Xik/F9i3iXivsf2I7aeBH5TxBrMvcJHtlbYfBK5vok1ERLTQupZ0n2f117xh+XsQ8E1gMrCgXKsVcKjtSeWxte2flfpPDjKOBtk/WPu+sW7YZ1/f2Wezs9HMWiMi2mhdS7oPA6+UtIWkscC7qB2DV9u+ATgR2BTYCLga+LgkAUjabQjj3AwcWq7tbgVMbVTJ9grgMUn7lKKjgN5Zbw+1DwEAfVcZv03S5pJeDrwXuKWJmG4EjpA0RtJEYP9mX0xERLTGOnVN1/Zz5brp7cADwL3AGOCCcqpXwOm2/yDpc8BXgSUl8fZQS9LNuAx4K3A3cF8Zr78VSUcDZ0oaB9wPHFPKvwx8X9JRvPhU8M3A+cDrgO81cT0XYDZwANBdYpo3cPWIiGg12TnjuCZI2sj2E5K2AO4A9irXd0fa7zSg0/bHRtpXMzo7O93V1UxOj4gIAEkLbHc22rdOzXQrdlVZ9fwy4HOtSLgREbF2S9IdAUk7UzvNW+9Z23vYnromxrR9DnBOnzi2AK5rUP2tth9ZE3FERMTQJemOgO1uYNIoiOOR0RBHREQMbF1bvRwREdE2SboREREVSdKNiIioSJJuRERERZJ0IyIiKpKkGxERUZEk3YiIiIrke7oxoO7lK+iYMafdYbRUz2kHtTuEiFhHZaYbERFRkSTdiIiIiiTpNiDpVEknrOExtpe0SNJdkrZdk2PVjTlXUsM7X0RExJqXpNs+7wV+aHs3279sdzAREbHmJekWkk6StEzStcB2pezvJd0pabGkyySNk7SxpAckbVDqbCKpp/d5g34nSZovaYmk2ZI2k/Q3wCeBv5N0Qz/tTpR0fNk+XdL1Zfutki4o22+XdJukhZJmSdqolE+WNE/SAklXS5rYp+/1JJ0r6fP9jD1dUpekrpVPrRjG0YyIiEaSdKklKeAIYDfgfcCby64f2H6z7V2BnwEfsf04MBfoXQJ7BHCZ7ef66f484F9s7wJ0A5+x/SPgTOB02/v30+5GYJ+y3QlsVBL73sBNkrYETgYOtP0moAv4dKnzDeAw25OBs4Av1PW7PnAhcJ/tkxsNbHum7U7bnWPGje8nvIiIGKp8ZahmH2C27acAJF1Ryncqs8FNgY2Aq0v5d4ETgcuBY4C/b9SppPHAprbnlaJzgVlNxrQAmCxpY+BZYCG15LsPcDzwFmAH4BZJAC8DbqM2S98JuKaUjwEequv328D3bdcn4oiIqECS7ipuUHYO8F7biyVNA6YC2L5FUoek/YAxtu9ueTD2c5J6qCX1W4ElwP7AttRm3dsC19j+QH07STsDS21P6afrW4H9Jf2X7WdaHXdERPQvp5drbgQOkfTyMrN8dynfGHionLI9sk+b84CLgLP769T2CuAxSb2niY8C5vVXv5+4Tih/bwKOAxbZNjAf2EvS6wDK9eY3AMuACZKmlPINJO1Y1+f/AD8CZknKh66IiArlP13A9kJJlwCLgF9RS3AA/wbcXsq6qSXhXhcCn6eWeAdyNHCmpHHA/dRmrs26CTgJuM32k5Ke6Y3N9u/K7PsiSWNL/ZNt3yfpMODr5fT2+sBXgaV1r/crZd/5ko60/UJ/Aez8qvF05RecIiJaQrVJUwxVSWwH2z6q3bGsSZ2dne7q6mp3GBERaw1JC2w3/E2EzHSHQdI3gL8G/qbdsURExNojSXcYbH+8b5mkbwJ79Sn+mu1+r/mWdlsA1zXY9Vbbjww/yoiIGG2SdFvE9keH2e4RYFKLw4mIiFEoq5cjIiIqkqQbERFRkSTdiIiIiiTpRkREVCRJNyIioiJJuhERERXJV4ZiQN3LV9AxY067w2ipnvysZUS0SWa6ERERFUnSjYiIqEiS7lpA0jRJZ4z2PiMiYmBJuhERERVJ0m0hSZdLWiBpqaTppewjku6TNFfSd3pnl5ImSLpM0p3l0fdmCf2N8aJ2ktaT1CNp07p6v5C01XDHiYiI1svq5dY61vajkl4O3ClpDvBvwJuAx4HrgcWl7teA023fLGlr4GrgjU2M8aJ2tt8o6YfAIcDZkvYAemw/LOl7Qx2nfGCYDjBmkwlDOwIREdGvJN3WOl7SIWX71cBRwDzbjwJImgW8oew/ENhBUm/bTSRtbPvxQcZo2A64BDgFOBs4ojwfqH6/bM8EZgKMnfh6DxJPREQ0KUm3RSRNpZbgpth+StJcYBn9zyrXK3WfHuJQDdtJug14naQJwHuBzw9Sf4jDRkTESOWabuuMBx4rCXd74C3AOGA/SZtJWh84tK7+T4GP9T6R1Ow9dRu2s21gNvAV4GflPr0jGSciIlosSbd1fgKsL2kJ8DlgPrAc+A/gduBa4B5gRal/PNApaYmke4DjmhxnoHaXAB9i1anlkYwTEREtptoEKdYUSRvZfqLMdGcDZ9me3e64mtXZ2emurq52hxERsdaQtMB2Z6N9memueadKWgTcDTwAXN7meCIiok2ykGoNs31Cs3UlHQN8ok/xLbY/2tqoIiKiHZJ0RxHbZ1P7yk9ERLwE5fRyRERERZJ0IyIiKpKkGxERUZEk3YiIiIok6UZERFQkSTciIqIiSboREREVyfd0Y0Ddy1fQMWNOu8OItUjPaQe1O4SIUSsz3YiIiIok6UZERFRkWElX0nGSPly2p0n6y5EGUvo5Y6T9rOk+R0pSh6QPDlKnU9LXh9jvqZKa/p3niIio3rCu6do+s+7pNGp30HmwFQGtAzqADwLf66+C7S4g99OLiHiJaWqmK+nD5SboiyWd3zurknQY0AlcKGmRpIMkza5r9zZJPxig32Mk3SdpHrBXXfkESZdJurM89pK0nqQeSZvW1fuFpK0a1W8w1mskXVdex3WSti7l50g6U9JNJZZ3lfJpki6XdKWkByR9TNKnJd0lab6kzUu9bSX9RNKC0sf2df1+XdKtku4vxwrgNGCfcrw+1c9xmSrpqrJ9qqSzJM0t/RxfV+8kScskXQtsV1feX0w/rDtD8Q+SLuxn/OmSuiR1rXxqRX//fBERMUSDJl1JOwInAQfY3pW6W8/ZvpTajOxI25OAHwFvlDShVDmGfu6aI2ki8O/Uku3bgB3qdn8NON32m4FDge/afgH4IXBIab8H0GP74Ub1Gwx5BnCe7V2AC4H607cdwH7AQcCZkjYs5TtRm5XuDnwBeMr2bsBtwIdLnZnAx21PBk4AvlXX70Rgb+Bd1JItwAzgJtuTbJ/e6Ng0sD3wjhLHZyRtIGkycASwG/A+4M119fuLaTpwiqR9gH8GPt5oMNszbXfa7hwzbnyTIUZExGCaOb18AHCp7d8D2H5UUsOKti3pfOBDks4GprAqOfW1BzDX9u8AJF0CvKHsOxDYoW6cTSRtDFwCnEItkR9Rng9Uv94UaskJ4HzgS3X7vl+S+s8l3U8tyQHcYPtx4HFJK4ArS3k3sIukjYA9gVl1Y4+t6/fy0u89krbq5zg0Y47tZ4FnJf0W2ArYB5ht+ykASVeUv/3GZPthSacANwCH2H50BDFFRMQQNZN0BXgIfZ5NLTk9A8yy/fwAdfvrdz1giu2nVwtEug14XZlJvxf4/CD1B4rT/WzXP3+2ruyFuucvUDt26wF/KLP8RurbDxjMIOr7Wcmqf7dGx2+wmHYGHgFGvPgtIiKGpplrutcBfytpC4Dea5l1Hgf+PKu0/SC1RVUnA+cM0O/twFRJW0jaADi8bt9PgY/1PpE0qfRtYDbwFeBnth8ZqH4ft1KbHQMcCdxct+/wcs14W+C1wLIB4v4z238EHpB0eBlXknYdpNlqx2sEbgQOkfTyMqt/92AxSdod+Gtqp6RPkLRNC+KIiIgmDZp0bS+ldj1znqTF1BJevXOoXQddJOnlpexC4Ne27xmg34eAU6ldH70WWFi3+3igsyx6ugc4rm7fJcCHWHVqebD69XWOkbQEOIq6a9PUkuw84MfAcbaf6S/uBo4EPlKOzVLg4EHqLwGeV21RWsOFVM2wvZDaMVgEXAbcNFBMksYC3wGOLR+M/hk4S4OcDoiIiNZRbfLY4k5r3429y/b/tLzzFpN0DnBVWRQWfXR2drqrK99eioholqQFtjsb7Wv5by9LWgA8SW0mFREREUXLk275mspqJN3O6qt6AY6y3d3q8YfK9rR2jS3pHcAX+xQ/YPuQdsQTERFrViV3GbK9RxXjrG1sXw1c3e44IiKiGrnhQUREREWSdCMiIiqSpBsREVGRJN2IiIiKJOlGRERUJEk3IiKiIkm6ERERFanke7qx9upevoKOGXPaHUa8BPWcdlC7Q4ioXGa6ERERFUnSjYiIqEiS7ign6RxJhw2z7VRJe7air4iIGLkk3TaRVMX19KnAnoNVioiIamQhVZMkdVC77+5O5fkJwEbAo8BxwPPAPbaPkPQK4BvAztSO8am2fyhpGnAQsCHwCuCABuOotD0AeABQ3b7JwFfKuL8Hptl+SNJcajez3x3YBDgW+G2Ja6WkDwEfL93sK+nTwF8AJza6j7Ck6cB0gDGbTBj6wYqIiIaSdEduBrCN7WclbVrKTgKut31sKbtD0rVl3xRgF9uP9tPfIcB21BL2VsA9wFmSNqCWjA+2/TtJ7we+QC3BArzC9p6S9gXOsr2TpDOBJ2x/GUDSR4CJwN7A9sAVwIuSru2ZwEyAsRNf72Eel4iI6CNJd+SWABdKuhy4vJS9HXhPmQ1DbWa7ddm+ZoCEC7AvcJHtlcCDkq4v5dsBOwHX1CbDjAEeqmt3EYDtGyVtUvcBoK/Lbb8A3CNpq6ZfZUREjFiSbvOeZ/Vr4BuWvwdRS5TvAf5N0o7UTgkfantZfQeS9gCebGKsRrNLAUttT2myTX8z1Gf79BkRERXJQqrmPQy8UtIWksYC76J2/F5t+wbgRGBTatdbrwY+Xq7PImm3IYxzI3CEpDGSJgL7l/JlwARJU0qfG5QE3+v9pXxvYIXtFcDjwMbDe7kREdFqmek2yfZzkj4L3E5tgdO91E7xXiBpPLVZ4+m2/yDpc8BXgSUl8fZQS9LNmE1tEVU3cB8wr4z/p/J1n6+X8dYvYywt7R6TdCurFlIBXAlcKulgVi2kioiINpGddTJru7J6+QTbXa3uu7Oz011dLe82IuIlS9IC252N9uX0ckREREVyerlNJO0MnN+n+Fnbewy1L9tTWxJURESsUUm6bWK7G5jU7jgiIqI6Ob0cERFRkSTdiIiIiiTpRkREVCRJNyIioiJJuhERERVJ0o2IiKhIvjIUA+pevoKOGXPaHUZERGV6TjtojfWdmW5ERERFknQjIiIqkqQbERFRkSTddYSkL0j6taQn2h1LRMS6Kkn3JUQ1/f2bXgnsXmU8ERGxuqxeHoUkfRH4le1vleenAgb2BTYDNgBOtv1DSR3Aj4EbgCnAe4Ff9e3T9vzSVzPjTwemA4zZZMJIX05ERBSZ6Y5OFwPvr3v+t8DZwCG23wTsD/yXVmXQ7YDzbO9m+0UJd6hsz7TdabtzzLjxI+0uIiKKzHRHIdt3SXqlpL8EJgCPAQ8Bp0vaF3gBeBWwVWnyq96ZbEREjF5JuqPXpcBhwF9Qm/keSS0BT7b9nKQeYMNS98m2RBgREUOSpDt6XQx8B9gS2I/aKebfloS7P/CadgYXERFDl6Q7StleKmljYLnthyRdCFwpqQtYBNw7lP4kfQn4IDBO0m+A79o+dbB2O79qPF1r8CfRIiLWJUm6o5jtneu2f09tdXIjOzXR14nAiS0KLSIihiGrlyMiIiqSme5LjKTbgbF9io+y3d2OeCIiYpUk3ZcY23u0O4aIiGgsp5cjIiIqItvtjpe5Va8AAAQ4SURBVCFGMUmPA8vaHUcTtgR+3+4gmpA4WytxtlbibI3X2G74G7o5vRyDWWa7s91BDEZSV+JsncTZWomztdaWOBvJ6eWIiIiKJOlGRERUJEk3BjOz3QE0KXG2VuJsrcTZWmtLnC+ShVQREREVyUw3IiKiIkm6ERERFUnSXUdJeqekZZJ+IWlGg/1jJV1S9t8uqaNu37+W8mWS3jEa45TUIelpSYvK48w2x7mvpIWSnpd0WJ99R0v6eXkcPYrjXFl3PK9oc5yflnSPpCWSrpP0mrp9o+l4DhRnZcezyViPk9Rd4rlZ0g51+yp5zw83xqrf7yNiO4917AGMAX4JvBZ4GbAY2KFPnX8CzizbRwCXlO0dSv2xwDalnzGjMM4O4O5RdDw7gF2A84DD6so3B+4vfzcr25uNtjjLvidG0fHcHxhXtv+x7t99tB3PhnFWeTyHEOsmddvvAX5Stit5z48wxsre7yN9ZKa7btod+IXt+23/CbgYOLhPnYOBc8v2pcBbJamUX2z7WdsPAL8o/Y22OKs0aJy2e2wvAV7o0/YdwDW2H7X9GHAN8M5RGGeVmonzBttPlafzgb8q26PtePYXZ9WaifWPdU9fAfSusq3qPT+SGNcaSbrrplcBv657/ptS1rCO7eeBFcAWTbYdDXECbCPpLknzJO2zhmJsNs410XaoRjrWhpK6JM2X9N7Whraaocb5EeDHw2w7EiOJE6o7ntBkrJI+KumXwJeA44fSts0xQnXv9xHJz0CumxrNBPt+YuyvTjNtW2UkcT4EbG37EUmTgcsl7djnk3KrjOSYjLbjOZCtbT8o6bXA9ZK6bf+yRbHVazpOSR8COoH9htq2BUYSJ1R3PKHJWG1/E/impA8CJwNHN9u2BUYSY5Xv9xHJTHfd9Bvg1XXP/wp4sL86ktYHxgOPNtm27XGWU2GPANheQO1a0RvaGOeaaDtUIxrL9oPl7/3AXGC3VgZXp6k4JR0InAS8x/azQ2k7CuKs8ng2HWudi4He2XdVx3TYMVb8fh+Zdl9UzqP6B7UzHPdTWxTRu2Bhxz51PsrqC5S+X7Z3ZPVFFfez5hZSjSTOCb1xUVuYsRzYvF1x1tU9hxcvpHqA2qKfzcr2aIxzM2Bs2d4S+Dl9FrlU/O++G7X/WF/fp3xUHc8B4qzseA4h1tfXbb8b6CrblbznRxhjZe/3Eb/OdgeQR5v+4eFvgPvKfwgnlbLPUvs0DrAhMIvaook7gNfWtT2ptFsG/PVojBM4FFha3rgLgXe3Oc43U/sk/yTwCLC0ru2xJf5fAMeMxjiBPYHucjy7gY+0Oc5rgYeBReVxxSg9ng3jrPp4Nhnr18p7ZhFwA3UJr6r3/HBjrPr9PpJHfgYyIiKiIrmmGxERUZEk3YiIiIok6UZERFQkSTciIqIiSboREREVSdKNiIioSJJuRERERf4//ebm264/3mMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "rf_importances = pd.Series(model_rf.feature_importances_, index=X.columns)\n",
    "rf_importances.nlargest(10).plot(kind='barh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on this data one can see that var_1 has the highest impact on ad click, followed by a user_depth, city_development_index and age_level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Best model\n",
    "\n",
    "To choose the best model I have tested five different algorithms and now I will check which one as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model</th>\n",
       "      <th>F1_score</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AdaBoost</td>\n",
       "      <td>0.891</td>\n",
       "      <td>0.904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Decision Tree</td>\n",
       "      <td>0.821</td>\n",
       "      <td>0.776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.814</td>\n",
       "      <td>0.763</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.647</td>\n",
       "      <td>0.537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Linear SVC</td>\n",
       "      <td>0.647</td>\n",
       "      <td>0.537</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Model  F1_score  Accuracy\n",
       "4             AdaBoost     0.891     0.904\n",
       "2        Decision Tree     0.821     0.776\n",
       "3        Random Forest     0.814     0.763\n",
       "0  Logistic Regression     0.647     0.537\n",
       "1           Linear SVC     0.647     0.537"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models = pd.DataFrame({\n",
    "                      'Model': ['Logistic Regression', 'Linear SVC', 'Decision Tree', 'Random Forest', 'AdaBoost'],\n",
    "                      'F1_score': [0.647, 0.647, 0.821, 0.814, 0.891],\n",
    "                      'Accuracy': [0.537, 0.537, 0.776, 0.763, 0.904]})\n",
    "\n",
    "models.sort_values(by='F1_score', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From above calculations we see that the best model is AdaBoost classifier with F1 score of 0.89 and accuracy score of 90%. This model has achaived the best result both in F1 score and accuracy score and this is signalling the characteristics of a reasonably good model with comparision to others ones.\n",
    "\n",
    "Interpreting the confusion matrix of the best model:\n",
    "\n",
    "The Confusion matrix indicates that is 472+100190 correct predictions and 7042+3558 incorrect ones. One can say that the model predicted that 472 users click ad and 3558 not ad click. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predictions\n",
    "\n",
    "\n",
    "In the las step based on previous best trained model (AdaBoost) I will  make a predictions for ad click data. For this  analysis I will use a new test dataset without a marks click_ad_variable.\n",
    "\n",
    "### Loading and prepare the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>session_id</th>\n",
       "      <th>DateTime</th>\n",
       "      <th>user_id</th>\n",
       "      <th>product</th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>webpage_id</th>\n",
       "      <th>product_category_1</th>\n",
       "      <th>product_category_2</th>\n",
       "      <th>user_group_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_level</th>\n",
       "      <th>user_depth</th>\n",
       "      <th>city_development_index</th>\n",
       "      <th>var_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>411705</td>\n",
       "      <td>2017-07-08 00:00</td>\n",
       "      <td>732573</td>\n",
       "      <td>J</td>\n",
       "      <td>404347</td>\n",
       "      <td>53587</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>208263</td>\n",
       "      <td>2017-07-08 00:00</td>\n",
       "      <td>172910</td>\n",
       "      <td>I</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>3</td>\n",
       "      <td>82527.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>239450</td>\n",
       "      <td>2017-07-08 00:00</td>\n",
       "      <td>172910</td>\n",
       "      <td>I</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>4</td>\n",
       "      <td>82527.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>547761</td>\n",
       "      <td>2017-07-08 00:00</td>\n",
       "      <td>557318</td>\n",
       "      <td>G</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>5</td>\n",
       "      <td>82527.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>574275</td>\n",
       "      <td>2017-07-08 00:00</td>\n",
       "      <td>923896</td>\n",
       "      <td>H</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>5</td>\n",
       "      <td>82527.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   session_id          DateTime  user_id product  campaign_id  webpage_id  \\\n",
       "0      411705  2017-07-08 00:00   732573       J       404347       53587   \n",
       "1      208263  2017-07-08 00:00   172910       I       118601       28529   \n",
       "2      239450  2017-07-08 00:00   172910       I       118601       28529   \n",
       "3      547761  2017-07-08 00:00   557318       G       118601       28529   \n",
       "4      574275  2017-07-08 00:00   923896       H       118601       28529   \n",
       "\n",
       "   product_category_1  product_category_2  user_group_id  gender  age_level  \\\n",
       "0                   1                 NaN            5.0    Male        5.0   \n",
       "1                   3             82527.0            NaN     NaN        NaN   \n",
       "2                   4             82527.0            NaN     NaN        NaN   \n",
       "3                   5             82527.0            1.0    Male        1.0   \n",
       "4                   5             82527.0            9.0  Female        3.0   \n",
       "\n",
       "   user_depth  city_development_index  var_1  \n",
       "0         3.0                     NaN      0  \n",
       "1         NaN                     NaN      1  \n",
       "2         NaN                     NaN      1  \n",
       "3         3.0                     1.0      0  \n",
       "4         1.0                     NaN      1  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.read_csv(r'C:\\Python Scripts\\Datasets\\CTR advertisement\\Ad_Click_prediciton_test.csv')\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "session_id                    0\n",
       "DateTime                      0\n",
       "user_id                       0\n",
       "product                       0\n",
       "campaign_id                   0\n",
       "webpage_id                    0\n",
       "product_category_1            0\n",
       "product_category_2        76171\n",
       "user_group_id              5684\n",
       "gender                     5684\n",
       "age_level                  5684\n",
       "user_depth                 5684\n",
       "city_development_index    34609\n",
       "var_1                         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing data\n",
    "\n",
    "To make a prediction need to making sure the input data have same columns as it was used for training the model. Then have to preparing data the same as for training data. \n",
    "\n",
    "Deleting missing values in gender, age_level, user_group_id, user_depth variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = df2.dropna(subset=['gender','age_level', 'user_group_id', 'user_depth'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract hour, day of week from DateTime feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['DateTime'] = pd.to_datetime(df2['DateTime'], errors='coerce')\n",
    "df2['hour'] = df2['DateTime'].dt.hour\n",
    "df2['day_of_week'] = df2['DateTime'].apply(lambda x: x.dayofweek)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filling missing values and convert non numeric features to numeric features (gender and product):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[\"city_development_index\"] = df2[\"city_development_index\"].fillna('0')\n",
    "df2[\"product_category_2\"] = df2[\"product_category_2\"].fillna(\"0\")\n",
    "df2['gender'] = df2['gender'].map({'Male' : 0, 'Female' : 1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_encoder = LabelEncoder()\n",
    "df2['product'] = label_encoder.fit_transform(df2['product'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop unnecessary values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = df2.drop(['session_id','user_id', 'DateTime'],axis=1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product</th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>webpage_id</th>\n",
       "      <th>product_category_1</th>\n",
       "      <th>product_category_2</th>\n",
       "      <th>user_group_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_level</th>\n",
       "      <th>user_depth</th>\n",
       "      <th>city_development_index</th>\n",
       "      <th>var_1</th>\n",
       "      <th>hour</th>\n",
       "      <th>day_of_week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9</td>\n",
       "      <td>404347</td>\n",
       "      <td>53587</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>5</td>\n",
       "      <td>82527</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>5</td>\n",
       "      <td>82527</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>8</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>4</td>\n",
       "      <td>82527</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>4</td>\n",
       "      <td>82527</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   product  campaign_id  webpage_id  product_category_1 product_category_2  \\\n",
       "0        9       404347       53587                   1                  0   \n",
       "3        6       118601       28529                   5              82527   \n",
       "4        7       118601       28529                   5              82527   \n",
       "5        8       118601       28529                   4              82527   \n",
       "6        3       118601       28529                   4              82527   \n",
       "\n",
       "   user_group_id  gender  age_level  user_depth city_development_index  var_1  \\\n",
       "0            5.0       0        5.0         3.0                      0      0   \n",
       "3            1.0       0        1.0         3.0                      1      0   \n",
       "4            9.0       1        3.0         1.0                      0      1   \n",
       "5            1.0       0        1.0         3.0                      4      1   \n",
       "6            2.0       0        2.0         3.0                      2      1   \n",
       "\n",
       "   hour  day_of_week  \n",
       "0     0            5  \n",
       "3     0            5  \n",
       "4     0            5  \n",
       "5     0            5  \n",
       "6     0            5  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating predictions from this data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_X = np.array(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = ada_model.predict(test_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The received prediction I mark to 'click_ prediction' variable in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "test['click_ prediction'] = predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product</th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>webpage_id</th>\n",
       "      <th>product_category_1</th>\n",
       "      <th>product_category_2</th>\n",
       "      <th>user_group_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_level</th>\n",
       "      <th>user_depth</th>\n",
       "      <th>city_development_index</th>\n",
       "      <th>var_1</th>\n",
       "      <th>hour</th>\n",
       "      <th>day_of_week</th>\n",
       "      <th>click_ prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9</td>\n",
       "      <td>404347</td>\n",
       "      <td>53587</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>5</td>\n",
       "      <td>82527</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>5</td>\n",
       "      <td>82527</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>8</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>4</td>\n",
       "      <td>82527</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3</td>\n",
       "      <td>118601</td>\n",
       "      <td>28529</td>\n",
       "      <td>4</td>\n",
       "      <td>82527</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   product  campaign_id  webpage_id  product_category_1 product_category_2  \\\n",
       "0        9       404347       53587                   1                  0   \n",
       "3        6       118601       28529                   5              82527   \n",
       "4        7       118601       28529                   5              82527   \n",
       "5        8       118601       28529                   4              82527   \n",
       "6        3       118601       28529                   4              82527   \n",
       "\n",
       "   user_group_id  gender  age_level  user_depth city_development_index  var_1  \\\n",
       "0            5.0       0        5.0         3.0                      0      0   \n",
       "3            1.0       0        1.0         3.0                      1      0   \n",
       "4            9.0       1        3.0         1.0                      0      1   \n",
       "5            1.0       0        1.0         3.0                      4      1   \n",
       "6            2.0       0        2.0         3.0                      2      1   \n",
       "\n",
       "   hour  day_of_week  click_ prediction  \n",
       "0     0            5                  0  \n",
       "3     0            5                  0  \n",
       "4     0            5                  0  \n",
       "5     0            5                  0  \n",
       "6     0            5                  0  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ad click prediction distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    120366\n",
       "1      2808\n",
       "Name: click_ prediction, dtype: int64"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test['click_ prediction'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "The aimed of this project was advertisement click prediction. Firstly I've started with basic data analysis to better meet them. Then I've cleaned data and prepared to the modelling. To determined the best model I've used five different classification algorithms such as Logistic Regression, Linear SVC, Decision Tree, Random Forest and AdaBoost. Finally I've evaluated models with a F1 score, accuracy score and confusion matrix to check which model is the best. After checked all of this metrics the best classification model was AdaBoost Classifier with F1 score 0.89 and accuracy score of approximately 90%. For end the project I've made predictions on test data with the best trained model (AdaBoost). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}