code/04-python-data-mining/03B-classification-and-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back](./03A-cluster-analysis.ipynb)\n",
    "\n",
    "---\n",
    "## `Classification and Regression`\n",
    "\n",
    "- **Classification** and **Regression** are considered as supervised learning in Machine Learning\n",
    "- Meaning, we have the known output, the output is labeled and therefore it is called supervised learning\n",
    "- There are two types of supervised learning\n",
    "  - Classification\n",
    "  - Regression\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Initial Setup`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imports of various classifiers and regressors from different libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Linear Regression is for regression\n",
    "# Logistic Regression is for classification\n",
    "from sklearn.linear_model import LinearRegression, LogisticRegression\n",
    "\n",
    "# SVM is Support Vector Machine\n",
    "# SVC is Support Vector Classifier\n",
    "# SVR is Support Vector Regressor\n",
    "from sklearn.svm import SVC, SVR\n",
    "\n",
    "# SKLearn Neighbors is another big library, that provides the KNeighborsClassifier and KNeighborsRegressor\n",
    "from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other imports and dataset creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>address</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>zipcode</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>rooms</th>\n",
       "      <th>squareFootage</th>\n",
       "      <th>lotSize</th>\n",
       "      <th>yearBuilt</th>\n",
       "      <th>lastSaleDate</th>\n",
       "      <th>lastSaleAmount</th>\n",
       "      <th>priorSaleDate</th>\n",
       "      <th>priorSaleAmount</th>\n",
       "      <th>estimated_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39525749</td>\n",
       "      <td>8171 E 84th Ave</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80022</td>\n",
       "      <td>39.84916</td>\n",
       "      <td>-104.893468</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1378</td>\n",
       "      <td>9968</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>2009-12-17</td>\n",
       "      <td>75000</td>\n",
       "      <td>2004-05-13</td>\n",
       "      <td>165700.0</td>\n",
       "      <td>239753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>184578398</td>\n",
       "      <td>10556 Wheeling St</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80022</td>\n",
       "      <td>39.88802</td>\n",
       "      <td>-104.830930</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1653</td>\n",
       "      <td>6970</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>2004-09-23</td>\n",
       "      <td>216935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>343963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>184430015</td>\n",
       "      <td>3190 Wadsworth Blvd</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80033</td>\n",
       "      <td>39.76171</td>\n",
       "      <td>-105.081070</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1882</td>\n",
       "      <td>23875</td>\n",
       "      <td>1917.0</td>\n",
       "      <td>2008-04-03</td>\n",
       "      <td>330000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>488840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>155129946</td>\n",
       "      <td>3040 Wadsworth Blvd</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80033</td>\n",
       "      <td>39.76078</td>\n",
       "      <td>-105.081060</td>\n",
       "      <td>4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2400</td>\n",
       "      <td>11500</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>2008-12-02</td>\n",
       "      <td>185000</td>\n",
       "      <td>2008-06-27</td>\n",
       "      <td>0.0</td>\n",
       "      <td>494073</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id              address    city state  zipcode  latitude  \\\n",
       "0   39525749      8171 E 84th Ave  Denver    CO    80022  39.84916   \n",
       "1  184578398    10556 Wheeling St  Denver    CO    80022  39.88802   \n",
       "2  184430015  3190 Wadsworth Blvd  Denver    CO    80033  39.76171   \n",
       "3  155129946  3040 Wadsworth Blvd  Denver    CO    80033  39.76078   \n",
       "\n",
       "    longitude  bedrooms  bathrooms  rooms  squareFootage  lotSize  yearBuilt  \\\n",
       "0 -104.893468         3        2.0      6           1378     9968     2003.0   \n",
       "1 -104.830930         2        2.0      6           1653     6970     2004.0   \n",
       "2 -105.081070         3        1.0      0           1882    23875     1917.0   \n",
       "3 -105.081060         4        3.0      0           2400    11500     1956.0   \n",
       "\n",
       "  lastSaleDate  lastSaleAmount priorSaleDate  priorSaleAmount  estimated_value  \n",
       "0   2009-12-17           75000    2004-05-13         165700.0           239753  \n",
       "1   2004-09-23          216935           NaN              NaN           343963  \n",
       "2   2008-04-03          330000           NaN              NaN           488840  \n",
       "3   2008-12-02          185000    2008-06-27              0.0           494073  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df = pd.read_csv(r'../../assets/single_family_home_values.csv')\n",
    "df.head(4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>rooms</th>\n",
       "      <th>squareFootage</th>\n",
       "      <th>lotSize</th>\n",
       "      <th>yearBuilt</th>\n",
       "      <th>priorSaleAmount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1378</td>\n",
       "      <td>9968</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>165700.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1653</td>\n",
       "      <td>6970</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1882</td>\n",
       "      <td>23875</td>\n",
       "      <td>1917.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2400</td>\n",
       "      <td>11500</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2305</td>\n",
       "      <td>5600</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   bedrooms  bathrooms  rooms  squareFootage  lotSize  yearBuilt  \\\n",
       "0         3        2.0      6           1378     9968     2003.0   \n",
       "1         2        2.0      6           1653     6970     2004.0   \n",
       "2         3        1.0      0           1882    23875     1917.0   \n",
       "3         4        3.0      0           2400    11500     1956.0   \n",
       "4         3        4.0      8           2305     5600     1998.0   \n",
       "\n",
       "   priorSaleAmount  \n",
       "0         165700.0  \n",
       "1              0.0  \n",
       "2              0.0  \n",
       "3              0.0  \n",
       "4              0.0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# X will be all the columns, without estimated_value\n",
    "X = df.drop('estimated_value', axis=1)\n",
    "X = X[['bedrooms', 'bathrooms', 'rooms', 'squareFootage',\n",
    "       'lotSize', 'yearBuilt', 'priorSaleAmount']]\n",
    "X.fillna(0, inplace=True)\n",
    "X.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have our features defined as `X`, to predict something.\n",
    "\n",
    "So, we can now define `y`. In the previous section we didn't want to have `y` because it was unsupervised learning, meaning there is no `y`, meaning **no non-outcome**.\n",
    "\n",
    "However, in `classification` and `regression` there are **non-outcomes**, therefore we want to consider `X` and `y`\n",
    "\n",
    "**NOTE::** `X` is always uppercase, meaning it is a `matrix` and `y` is lowercase, meaning it is a `vector`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         239753\n",
       "1         343963\n",
       "2         488840\n",
       "3         494073\n",
       "4         513676\n",
       "          ...   \n",
       "14995    1080081\n",
       "14996     807306\n",
       "14997    1737156\n",
       "14998    2008794\n",
       "14999    1421401\n",
       "Name: estimated_value, Length: 15000, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = df.estimated_value\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Classification versus Regression`\n",
    "\n",
    "> **Classification**\n",
    "\n",
    "\n",
    "- As the name suggests, it to classify something, meaning, the output variable is a categorical variable\n",
    "- e.g., it is going to be [red | blue], [one | two]\n",
    "- or, it can be more than two class, meaning, it can be [red | blue | yellow | orange]\n",
    "- This is considered as classification, so the target variable is categorical variable.\n",
    "\n",
    "\n",
    "> **Regression**\n",
    "\n",
    "- It is considered as predicting a numerical variable.\n",
    "- What is being tried to predict is going to be a numerical value.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Linear and Logistic Regression`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `LineraRegression`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7648477834199695"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lg = LinearRegression()\n",
    "lg.fit(X=X, y=y) # Training the model\n",
    "lg.score(X=X, y=y) # To evaluate the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a concept in `DataScience` called **over-fitting** and **under-fitting**\n",
    "- over-fitting meaning that it's going after the noise.\n",
    "- meaning, if the model sees a new data-set, it would not perform that well because it would be very very specific to this data set that we used to train\n",
    "\n",
    "So, in `sklearn`, luckily, we have a package called `model_selection` _(previously known as **cross_validation**)_ from which we can import and use `train_test_split` to partition our data-set for training and testing of 80% and 20% by default respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# By default, 80% will go to training data-set and 20% will go to test data-set\n",
    "# And it returns values in the order of X_train, X_test, y_train, y_test\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((11250, 7), (11250,))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can make use of this data-set which is split specifically for training to train our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7580178906271744"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lg = LinearRegression()\n",
    "lg.fit(X=X_train, y=y_train) # Training the model\n",
    "lg.score(X=X_test, y=y_test) # To evaluate the model / score / R-square\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for Logistic Regression, our target variable should be **categorical variable**.\n",
    "\n",
    "So, we cannot use `y` variable as is.\n",
    "\n",
    "But, what we can do is, create a `y2` variable that's going to be using the **estimated_value**, but it's going to be categorical.\n",
    "\n",
    "And we obtain that as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "low     11500\n",
       "high     3500\n",
       "Name: estimated_value_bins, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['estimated_value_bins'] = df.estimated_value.apply(lambda x: 'high' if x > 700000 else 'low')\n",
    "df.estimated_value_bins.value_counts()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we see that the estimates of `low` and `high` are not that balanced, so we'll try to recalculate it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "high    11197\n",
       "low      3803\n",
       "Name: estimated_value_bins, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['estimated_value_bins'] = df.estimated_value.apply(lambda x: 'high' if x > 400000 else 'low')\n",
    "df.estimated_value_bins.value_counts()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not much of a change, so instead of guessing the value, we can try generating a histogram and easily pick the right value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD4CAYAAAAKA1qZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAV1klEQVR4nO3df6zd9X3f8ee7hhLHDmBKcuVir7iaVw3wSvCVw8YSXQdWbhAqVBqSkRbMQuWKkSlZLS2m1dZElSXalbYCCotbMptBc+s1yYxCaEstrpJuMGJTkoshHm6MmLFnt4EYHEUsJu/98f1QH+zje+8598f5wuf5kI7O93y+3+85r3N97nnd749zHJmJJKlOPzHoAJKkwbEEJKliloAkVcwSkKSKWQKSVLEzBh1gKueff35eeOGFfa37gx/8gEWLFs1uoFlgrt6Yqzfm6s27Ndfu3bv/LjPfP+WCmdnqy+rVq7Nfjz/+eN/rziVz9cZcvTFXb96tuYBdOY33WHcHSVLFLAFJqpglIEkVswQkqWKWgCRVzBKQpIpZApJUMUtAkipmCUhSxVr/tRHSxMtHuXnTI4OOcYqto+37qgGpV24JSFLFLAFJqpglIEkVswQkqWKWgCRVzBKQpIpZApJUMUtAkirmh8X099r6oayNqwadQHr3cktAkipmCUhSxaYsgYh4T0Q8FRHfiog9EfG5Mn5eRDwWES+U6yUd69weEfsiYm9EXN0xvjoiJsq8uyIi5uZpSZKmYzpbAm8AH83MnwcuBUYj4nJgE7AzM1cCO8ttIuIiYB1wMTAK3BsRC8p93QdsAFaWy+jsPRVJUq+mLIFsHCs3zyyXBK4DtpXxbcD1Zfo6YCwz38jM/cA+YE1ELAXOzswnMjOBBzrWkSQNQDTvx1Ms1Pwlvxv4h8AfZOZnIuL7mXluxzKvZuaSiLgHeDIzHyzj9wOPAi8Cd2TmVWX8w8BnMvPaLo+3gWaLgaGhodVjY2N9Pbljx46xePHivtadS23NdeSVoxz+4aBTnGpoIa3MteKcBa38d2zr68tcvZlprrVr1+7OzOGplpvWKaKZ+SZwaUScC3wlIi6ZZPFu+/lzkvFuj7cF2AIwPDycIyMj04l5ivHxcfpddy61NdfdD+3gzon2nTW8cdXxVubaOrqolf+ObX19mas385Wrp7ODMvP7wDjNvvzDZRcP5fpIWewAsLxjtWXAwTK+rMu4JGlApnN20PvLFgARsRC4CvgO8DCwviy2HthRph8G1kXEWRGxguYA8FOZeQh4PSIuL2cF3dSxjiRpAKazjb0U2FaOC/wEsD0zvxoRTwDbI+IW4CXgBoDM3BMR24HngOPAbWV3EsCtwFZgIc1xgkdn88lIknozZQlk5reBD3YZ/x5w5WnW2Qxs7jK+C5jseIIkaR75iWFJqpglIEkVswQkqWKWgCRVzBKQpIpZApJUMUtAkipmCUhSxSwBSaqYJSBJFbMEJKliloAkVcwSkKSKWQKSVDFLQJIqZglIUsUsAUmqmCUgSRWzBCSpYpaAJFXMEpCkip0x6ADSO9XEy0e5edMjg45xiq2jiwYdQe8gbglIUsWmLIGIWB4Rj0fE8xGxJyI+VcY/GxEvR8Qz5XJNxzq3R8S+iNgbEVd3jK+OiIky766IiLl5WpKk6ZjO7qDjwMbMfDoi3gfsjojHyrzfy8zf6Vw4Ii4C1gEXAz8N/GVE/KPMfBO4D9gAPAl8DRgFHp2dpyJJ6tWUWwKZeSgzny7TrwPPAxdMssp1wFhmvpGZ+4F9wJqIWAqcnZlPZGYCDwDXz/QJSJL6F8378TQXjrgQ+DpwCfCrwM3Aa8Aumq2FVyPiHuDJzHywrHM/zV/7LwJ3ZOZVZfzDwGcy89ouj7OBZouBoaGh1WNjY309uWPHjrF48eK+1p1Lbc115JWjHP7hoFOcamgh5urBinMWtPL11dbX/bs119q1a3dn5vBUy0377KCIWAx8Cfh0Zr4WEfcBvwlkub4T+ATQbT9/TjJ+6mDmFmALwPDwcI6MjEw35tuMj4/T77pzqa257n5oB3dOtO+EsY2rjpurB1tHF7Xy9dXW133tuaZ1dlBEnElTAA9l5pcBMvNwZr6ZmT8G/hBYUxY/ACzvWH0ZcLCML+syLkkakOmcHRTA/cDzmfm7HeNLOxb7JeDZMv0wsC4izoqIFcBK4KnMPAS8HhGXl/u8CdgxS89DktSH6WzLXgF8HJiIiGfK2K8BN0bEpTS7dF4EfgUgM/dExHbgOZozi24rZwYB3ApsBRbSHCfwzCBJGqApSyAz/4ru+/O/Nsk6m4HNXcZ30RxUliS1gJ8YlqSKWQKSVDFLQJIqZglIUsUsAUmqmCUgSRWzBCSpYpaAJFXMEpCkilkCklQxS0CSKmYJSFLFLAFJqpglIEkVswQkqWKWgCRVzBKQpIpZApJUMUtAkipmCUhSxSwBSaqYJSBJFbMEJKliU5ZARCyPiMcj4vmI2BMRnyrj50XEYxHxQrle0rHO7RGxLyL2RsTVHeOrI2KizLsrImJunpYkaTqmsyVwHNiYmf8YuBy4LSIuAjYBOzNzJbCz3KbMWwdcDIwC90bEgnJf9wEbgJXlMjqLz0WS1KMpSyAzD2Xm02X6deB54ALgOmBbWWwbcH2Zvg4Yy8w3MnM/sA9YExFLgbMz84nMTOCBjnUkSQMQzfvxNBeOuBD4OnAJ8FJmntsx79XMXBIR9wBPZuaDZfx+4FHgReCOzLyqjH8Y+ExmXtvlcTbQbDEwNDS0emxsrK8nd+zYMRYvXtzXunOprbmOvHKUwz8cdIpTDS3EXD1Ycc6CVr6+2vq6f7fmWrt27e7MHJ5quTOme4cRsRj4EvDpzHxtkt353WbkJOOnDmZuAbYADA8P58jIyHRjvs34+Dj9rjuX2prr7od2cOfEtF8S82bjquPm6sHW0UWtfH219XVfe65pnR0UEWfSFMBDmfnlMny47OKhXB8p4weA5R2rLwMOlvFlXcYlSQMynbODArgfeD4zf7dj1sPA+jK9HtjRMb4uIs6KiBU0B4CfysxDwOsRcXm5z5s61pEkDcB0tmWvAD4OTETEM2Xs14A7gO0RcQvwEnADQGbuiYjtwHM0ZxbdlplvlvVuBbYCC2mOEzw6O09DktSPKUsgM/+K7vvzAa48zTqbgc1dxnfRHFSWJLWAnxiWpIq179QGSTMy8fJRbt70yKBjnGLr6KJBR1AXbglIUsUsAUmqmCUgSRWzBCSpYpaAJFXMEpCkilkCklQxS0CSKmYJSFLFLAFJqpglIEkVswQkqWKWgCRVzBKQpIpZApJUMUtAkipmCUhSxSwBSaqYJSBJFbMEJKliloAkVWzKEoiIL0TEkYh4tmPssxHxckQ8Uy7XdMy7PSL2RcTeiLi6Y3x1REyUeXdFRMz+05Ek9WI6WwJbgdEu47+XmZeWy9cAIuIiYB1wcVnn3ohYUJa/D9gArCyXbvcpSZpHU5ZAZn4deGWa93cdMJaZb2TmfmAfsCYilgJnZ+YTmZnAA8D1fWaWJM2SM2aw7icj4iZgF7AxM18FLgCe7FjmQBn7UZk+ebyriNhAs9XA0NAQ4+PjfQU8duxY3+vOpbbmGloIG1cdH3SMU5irN23N1dbXfe25+i2B+4DfBLJc3wl8Aui2nz8nGe8qM7cAWwCGh4dzZGSkr5Dj4+P0u+5camuuux/awZ0TM/m7YG5sXHXcXD1oa66to4ta+bpv6+/jfOXq6+ygzDycmW9m5o+BPwTWlFkHgOUdiy4DDpbxZV3GJUkD1FcJlH38b/kl4K0zhx4G1kXEWRGxguYA8FOZeQh4PSIuL2cF3QTsmEFuSdIsmHKbMSK+CIwA50fEAeA3gJGIuJRml86LwK8AZOaeiNgOPAccB27LzDfLXd1Kc6bRQuDRcpEkDdCUJZCZN3YZvn+S5TcDm7uM7wIu6SmdJGlO+YlhSaqYJSBJFbMEJKliloAkVcwSkKSKWQKSVDFLQJIqZglIUsUsAUmqmCUgSRWzBCSpYpaAJFXMEpCkilkCklSx9v0fdJLelSZePsrNmx4ZdIxTbB1dNOgIA+WWgCRVzBKQpIpZApJUMUtAkipmCUhSxSwBSaqYJSBJFbMEJKliU5ZARHwhIo5ExLMdY+dFxGMR8UK5XtIx7/aI2BcReyPi6o7x1RExUebdFREx+09HktSL6WwJbAVGTxrbBOzMzJXAznKbiLgIWAdcXNa5NyIWlHXuAzYAK8vl5PuUJM2zKUsgM78OvHLS8HXAtjK9Dbi+Y3wsM9/IzP3APmBNRCwFzs7MJzIzgQc61pEkDUi/3x00lJmHADLzUER8oIxfADzZsdyBMvajMn3yeFcRsYFmq4GhoSHGx8f7Cnns2LG+151Lbc01tBA2rjo+6BinMFdvzNWbtv4+zleu2f4CuW77+XOS8a4ycwuwBWB4eDhHRkb6CjM+Pk6/686ltua6+6Ed3DnRvu8U3LjquLl6YK7ebB1d1Mrfx/l6n+j37KDDZRcP5fpIGT8ALO9YbhlwsIwv6zIuSRqgfkvgYWB9mV4P7OgYXxcRZ0XECpoDwE+VXUevR8Tl5aygmzrWkSQNyJTbZhHxRWAEOD8iDgC/AdwBbI+IW4CXgBsAMnNPRGwHngOOA7dl5pvlrm6lOdNoIfBouUiSBmjKEsjMG08z68rTLL8Z2NxlfBdwSU/pJElzyk8MS1LFLAFJqpglIEkVswQkqWKWgCRVzBKQpIpZApJUMUtAkipmCUhSxSwBSaqYJSBJFbMEJKli7fsfHiow8fJRbt70yKBjnGLjqkEnkDTf3BKQpIpZApJUMUtAkipmCUhSxSwBSaqYJSBJFbMEJKlifk5AUtXa+rmdraOL5uVx3BKQpIpZApJUsRmVQES8GBETEfFMROwqY+dFxGMR8UK5XtKx/O0RsS8i9kbE1TMNL0mamdnYElibmZdm5nC5vQnYmZkrgZ3lNhFxEbAOuBgYBe6NiAWz8PiSpD7Nxe6g64BtZXobcH3H+FhmvpGZ+4F9wJo5eHxJ0jRFZva/csR+4FUggc9n5paI+H5mntuxzKuZuSQi7gGezMwHy/j9wKOZ+add7ncDsAFgaGho9djYWF/5jh07xuLFi/tady4deeUoh3846BSnGlqIuXpgrt6Yqzcrzlkwo/evtWvX7u7YQ3NaMz1F9IrMPBgRHwAei4jvTLJsdBnr2kCZuQXYAjA8PJwjIyN9hRsfH6ffdefS3Q/t4M6J9p2du3HVcXP1wFy9MVdvto4umpf3rxntDsrMg+X6CPAVmt07hyNiKUC5PlIWPwAs71h9GXBwJo8vSZqZvksgIhZFxPvemgZ+AXgWeBhYXxZbD+wo0w8D6yLirIhYAawEnur38SVJMzeTbaAh4CsR8db9/HFm/llEfBPYHhG3AC8BNwBk5p6I2A48BxwHbsvMN2eUXpI0I32XQGZ+F/j5LuPfA648zTqbgc39PqYkaXb5iWFJqpglIEkVswQkqWKWgCRVzBKQpIpZApJUMUtAkipmCUhSxSwBSaqYJSBJFbMEJKliloAkVcwSkKSKWQKSVDFLQJIqZglIUsUsAUmqmCUgSRWzBCSpYpaAJFXMEpCkilkCklQxS0CSKjbvJRARoxGxNyL2RcSm+X58SdIJ81oCEbEA+APgY8BFwI0RcdF8ZpAknTDfWwJrgH2Z+d3M/H/AGHDdPGeQJBWRmfP3YBH/EhjNzF8utz8OfCgzP3nSchuADeXmzwF7+3zI84G/63PduWSu3pirN+bqzbs1189k5vunWuiMGTxAP6LL2CktlJlbgC0zfrCIXZk5PNP7mW3m6o25emOu3tSea753Bx0AlnfcXgYcnOcMkqRivkvgm8DKiFgRET8JrAMenucMkqRiXncHZebxiPgk8OfAAuALmblnDh9yxruU5oi5emOu3pirN1XnmtcDw5KkdvETw5JUMUtAkmqWma27AF8AjgDPdoydBzwGvFCul3TMux3YR/N5gqs7xlcDE2XeXZzY/XUW8Cdl/H8BF3ass748xgvA+pNyLQceB54H9gCfakM24D3AU8C3Sq7PtSFXmbcA+Gvgq23JVOa/WO7zGWBXW7IB5wJ/CnyH5nX2Twedi+azOs90XF4DPj3oXGXev6N5zT8LfJHmd6ENuT5VMu0BPt2W11fX99vZeNOe7QvwEeAy3l4Cvw1sKtObgN8q0xfRvPmdBawA/gZYUOY9RfNLFMCjwMfK+L8B/nOZXgf8Scc/0nfL9ZIy3fkPtRS4rEy/D/jf5fEHmq3cx+IyfWZ5UVw+6Fxl/q8Cf8yJEhh4prLMi8D5J40NPBuwDfjlMv2TNKUw8Fwd+RYA/xf4mUHnAi4A9gMLy+3twM0tyHUJTQG8l+bkm78EVg461zuqBMqTuZC3l8BeYGnHm/HeMn07cHvHcn9efmhLge90jN8IfL5zmTJ9Bs2n8qJzmTLv88CNk2TcAfyLNmUrL7yngQ8NOhfN50B2Ah/lRAm04mdF9xIY9M/rbJo3tWhTrpOy/ALwP9qQi6YE/g/NG94ZwFdLvkHnugH4o455/wH494POdbrLO+mYwFBmHgIo1x8o42+9EN5yoIxdUKZPHn/bOpl5HDgK/NQk93WKiLgQ+CDNX90DzxYRCyLiGZrdaI9lZhty/T7Ni//HHfMHnektCfxFROwuX1PShmw/C/wt8F8i4q8j4o8iYlELcnVaR7PbhUHnysyXgd8BXgIOAUcz8y8GnYtmK+AjEfFTEfFe4BqaXcmDztXVO6kETud0X0Ux2VdU9LPOiQeMWAx8iWZf32ttyJaZb2bmpTR/fa+JiEsGmSsirgWOZObuSXLMa6aTbl+RmZfRfKPtbRHxkRZkO4NmN+h9mflB4Ac0uw0GnatZsfmA5y8C/22STPOWKyKW0HwB5Qrgp4FFEfGvBp0rM58Hfotmv/+f0ezqOT7oXKfzTiqBwxGxFKBcHynjp/sqigNl+uTxt60TEWcA5wCvTHJffy8izqQpgIcy88ttygaQmd8HxoHRAee6AvjFiHiR5ttiPxoRDw44U+fP6WC5PgJ8heYbbged7QBwoGzFQXOA+LIW5HrLx4CnM/NwuT3oXFcB+zPzbzPzR8CXgX/Wglxk5v2ZeVlmfqQs/0IbcnU12b6iQV449ZjAf+LtB1V+u0xfzNsPqnyXEwdVvklzgPStgyrXlPHbePtBle1l+jyafbJLymU/cF5HhgAeAH7/pKwDzQa8Hzi3TC8EvgFcO+hcHT+fEU4cExh4JmAR8L6O6f9JU5ptyPYN4OfK9GdLpoHnKsuMAf+6Ra/7D9GcffPecn/bgH876Fxl/gfK9T+gOdNrSRtydX2vHfSb/WkK4Is0+/h+RNNst9Ds79pJ06g7T/qB/zrNEfW9lKPnZXyYZv/c3wD3cOL0qvfQbNLuozn6/rMd63yijO+j4wVf5v1zmk2rb3PidLlrBp0N+Cc0p2F+u9znfyzjA/+ZdSmBgWei2ff+LU6cUvvrLcp2KbCr/Fv+d5pf5Dbkei/wPeCcjrE25PoczZvss8B/pXkjbUOubwDP0bzGrmzLz6vbxa+NkKSKvZOOCUiSZpklIEkVswQkqWKWgCRVzBKQpIpZApJUMUtAkir2/wEBEi7qbt54rgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.estimated_value.hist(bins=[100000, 200000, 300000, 400000, 500000, 600000, 700000, 800000, 900000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "high    7963\n",
       "low     7037\n",
       "Name: estimated_value_bins, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['estimated_value_bins'] = df.estimated_value.apply(lambda x: 'high' if x > 500000 else 'low')\n",
    "df.estimated_value_bins.value_counts()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see that the `estimated_value_bins` is clearly a categorical value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>address</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>zipcode</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>rooms</th>\n",
       "      <th>squareFootage</th>\n",
       "      <th>lotSize</th>\n",
       "      <th>yearBuilt</th>\n",
       "      <th>lastSaleDate</th>\n",
       "      <th>lastSaleAmount</th>\n",
       "      <th>priorSaleDate</th>\n",
       "      <th>priorSaleAmount</th>\n",
       "      <th>estimated_value</th>\n",
       "      <th>estimated_value_bins</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39525749</td>\n",
       "      <td>8171 E 84th Ave</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80022</td>\n",
       "      <td>39.849160</td>\n",
       "      <td>-104.893468</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1378</td>\n",
       "      <td>9968</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>2009-12-17</td>\n",
       "      <td>75000</td>\n",
       "      <td>2004-05-13</td>\n",
       "      <td>165700.0</td>\n",
       "      <td>239753</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>184578398</td>\n",
       "      <td>10556 Wheeling St</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80022</td>\n",
       "      <td>39.888020</td>\n",
       "      <td>-104.830930</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1653</td>\n",
       "      <td>6970</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>2004-09-23</td>\n",
       "      <td>216935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>343963</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>184430015</td>\n",
       "      <td>3190 Wadsworth Blvd</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80033</td>\n",
       "      <td>39.761710</td>\n",
       "      <td>-105.081070</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1882</td>\n",
       "      <td>23875</td>\n",
       "      <td>1917.0</td>\n",
       "      <td>2008-04-03</td>\n",
       "      <td>330000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>488840</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>155129946</td>\n",
       "      <td>3040 Wadsworth Blvd</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80033</td>\n",
       "      <td>39.760780</td>\n",
       "      <td>-105.081060</td>\n",
       "      <td>4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2400</td>\n",
       "      <td>11500</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>2008-12-02</td>\n",
       "      <td>185000</td>\n",
       "      <td>2008-06-27</td>\n",
       "      <td>0.0</td>\n",
       "      <td>494073</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>245107</td>\n",
       "      <td>5615 S Eaton St</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80123</td>\n",
       "      <td>39.616181</td>\n",
       "      <td>-105.058812</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2305</td>\n",
       "      <td>5600</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>2012-07-18</td>\n",
       "      <td>308000</td>\n",
       "      <td>2011-12-29</td>\n",
       "      <td>0.0</td>\n",
       "      <td>513676</td>\n",
       "      <td>high</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3523925</td>\n",
       "      <td>6535 W Sumac Ave</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80123</td>\n",
       "      <td>39.615136</td>\n",
       "      <td>-105.069018</td>\n",
       "      <td>3</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2051</td>\n",
       "      <td>6045</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>2006-09-12</td>\n",
       "      <td>363500</td>\n",
       "      <td>2005-05-16</td>\n",
       "      <td>339000.0</td>\n",
       "      <td>496062</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>30560679</td>\n",
       "      <td>6673 W Berry Ave</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80123</td>\n",
       "      <td>39.616350</td>\n",
       "      <td>-105.070760</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2051</td>\n",
       "      <td>6315</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>2014-12-15</td>\n",
       "      <td>420000</td>\n",
       "      <td>2006-07-07</td>\n",
       "      <td>345000.0</td>\n",
       "      <td>514953</td>\n",
       "      <td>high</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>39623928</td>\n",
       "      <td>5640 S Otis St</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80123</td>\n",
       "      <td>39.615213</td>\n",
       "      <td>-105.070617</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2051</td>\n",
       "      <td>8133</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>2004-03-15</td>\n",
       "      <td>328700</td>\n",
       "      <td>1998-04-09</td>\n",
       "      <td>225200.0</td>\n",
       "      <td>494321</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>148975825</td>\n",
       "      <td>5342 S Gray St</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80123</td>\n",
       "      <td>39.620158</td>\n",
       "      <td>-105.063094</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>7</td>\n",
       "      <td>1796</td>\n",
       "      <td>5038</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>2011-08-16</td>\n",
       "      <td>274900</td>\n",
       "      <td>2011-01-10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>496079</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>184623176</td>\n",
       "      <td>4967 S Wadsworth Blvd</td>\n",
       "      <td>Denver</td>\n",
       "      <td>CO</td>\n",
       "      <td>80123</td>\n",
       "      <td>39.626770</td>\n",
       "      <td>-105.081990</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2054</td>\n",
       "      <td>4050</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>2015-12-01</td>\n",
       "      <td>407000</td>\n",
       "      <td>2012-10-30</td>\n",
       "      <td>312000.0</td>\n",
       "      <td>424514</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id                address    city state  zipcode   latitude  \\\n",
       "0   39525749        8171 E 84th Ave  Denver    CO    80022  39.849160   \n",
       "1  184578398      10556 Wheeling St  Denver    CO    80022  39.888020   \n",
       "2  184430015    3190 Wadsworth Blvd  Denver    CO    80033  39.761710   \n",
       "3  155129946    3040 Wadsworth Blvd  Denver    CO    80033  39.760780   \n",
       "4     245107        5615 S Eaton St  Denver    CO    80123  39.616181   \n",
       "5    3523925       6535 W Sumac Ave  Denver    CO    80123  39.615136   \n",
       "6   30560679       6673 W Berry Ave  Denver    CO    80123  39.616350   \n",
       "7   39623928         5640 S Otis St  Denver    CO    80123  39.615213   \n",
       "8  148975825         5342 S Gray St  Denver    CO    80123  39.620158   \n",
       "9  184623176  4967 S Wadsworth Blvd  Denver    CO    80123  39.626770   \n",
       "\n",
       "    longitude  bedrooms  bathrooms  rooms  squareFootage  lotSize  yearBuilt  \\\n",
       "0 -104.893468         3        2.0      6           1378     9968     2003.0   \n",
       "1 -104.830930         2        2.0      6           1653     6970     2004.0   \n",
       "2 -105.081070         3        1.0      0           1882    23875     1917.0   \n",
       "3 -105.081060         4        3.0      0           2400    11500     1956.0   \n",
       "4 -105.058812         3        4.0      8           2305     5600     1998.0   \n",
       "5 -105.069018         3        5.0      7           2051     6045     1996.0   \n",
       "6 -105.070760         4        4.0      8           2051     6315     1997.0   \n",
       "7 -105.070617         3        3.0      7           2051     8133     1997.0   \n",
       "8 -105.063094         3        3.0      7           1796     5038     1999.0   \n",
       "9 -105.081990         3        3.0      0           2054     4050     2007.0   \n",
       "\n",
       "  lastSaleDate  lastSaleAmount priorSaleDate  priorSaleAmount  \\\n",
       "0   2009-12-17           75000    2004-05-13         165700.0   \n",
       "1   2004-09-23          216935           NaN              NaN   \n",
       "2   2008-04-03          330000           NaN              NaN   \n",
       "3   2008-12-02          185000    2008-06-27              0.0   \n",
       "4   2012-07-18          308000    2011-12-29              0.0   \n",
       "5   2006-09-12          363500    2005-05-16         339000.0   \n",
       "6   2014-12-15          420000    2006-07-07         345000.0   \n",
       "7   2004-03-15          328700    1998-04-09         225200.0   \n",
       "8   2011-08-16          274900    2011-01-10              0.0   \n",
       "9   2015-12-01          407000    2012-10-30         312000.0   \n",
       "\n",
       "   estimated_value estimated_value_bins  \n",
       "0           239753                  low  \n",
       "1           343963                  low  \n",
       "2           488840                  low  \n",
       "3           494073                  low  \n",
       "4           513676                 high  \n",
       "5           496062                  low  \n",
       "6           514953                 high  \n",
       "7           494321                  low  \n",
       "8           496079                  low  \n",
       "9           424514                  low  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "y2 = df.estimated_value_bins"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating the `train_test_split` data based on the new categorical value `y2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y2_train, y2_test = train_test_split(X, y2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `LogisticRegression`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log = LogisticRegression()\n",
    "log.fit(X_train, y2_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8221333333333334"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log.score(X_test, y2_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another function that is pretty useful is `.predict()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['low', 'high', 'high', ..., 'low', 'high', 'low'], dtype=object)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = log.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['low', 'high', 'high', ..., 'low', 'high', 'low'], dtype=object)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(y2_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we see the `prediction` and the actual `y2`, almost all of the values displayed are matching.\n",
    "\n",
    "A better way of testing this, again pretty much using another functionality from `sklearn` library, i.e., `confusion_matrix`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1633,  386],\n",
       "       [ 281, 1450]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "confusion_matrix(y_true=y2_test, y_pred=y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to understand this result, just remember that the diagonal (0x0 and 1x1) is the right classification / y_prediction\n",
    "\n",
    "Where as\n",
    "\n",
    "`0x1` is the **type1 error** -> predicted high, but it was low\n",
    "\n",
    "`1x0` is the **type2 error** -> predicted low, but it was high"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Support Vector Machine`\n",
    "\n",
    "As in the image below, this is a sample plot from a `SVM`.\n",
    "\n",
    "Here, we have two features `X` and `Y` which are the red dots and the blue dots.\n",
    "\n",
    "SVM classifier (SVC), will classify the data points had created a buffer in between that has pushed away the two class from each\n",
    "other as much as possible, this is what SVM is doing under the hood.\n",
    "\n",
    "As any other model, there is classifier and a regressor. The difference is in the target variable.\n",
    "\n",
    "In SVC, the target variable is a categorical variable and what you are trying to predict is a categorical variable, then we can use a classifier (SVC). Else, we need to use a regressor, which will predict a numerical value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../../assets/svm.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(url='../../assets/svm.png')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Support Vector Regressor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "svr = SVR()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR()"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svr.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.05067683153191793"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the score, we see that it is bad\n",
    "\n",
    "Let's try and see SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Support Vector Classifier`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "svc = SVC()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7045333333333333"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svc.fit(X_train, y2_train)\n",
    "svc.score(X_test, y2_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this is still worse that **LogisticRegression**\n",
    "\n",
    "Let's create the confusion_matrix and see the result, to see the type1 & type2 errors as well as the correct classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "y2_pred = svc.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1069,  950],\n",
       "       [ 158, 1573]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y2_test, y2_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we can see that these (SVC and SVR) are not the best models for this problem!\n",
    "\n",
    "Usually this happens because for us to find the optimal model and optimal features we have go back a few times and do a few iterations\n",
    "\n",
    "And sometimes more than few iterations to really understand and then pick the best model and set of features.\n",
    "\n",
    "Also, sometime it might be combination of the model and features, so definitely it takes a lot of trial and error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `KNN - K Nearest Neighbors`\n",
    "\n",
    "Consider you have a lot of data points, x's and o's\n",
    "\n",
    "```\n",
    "| xxxxxxxxx\n",
    "|   ?\n",
    "| xxxxxx\n",
    "|\n",
    "|\n",
    "|       ooooooooooo\n",
    "|           ?\n",
    "|     oooooooo\n",
    "|________________________________\n",
    "```\n",
    "\n",
    "And if we have some unknown data point in the cluster of `x` and another data point in the cluster of `o`\n",
    "\n",
    "It is likely that the unknowns belong to the cluster\n",
    "\n",
    "```\n",
    "| xxxxxxxxx\n",
    "|   unknown=x\n",
    "| xxxxxx\n",
    "|\n",
    "|\n",
    "|       ooooooooooo\n",
    "|           unknown=o\n",
    "|     oooooooo\n",
    "|________________________________\n",
    "```\n",
    "\n",
    "So, `KNN` says that, given all these data points, this is supervised learning, we have the labels, we know it is `x`, we know it is `o`, the actual labels.\n",
    "\n",
    "And then given all these non-labeled data points, meaning we have some unknown data points that we are interested in classifying for instance.\n",
    "\n",
    "So, by looking at the neighbors on an unknown data point, either it's likely five closest neighbors and if their labels are mostly `x` then we can conclude that this unknown data point is going to be classified as `x`. However, if the five closest neighbors are classified as `o`, then this unknown data point has to be classified as `o`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `KNeighborsRegressor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "knnr = KNeighborsRegressor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsRegressor()"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knnr.fit(X_train, y_train) # y_train is the numerical values, and being used with a regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.16905077587980943"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knnr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not great either"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `KNeighborsClassifier`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "knnc = KNeighborsClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier()"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knnc.fit(X_train, y2_train) # y2_train is the categorical (high & low) data points and is used with the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8218666666666666"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knnc.score(X_test, y2_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks better, so now to score is good enough, we can take a look at the **confusion_matrix**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "kncc_pred = knnc.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1621,  398],\n",
       "       [ 270, 1461]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y2_test, kncc_pred)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like almost similar to the **LogisticRegressor** based on the type1 and type2 error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Conclusion`\n",
    "\n",
    "So, by comparing the **confusion_matrix**, we can decide to proceed with the model of our choice, by seeing if **type1 error** is going to cost us more or if **type2 error** is going to cost us more.\n",
    "\n",
    "Also, **train_split_test** is really important, to split our dataset to train and test datasets\n",
    "\n",
    "---\n",
    "[next](./03C-association-and-correlation.ipynb)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "0b4484ae063bf9ef0d5115063857dd0e382a2c73ed49a11a000ca56af03302b6"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}