logistic_regression.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Logistic Regression on Telecommunication company users data"
      ],
      "metadata": {
        "id": "tFvbam8OzcoT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we use logistic regression to create a model for a telecommunication company, to predict when its customers will leave for a competitor, so that they can take some action to retain the customers."
      ],
      "metadata": {
        "id": "jmBv6vHNzfnf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Installing and Importing necessary packages"
      ],
      "metadata": {
        "id": "FGcrgz0VKk3D"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5GNAw99qDfma"
      },
      "outputs": [],
      "source": [
        "!pip install pandas\n",
        "!pip install matplotlib\n",
        "!pip install numpy\n",
        "!pip install scikit-learn\n",
        "!pip install scipy\n",
        "!pip install requests"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import pylab as pl\n",
        "import numpy as np\n",
        "import scipy.optimize as opt\n",
        "from sklearn import preprocessing\n",
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt"
      ],
      "metadata": {
        "id": "jHzVxvmXKiq5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Downloading dataset from IBM Object Storage"
      ],
      "metadata": {
        "id": "cMZ5KleEK-cD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "A telecommunications company is concerned about the number of customers leaving their land-line business for cable competitors. They need to understand who is leaving. We have to find out who is leaving and why.\n",
        "\n",
        "We will use a telecommunications dataset for predicting customer churn. This is a historical customer dataset where each row represents one customer. Typically it is less expensive to keep customers than acquire new ones, so the focus of this analysis is to predict the customers who will stay with the company.\n",
        "\n",
        "The dataset includes information about:\n",
        "\n",
        "- Customers who left within the last month – the column is called Churn\n",
        "- Services that each customer has signed up for – phone, multiple lines, internet, online security, online backup, device protection, tech support, and streaming TV and movies\n",
        "- Customer account information – how long they had been a customer, contract, payment method, paperless billing, monthly charges, and total charges\n",
        "- Demographic info about customers – gender, age range, and if they have partners and dependents\n"
      ],
      "metadata": {
        "id": "03v7I26PLAOP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -O ChurnData.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/data/ChurnData.csv"
      ],
      "metadata": {
        "id": "NEa-fUUEaBCN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Exploring the dataset"
      ],
      "metadata": {
        "id": "ETt5Yu8CMrhX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Reading the downloaded data set and exploring it."
      ],
      "metadata": {
        "id": "dLML6WL02OGM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "churn_df = pd.read_csv(\"ChurnData.csv\")\n",
        "churn_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "id": "PIypqgop2Ed9",
        "outputId": "719d2415-b8e5-445a-d4a5-aea9a3105fd8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   tenure   age  address  income   ed  employ  equip  callcard  wireless  \\\n",
              "0    11.0  33.0      7.0   136.0  5.0     5.0    0.0       1.0       1.0   \n",
              "1    33.0  33.0     12.0    33.0  2.0     0.0    0.0       0.0       0.0   \n",
              "2    23.0  30.0      9.0    30.0  1.0     2.0    0.0       0.0       0.0   \n",
              "3    38.0  35.0      5.0    76.0  2.0    10.0    1.0       1.0       1.0   \n",
              "4     7.0  35.0     14.0    80.0  2.0    15.0    0.0       1.0       0.0   \n",
              "\n",
              "   longmon  ...  pager  internet  callwait  confer  ebill  loglong  logtoll  \\\n",
              "0     4.40  ...    1.0       0.0       1.0     1.0    0.0    1.482    3.033   \n",
              "1     9.45  ...    0.0       0.0       0.0     0.0    0.0    2.246    3.240   \n",
              "2     6.30  ...    0.0       0.0       0.0     1.0    0.0    1.841    3.240   \n",
              "3     6.05  ...    1.0       1.0       1.0     1.0    1.0    1.800    3.807   \n",
              "4     7.10  ...    0.0       0.0       1.0     1.0    0.0    1.960    3.091   \n",
              "\n",
              "   lninc  custcat  churn  \n",
              "0  4.913      4.0    1.0  \n",
              "1  3.497      1.0    1.0  \n",
              "2  3.401      3.0    0.0  \n",
              "3  4.331      4.0    0.0  \n",
              "4  4.382      3.0    0.0  \n",
              "\n",
              "[5 rows x 28 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-673882b9-3112-4641-b867-9167751abd8a\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tenure</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>income</th>\n",
              "      <th>ed</th>\n",
              "      <th>employ</th>\n",
              "      <th>equip</th>\n",
              "      <th>callcard</th>\n",
              "      <th>wireless</th>\n",
              "      <th>longmon</th>\n",
              "      <th>...</th>\n",
              "      <th>pager</th>\n",
              "      <th>internet</th>\n",
              "      <th>callwait</th>\n",
              "      <th>confer</th>\n",
              "      <th>ebill</th>\n",
              "      <th>loglong</th>\n",
              "      <th>logtoll</th>\n",
              "      <th>lninc</th>\n",
              "      <th>custcat</th>\n",
              "      <th>churn</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>11.0</td>\n",
              "      <td>33.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>136.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>4.40</td>\n",
              "      <td>...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.482</td>\n",
              "      <td>3.033</td>\n",
              "      <td>4.913</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>33.0</td>\n",
              "      <td>33.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>33.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>9.45</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.246</td>\n",
              "      <td>3.240</td>\n",
              "      <td>3.497</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6.30</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.841</td>\n",
              "      <td>3.240</td>\n",
              "      <td>3.401</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>38.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>76.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>6.05</td>\n",
              "      <td>...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.800</td>\n",
              "      <td>3.807</td>\n",
              "      <td>4.331</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.10</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.960</td>\n",
              "      <td>3.091</td>\n",
              "      <td>4.382</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 28 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-673882b9-3112-4641-b867-9167751abd8a')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-673882b9-3112-4641-b867-9167751abd8a button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-673882b9-3112-4641-b867-9167751abd8a');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-df97d30b-8eb4-4447-abad-c5557cc7bb7a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-df97d30b-8eb4-4447-abad-c5557cc7bb7a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-df97d30b-8eb4-4447-abad-c5557cc7bb7a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Preprocessing"
      ],
      "metadata": {
        "id": "1f8T7GJXrcvX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Feature Selection"
      ],
      "metadata": {
        "id": "UlSWUh7TuPil"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "churn_df = churn_df[['tenure', 'age', 'address', 'income', 'ed', 'employ', 'equip', 'callcard', 'wireless','churn']]\n",
        "churn_df['churn'] = churn_df['churn'].astype('int')\n",
        "churn_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "HwGxPLVD3xvM",
        "outputId": "fb8089e2-d718-476f-fc12-41a86faf58a7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   tenure   age  address  income   ed  employ  equip  callcard  wireless  \\\n",
              "0    11.0  33.0      7.0   136.0  5.0     5.0    0.0       1.0       1.0   \n",
              "1    33.0  33.0     12.0    33.0  2.0     0.0    0.0       0.0       0.0   \n",
              "2    23.0  30.0      9.0    30.0  1.0     2.0    0.0       0.0       0.0   \n",
              "3    38.0  35.0      5.0    76.0  2.0    10.0    1.0       1.0       1.0   \n",
              "4     7.0  35.0     14.0    80.0  2.0    15.0    0.0       1.0       0.0   \n",
              "\n",
              "   churn  \n",
              "0      1  \n",
              "1      1  \n",
              "2      0  \n",
              "3      0  \n",
              "4      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-de9e25ab-eb4a-4e4c-904b-c541dabe695f\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tenure</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>income</th>\n",
              "      <th>ed</th>\n",
              "      <th>employ</th>\n",
              "      <th>equip</th>\n",
              "      <th>callcard</th>\n",
              "      <th>wireless</th>\n",
              "      <th>churn</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>11.0</td>\n",
              "      <td>33.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>136.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>33.0</td>\n",
              "      <td>33.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>33.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>38.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>76.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-de9e25ab-eb4a-4e4c-904b-c541dabe695f')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-de9e25ab-eb4a-4e4c-904b-c541dabe695f button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-de9e25ab-eb4a-4e4c-904b-c541dabe695f');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-e375adf0-5e26-4e3e-83b1-7df65f6e79ad\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e375adf0-5e26-4e3e-83b1-7df65f6e79ad')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-e375adf0-5e26-4e3e-83b1-7df65f6e79ad button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Dataset Exploration"
      ],
      "metadata": {
        "id": "kzsD5VgEari6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "churn_df.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OMI_x0auat7r",
        "outputId": "5d21c487-9367-4d78-9042-252d33d507e9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(200, 10)"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Seperating target feature from the independant features"
      ],
      "metadata": {
        "id": "Cw2ZaxrlazKE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X = np.asarray(churn_df[['tenure', 'age', 'address', 'income', 'ed', 'employ', 'equip']])\n",
        "X[0:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PAl2_IBha4DU",
        "outputId": "1a35a8aa-0550-4e37-97f5-c403624ba06a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 11.,  33.,   7., 136.,   5.,   5.,   0.],\n",
              "       [ 33.,  33.,  12.,  33.,   2.,   0.,   0.],\n",
              "       [ 23.,  30.,   9.,  30.,   1.,   2.,   0.],\n",
              "       [ 38.,  35.,   5.,  76.,   2.,  10.,   1.],\n",
              "       [  7.,  35.,  14.,  80.,   2.,  15.,   0.]])"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y = np.asarray(churn_df['churn'])\n",
        "y [0:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l4TuT5AOa5Az",
        "outputId": "ac451f42-b61f-4fcf-fb57-1b320f8cffcb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 1, 0, 0, 0])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Normalizing the Dataset"
      ],
      "metadata": {
        "id": "nrM5Gh4Za8lB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import preprocessing\n",
        "X = preprocessing.StandardScaler().fit(X).transform(X)\n",
        "X[0:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tcN6sKgTa97o",
        "outputId": "e64998aa-4a44-4a10-a2b6-6b084ba6437b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[-1.13518441, -0.62595491, -0.4588971 ,  0.4751423 ,  1.6961288 ,\n",
              "        -0.58477841, -0.85972695],\n",
              "       [-0.11604313, -0.62595491,  0.03454064, -0.32886061, -0.6433592 ,\n",
              "        -1.14437497, -0.85972695],\n",
              "       [-0.57928917, -0.85594447, -0.261522  , -0.35227817, -1.42318853,\n",
              "        -0.92053635, -0.85972695],\n",
              "       [ 0.11557989, -0.47262854, -0.65627219,  0.00679109, -0.6433592 ,\n",
              "        -0.02518185,  1.16316   ],\n",
              "       [-1.32048283, -0.47262854,  0.23191574,  0.03801451, -0.6433592 ,\n",
              "         0.53441472, -0.85972695]])"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Dataset Train/Test Split"
      ],
      "metadata": {
        "id": "GW28Uj3B4Ei8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)\n",
        "print ('Train set:', X_train.shape,  y_train.shape)\n",
        "print ('Test set:', X_test.shape,  y_test.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qRfH0tFlsl2I",
        "outputId": "26968048-5993-4b59-9335-5391a88eeda1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train set: (160, 7) (160,)\n",
            "Test set: (40, 7) (40,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Logistic Regression Modelling"
      ],
      "metadata": {
        "id": "HIlNBShA40q8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using Scikit-learn Logistic Regression, we can use the regularization technique to solve the overfitting problem.\n",
        "\n",
        "The c parameter indicates **inverse of regularization strength** which must be a positive float. smaller values specify stronger regularization.\n",
        "\n",
        "In addition, We can use different numerical optimizers to find parameters, including ‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’ solvers."
      ],
      "metadata": {
        "id": "6EVNaweTbm7M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import confusion_matrix\n",
        "LR = LogisticRegression(C=0.01, solver='liblinear').fit(X_train,y_train)\n",
        "LR"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        },
        "id": "CpK9IVaC4orJ",
        "outputId": "fb69e442-e89a-4803-c114-76926b016334"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LogisticRegression(C=0.01, solver='liblinear')"
            ],
            "text/html": [
              "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(C=0.01, solver=&#x27;liblinear&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(C=0.01, solver=&#x27;liblinear&#x27;)</pre></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Prediction"
      ],
      "metadata": {
        "id": "NwzN2LdZcRBa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "yhat = LR.predict(X_test)\n",
        "yhat"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GgVgAV-rcSk8",
        "outputId": "fb9af18c-e5ba-449e-81f3-6e99d2d5a7a2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0,\n",
              "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0])"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "yhat_prob = LR.predict_proba(X_test)\n",
        "yhat_prob"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nX43PSpzcbOM",
        "outputId": "b85f0851-f0c0-4798-97b1-05e70ff28ead"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.54, 0.46],\n",
              "       [0.61, 0.39],\n",
              "       [0.56, 0.44],\n",
              "       [0.63, 0.37],\n",
              "       [0.56, 0.44],\n",
              "       [0.55, 0.45],\n",
              "       [0.52, 0.48],\n",
              "       [0.61, 0.39],\n",
              "       [0.41, 0.59],\n",
              "       [0.63, 0.37],\n",
              "       [0.58, 0.42],\n",
              "       [0.63, 0.37],\n",
              "       [0.48, 0.52],\n",
              "       [0.43, 0.57],\n",
              "       [0.66, 0.34],\n",
              "       [0.55, 0.45],\n",
              "       [0.52, 0.48],\n",
              "       [0.49, 0.51],\n",
              "       [0.49, 0.51],\n",
              "       [0.52, 0.48],\n",
              "       [0.62, 0.38],\n",
              "       [0.53, 0.47],\n",
              "       [0.64, 0.36],\n",
              "       [0.52, 0.48],\n",
              "       [0.51, 0.49],\n",
              "       [0.71, 0.29],\n",
              "       [0.55, 0.45],\n",
              "       [0.52, 0.48],\n",
              "       [0.52, 0.48],\n",
              "       [0.71, 0.29],\n",
              "       [0.68, 0.32],\n",
              "       [0.51, 0.49],\n",
              "       [0.42, 0.58],\n",
              "       [0.71, 0.29],\n",
              "       [0.6 , 0.4 ],\n",
              "       [0.64, 0.36],\n",
              "       [0.4 , 0.6 ],\n",
              "       [0.52, 0.48],\n",
              "       [0.66, 0.34],\n",
              "       [0.51, 0.49]])"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Evaluation"
      ],
      "metadata": {
        "id": "ZNHHViSQN35J"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We use the jaccard index for accuracy evaluation. we can define jaccard as the size of the intersection divided by the size of the union of the two label sets. If the entire set of predicted labels for a sample strictly matches with the true set of labels, then the subset accuracy is 1.0; otherwise it is 0.0."
      ],
      "metadata": {
        "id": "tIM5GjnhciIT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import jaccard_score\n",
        "jaccard_score(y_test, yhat,pos_label=0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gebahLCZvQYy",
        "outputId": "ebe2431a-9847-4fff-c341-bbf5788a083a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7058823529411765"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Confusion matrix"
      ],
      "metadata": {
        "id": "_BBUFIOoctrI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import classification_report, confusion_matrix\n",
        "import itertools\n",
        "def plot_confusion_matrix(cm, classes,\n",
        "                          normalize=False,\n",
        "                          title='Confusion matrix',\n",
        "                          cmap=plt.cm.Blues):\n",
        "    \"\"\"\n",
        "    This function prints and plots the confusion matrix.\n",
        "    Normalization can be applied by setting `normalize=True`.\n",
        "    \"\"\"\n",
        "    if normalize:\n",
        "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "        print(\"Normalized confusion matrix\")\n",
        "    else:\n",
        "        print('Confusion matrix, without normalization')\n",
        "\n",
        "    print(cm)\n",
        "\n",
        "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    plt.title(title)\n",
        "    plt.colorbar()\n",
        "    tick_marks = np.arange(len(classes))\n",
        "    plt.xticks(tick_marks, classes, rotation=45)\n",
        "    plt.yticks(tick_marks, classes)\n",
        "\n",
        "    fmt = '.2f' if normalize else 'd'\n",
        "    thresh = cm.max() / 2.\n",
        "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "        plt.text(j, i, format(cm[i, j], fmt),\n",
        "                 horizontalalignment=\"center\",\n",
        "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.ylabel('True label')\n",
        "    plt.xlabel('Predicted label')\n",
        "print(confusion_matrix(y_test, yhat, labels=[1,0]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OmoaUsYjcv3E",
        "outputId": "2d161f9a-507e-4a82-c0fc-cd1049555329"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[ 6  9]\n",
            " [ 1 24]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Compute confusion matrix\n",
        "cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])\n",
        "np.set_printoptions(precision=2)\n",
        "\n",
        "\n",
        "# Plot non-normalized confusion matrix\n",
        "plt.figure()\n",
        "plot_confusion_matrix(cnf_matrix, classes=['churn=1','churn=0'],normalize= False,  title='Confusion matrix')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 558
        },
        "id": "QPfIG-XWc1ko",
        "outputId": "ebcdcc4f-8668-4ea0-a2d3-d1bd9e4eef9a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Confusion matrix, without normalization\n",
            "[[ 6  9]\n",
            " [ 1 24]]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAHpCAYAAACybSeHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABArElEQVR4nO3dd3wUdf7H8fdsIIWQBEILkVBDCQKBgxMREDiQYgOReiABAXuhCQI/qkIUVIoiFpQIioIiKHjiQeiC3oFixRyBICBVEEJNQjK/PzCrayi7ZMPsbF5PH/N4sDOz3/ksN7d+/Hy+31nDNE1TAAAAFnJYHQAAAAAJCQAAsBwJCQAAsBwJCQAAsBwJCQAAsBwJCQAAsBwJCQAAsBwJCQAAsBwJCQAAsBwJCeBnduzYobZt2yoiIkKGYWjp0qVeHX/37t0yDENJSUleHdcfVK5cWX379rU6DMCWSEiAArBz507df//9qlq1qoKDgxUeHq6mTZtqxowZOnv2bIFeOyEhQd99950mTZqk+fPnq1GjRgV6PX/0448/avz48dq9e7fVoQCFhsFv2QDe9cknn6hr164KCgpSnz59VKdOHWVmZmrjxo1avHix+vbtq9dee61Arn327FkVK1ZMo0eP1tNPP10g1zBNUxkZGSpatKgCAgIK5BpW++CDD9S1a1etWbNGLVu2dPt9GRkZcjgcKlq0aMEFB/ipIlYHAPiTtLQ09ejRQ5UqVdLq1atVvnx557GHH35Yqamp+uSTTwrs+keOHJEklShRosCuYRiGgoODC2x8uzFNU+fOnVNISIiCgoKsDgewLVo2gBdNmTJFp06d0htvvOGSjOSKjY3V448/7nx9/vx5PfXUU6pWrZqCgoJUuXJljRo1ShkZGS7vq1y5sm6//XZt3LhRN9xwg4KDg1W1alXNmzfPec748eNVqVIlSdITTzwhwzBUuXJlSVLfvn2df/6z8ePHyzAMl30rV65Us2bNVKJECRUvXlw1a9bUqFGjnMcvNYdk9erVat68uUJDQ1WiRAl17NhR27dvv+j1UlNT1bdvX5UoUUIRERHq16+fzpw5c+m/2N+1bNlSderU0bfffqsWLVqoWLFiio2N1QcffCBJWrdunRo3bqyQkBDVrFlTq1atcnn/zz//rIceekg1a9ZUSEiISpUqpa5du7q0ZpKSktS1a1dJUqtWrWQYhgzD0Nq1ayX98b/FZ599pkaNGikkJESvvvqq81juHBLTNNWqVSuVKVNGhw8fdo6fmZmpunXrqlq1ajp9+vQVPzNQWJCQAF60bNkyVa1aVTfddJNb5w8YMEBjx47V3/72N02bNk0tWrRQYmKievTokefc1NRUdenSRbfccouef/55lSxZUn379tUPP/wgSercubOmTZsmSerZs6fmz5+v6dOnexT/Dz/8oNtvv10ZGRmaOHGinn/+ed155536/PPPL/u+VatWqV27djp8+LDGjx+vIUOGaNOmTWratOlF52F069ZNJ0+eVGJiorp166akpCRNmDDBrRh/++033X777WrcuLGmTJmioKAg9ejRQwsXLlSPHj1066236plnntHp06fVpUsXnTx50vne//73v9q0aZN69OihmTNn6oEHHlBycrJatmzpTIhuvvlmPfbYY5KkUaNGaf78+Zo/f77i4uKc46SkpKhnz5665ZZbNGPGDNWvXz9PnIZh6M0339S5c+f0wAMPOPePGzdOP/zwg+bOnavQ0FC3PjNQKJgAvOLEiROmJLNjx45unb9t2zZTkjlgwACX/cOGDTMlmatXr3buq1SpkinJXL9+vXPf4cOHzaCgIHPo0KHOfWlpaaYkc+rUqS5jJiQkmJUqVcoTw7hx48w/fw1MmzbNlGQeOXLkknHnXmPu3LnOffXr1zfLli1rHj161Lnvm2++MR0Oh9mnT58817v33ntdxrzrrrvMUqVKXfKauVq0aGFKMhcsWODc99NPP5mSTIfDYX7xxRfO/Z999lmeOM+cOZNnzM2bN5uSzHnz5jn3vf/++6Ykc82aNXnOz/3fYsWKFRc9lpCQ4LLv1VdfNSWZb7/9tvnFF1+YAQEB5qBBg674WYHChgoJ4CXp6emSpLCwMLfO/9e//iVJGjJkiMv+oUOHSlKeuSa1a9dW8+bNna/LlCmjmjVrateuXVcd81/lzj356KOPlJOT49Z7Dhw4oG3btqlv376KjIx07q9Xr55uueUW5+f8sz9XDCSpefPmOnr0qPPv8HKKFy/uUkGqWbOmSpQoobi4ODVu3Ni5P/fPf/77CQkJcf45KytLR48eVWxsrEqUKKGvvvrKjU97QZUqVdSuXTu3zr3vvvvUrl07Pfroo7rnnntUrVo1TZ482e1rAYUFCQngJeHh4ZLk0iK4nJ9//lkOh0OxsbEu+6OiolSiRAn9/PPPLvsrVqyYZ4ySJUvqt99+u8qI8+revbuaNm2qAQMGqFy5curRo4cWLVp02eQkN86aNWvmORYXF6dff/01z1yJv36WkiVLSpJbn6VChQp55r1EREQoJiYmz76/jnn27FmNHTtWMTExCgoKUunSpVWmTBkdP35cJ06cuOK1c1WpUsXtcyXpjTfe0JkzZ7Rjxw4lJSW5JEYALiAhAbwkPDxc0dHR+v777z1631//5Xopl1pia7qxcv9S18jOznZ5HRISovXr12vVqlW655579O2336p79+665ZZb8pybH/n5LJd6rztjPvroo5o0aZK6deumRYsW6d///rdWrlypUqVKuV0RkuRxQrF27VrnROXvvvvOo/cChQUJCeBFt99+u3bu3KnNmzdf8dxKlSopJydHO3bscNl/6NAhHT9+3LlixhtKliyp48eP59n/1yqMJDkcDrVu3VovvPCCfvzxR02aNEmrV6/WmjVrLjp2bpwpKSl5jv30008qXbq0z0ze/OCDD5SQkKDnn3/eOUG4WbNmef5u3E0S3XHgwAE9+uijatu2rW6//XYNGzbson/vQGFHQgJ40fDhwxUaGqoBAwbo0KFDeY7v3LlTM2bMkCTdeuutkpRnJcwLL7wgSbrtttu8Fle1atV04sQJffvtt859Bw4c0JIlS1zOO3bsWJ735q4g+etS5Fzly5dX/fr19dZbb7n8i/3777/Xv//9b+fn9AUBAQF5qjAvvvhinupPbgJ1sSTOUwMHDlROTo7eeOMNvfbaaypSpIj69+/vVjUIKEx4MBrgRdWqVdOCBQvUvXt3xcXFuTypddOmTXr//fedz6mIj49XQkKCXnvtNR0/flwtWrTQf/7zH7311lvq1KmTWrVq5bW4evTooREjRuiuu+7SY489pjNnzmj27NmqUaOGy2TOiRMnav369brttttUqVIlHT58WC+//LIqVKigZs2aXXL8qVOnqkOHDmrSpIn69++vs2fP6sUXX1RERITGjx/vtc+RX7fffrvmz5+viIgI1a5dW5s3b9aqVatUqlQpl/Pq16+vgIAAPfvsszpx4oSCgoL0j3/8Q2XLlvXoenPnztUnn3yipKQkVahQQdKFBKh3796aPXu2HnroIa99NsDuSEgAL7vzzjv17bffaurUqfroo480e/ZsBQUFqV69enr++ec1cOBA57lz5sxR1apVlZSUpCVLligqKkojR47UuHHjvBpTqVKltGTJEg0ZMkTDhw9XlSpVlJiYqB07drgkJHfeead2796tN998U7/++qtKly6tFi1aaMKECc5JohfTpk0brVixQuPGjdPYsWNVtGhRtWjRQs8++6zHE0AL0owZMxQQEKB33nlH586dU9OmTZ3PUPmzqKgovfLKK0pMTFT//v2VnZ2tNWvWeJSQ7Nu3T4MHD9Ydd9yhhIQE5/5evXpp8eLFGj58uDp06OBTfz+AlfgtGwAAYDnmkAAAAMuRkAAAAMuRkAAAAMuRkAAAAMuRkAAAAMuRkAAAAMvxHBIfkZOTo/379yssLMyrj60GABQM0zR18uRJRUdHy+G4Nv99f+7cOWVmZnplrMDAQAUHB3tlLG8gIfER+/fvz/NrpQAA37d3717nk3gL0rlz5xQSVko6f8Yr40VFRSktLc1nkhISEh8RFhYmSfpk048KLR5mcTSAd6X+dtLqEACvO3v6lB699Qbn93dBy8zMlM6fUdD1/aSAwPwNlp2pgz/MVWZmJgkJXOW2aUKLh6l4WLjF0QDeVSzL6giAgnPN2+wBgTLymZD44iPaSUgAALATQ1J+kyAfnKpIQgIAgJ0YjgtbfsfwMSQkAADYiWF4oULieyUS30uRAABAoUOFBAAAO6FlAwAALEfLBgAAoGBQIQEAwFa80LLxwXoECQkAAHZCywYAAKBgUCEBAMBOWGUDAAAs56ctGxISAADsxE8rJL4XEQAAKHSokAAAYCe0bAAAgOVo2QAAABQMKiQAANiJYXihQkLLBgAA5IfDuLDldwwfQ8sGAABYjgoJAAB24qeTWklIAACwE5b9AgAAy/lphcT3IgIAAIUOFRIAAOyElg0AALAcLRsAAICCQYUEAAA7oWUDAAAsR8sGAACgYFAhAQDATmjZAAAA63mhZeODDRISEgAA7MRPKyS+lyIBAIBChwoJAAB2YhheWGXjexUSEhIAAOyEZb8AAAAFgwoJAAB24qeTWklIAACwE1o2AAAABYMKCQAAdkLLBgAAWM5PWzYkJAAA2ImfVkh8L0UCAACFDhUSAABsxDAMGX5YISEhAQDARvw1IaFlAwAALEeFBAAAOzF+3/I7ho8hIQEAwEZo2QAAABQQKiQAANiIv1ZISEgAALAREhIAAGA5f01ImEMCAAAsR4UEAAA7YdkvAACwGi0bAABQ6CQmJurvf/+7wsLCVLZsWXXq1EkpKSku55w7d04PP/ywSpUqpeLFi+vuu+/WoUOHPLoOCQkAADZiGH9USa5+c/9669at08MPP6wvvvhCK1euVFZWltq2bavTp087zxk8eLCWLVum999/X+vWrdP+/fvVuXNnjz4XLRsAAGzEkBdaNh5MIlmxYoXL66SkJJUtW1Zbt27VzTffrBMnTuiNN97QggUL9I9//EOSNHfuXMXFxemLL77QjTfe6NZ1qJAAAFBIpaenu2wZGRlXfM+JEyckSZGRkZKkrVu3KisrS23atHGeU6tWLVWsWFGbN292OxYSEgAAbCT/7Zo/KiwxMTGKiIhwbomJiZe9dk5OjgYNGqSmTZuqTp06kqSDBw8qMDBQJUqUcDm3XLlyOnjwoNufi5YNAAB24sVlv3v37lV4eLhzd1BQ0GXf9vDDD+v777/Xxo0b8xlAXiQkAADYiReW/Zq/vz88PNwlIbmcRx55RMuXL9f69etVoUIF5/6oqChlZmbq+PHjLlWSQ4cOKSoqyu2YaNkAAIBLMk1TjzzyiJYsWaLVq1erSpUqLscbNmyookWLKjk52bkvJSVFe/bsUZMmTdy+DhUSAABsxBsPRvPk/Q8//LAWLFigjz76SGFhYc55IREREQoJCVFERIT69++vIUOGKDIyUuHh4Xr00UfVpEkTt1fYSCQkAADYyrVOSGbPni1Jatmypcv+uXPnqm/fvpKkadOmyeFw6O6771ZGRobatWunl19+2aOYSEgAAMAlmaZ5xXOCg4M1a9YszZo166qvQ0ICAICd8ON6AADAate6ZXOtsMoGAABYjgoJAAA24q8VEhISAABshIQEAABYzl8TEuaQAAAAy1EhAQDATlj2CwAArEbLBgAAoIBQIQEAwEb8tUJCQgIAgI34a0JCywaFyuGD+zVm0EC1blBZTWuVU/f2TfTjt19ZHRaQb2dPn9K858brsdtuVMJNsRrXr5N2/rDN6rAAt1EhQaGRfuI39e/STo2aNNeMuYtVslQp7U3bqfCIElaHBuTb6089ob07/6cHn5qukmXKaeO/lmjyg//U1A+SFVm2vNXhwZtYZQPY21uvTFe58tdp3NSXnfuui6lsXUCAl2SeO6v/rP5UQ59/Q3F/u1GS1OX+Ifpq/Sqt+mC+uj003OII4U20bACbW7/qU8XVa6ARD/XRLY2q6Z+3NdOSd5OsDgvIt+zsbOVkZ6toUJDL/sCgYKVs+69FUaGg5CYk+d18jU8mJLt375ZhGNq2bZvVocCP/LJntxa//YYqVqmmF9/6UF169ddzE0Zo+eIFVocG5EtIaHFVr9dQS+bM0G9HDionO1sb//Whdny3Vcd/PWx1eIBbfDIhsZv169frjjvuUHR0tAzD0NKlS60OCReRY+aoVp14PfzEONW6Pl6d/9lPnXokaPE7b1odGpBvD02cLtM09XD7v6tPk2pa8d6buqldRxkGX/P+xpAXKiQ+OImkUN2pmZmZBTLu6dOnFR8fr1mzZhXI+PCO0mWiVCW2psu+KrE1dHD/PosiArynXExljX39A725MUUvfvKlnp63XNnns1T2uopWhwYvo2VTAHJycjRlyhTFxsYqKChIFStW1KRJk5zHd+3apVatWqlYsWKKj4/X5s2bncfGjx+v+vXru4w3ffp0Va5c2fm6b9++6tSpkyZNmqTo6GjVrFnT2Q768MMPLzm2pzp06KCnn35ad91111WPgYIX36ixft6V6rLv57SdKn9djEURAd4XHFJMJcuU06n04/p283o1bNnW6pAAt1iakIwcOVLPPPOMxowZox9//FELFixQuXLlnMdHjx6tYcOGadu2bapRo4Z69uyp8+fPe3SN5ORkpaSkaOXKlVq+fLlbY+/Zs0fFixe/7DZ58uR8ffaMjAylp6e7bChY/7z3IX237b96c9Zz2rt7p1Z89L6WvJukrvcMtDo0IN++2bRW32xao8O/7NF3X6zXpPu7K7pyNbW4o5vVocHbDC9tPsayZb8nT57UjBkz9NJLLykhIUGSVK1aNTVr1ky7d++WJA0bNky33XabJGnChAm6/vrrlZqaqlq1arl9ndDQUM2ZM0eBgYGS5NbY0dHRV5xQGxkZ6cGnzSsxMVETJkzI1xjwzPXxDfXcK+/opakTNGfmFEXHVNLQMYnq0IkvbNjf2VMn9d5Lz+jY4YMqHl5Cf2/dQd0fGq4iRYtaHRq8zF+X/VqWkGzfvl0ZGRlq3br1Jc+pV6+e88/ly194sM/hw4c9Skjq1q3rTEbcHbtIkSKKjY11+xpXY+TIkRoyZIjzdXp6umJiaB0UtOat26t56/ZWhwF43Y1t79CNbe+wOgzgqlnWsgkJCbniOUX/lNnnZnM5OTmSJIfDIdM0Xc7PysrKM0ZoaKjHY1+Llk1QUJDCw8NdNgAArsRfJ7VaViGpXr26QkJClJycrAEDBnj8/jJlyujgwYMyTdP5F+ut55Zci5YNAABXwzAubPkdw9dYlpAEBwdrxIgRGj58uAIDA9W0aVMdOXJEP/zww2XbOLlatmypI0eOaMqUKerSpYtWrFihTz/91CuVBk9bNqdOnVJq6h+rN9LS0rRt2zZFRkaqYkWW3AEAvOdCQpLfOSReCsaLLF1lM2bMGA0dOlRjx45VXFycunfvrsOH3XuqYFxcnF5++WXNmjVL8fHx+s9//qNhw4YVcMQXt2XLFjVo0EANGjSQJA0ZMkQNGjTQ2LFjLYkHAAC7Mcy/TsSAJdLT0xUREaG13+5V8TDmk8C//O8Yy9rhf86cOqkBLWrrxIkT12QeYO6/J6o+9oECgi4+P9Jd2RmntWtml2sWuzv4tV8AAGzEX5f9FqpHxwMAAN9EhQQAABthlQ0AALCcw2HI4chfRmHm8/0FgZYNAACwHBUSAABshJYNAACwnL+usiEhAQDARvy1QsIcEgAAYDkqJAAA2AgtGwAAYDl/TUho2QAAAMtRIQEAwEb8dVIrCQkAADZiyAstG/leRkLLBgAAWI4KCQAANkLLBgAAWM5fV9mQkAAAYCP+WiFhDgkAALAcFRIAAGyElg0AALAcLRsAAIACQoUEAAAboWUDAACs54WWjQ8+qJWWDQAAsB4VEgAAbISWDQAAsJy/rrIhIQEAwEb8tULCHBIAAGA5KiQAANgILRsAAGA5WjYAAAAFhAoJAAA24q8VEhISAABsxF/nkNCyAQAAlqNCAgCAjdCyAQAAlvPXlg0JCQAANuKvFRLmkAAAAMtRIQEAwEYMeaFl45VIvIuEBAAAG3EYhhz5zEjy+/6CQMsGAABc1vr163XHHXcoOjpahmFo6dKlLsf79u3rnNuSu7Vv396ja5CQAABgI7mrbPK7eeL06dOKj4/XrFmzLnlO+/btdeDAAef27rvvenQNWjYAANiIFatsOnTooA4dOlz2nKCgIEVFRV11TFRIAAAopNLT0122jIyMqx5r7dq1Klu2rGrWrKkHH3xQR48e9ej9JCQAANiIw/DOJkkxMTGKiIhwbomJiVcVU/v27TVv3jwlJyfr2Wef1bp169ShQwdlZ2e7PQYtGwAA7MTwwoPNfn/73r17FR4e7twdFBR0VcP16NHD+ee6deuqXr16qlatmtauXavWrVu7NQYVEgAAbMSbk1rDw8NdtqtNSP6qatWqKl26tFJTU91+DwkJAADwqn379uno0aMqX7682++hZQMAgI0Yv/+T3zE8cerUKZdqR1pamrZt26bIyEhFRkZqwoQJuvvuuxUVFaWdO3dq+PDhio2NVbt27dy+BgkJAAA28udJqfkZwxNbtmxRq1atnK+HDBkiSUpISNDs2bP17bff6q233tLx48cVHR2ttm3b6qmnnvKoBURCAgAALqtly5YyTfOSxz/77LN8X4OEBAAAG7HiwWjXglsJyccff+z2gHfeeedVBwMAAC7vah79frExfI1bCUmnTp3cGswwDI8eggIAACC5mZDk5OQUdBwAAMANDsOQI58ljvy+vyDkaw7JuXPnFBwc7K1YAADAFfhry8bjB6NlZ2frqaee0nXXXafixYtr165dkqQxY8bojTfe8HqAAADgD7mTWvO7+RqPE5JJkyYpKSlJU6ZMUWBgoHN/nTp1NGfOHK8GBwAACgePE5J58+bptddeU69evRQQEODcHx8fr59++smrwQEAAFfe/C0bX+LxHJJffvlFsbGxefbn5OQoKyvLK0EBAICL89dJrR5XSGrXrq0NGzbk2f/BBx+oQYMGXgkKAAAULh5XSMaOHauEhAT98ssvysnJ0YcffqiUlBTNmzdPy5cvL4gYAQDA74zft/yO4Ws8rpB07NhRy5Yt06pVqxQaGqqxY8dq+/btWrZsmW655ZaCiBEAAPzOX1fZXNVzSJo3b66VK1d6OxYAAFBIXfWD0bZs2aLt27dLujCvpGHDhl4LCgAAXJzDuLDldwxf43FCsm/fPvXs2VOff/65SpQoIUk6fvy4brrpJr333nuqUKGCt2MEAAC/89df+/V4DsmAAQOUlZWl7du369ixYzp27Ji2b9+unJwcDRgwoCBiBAAAf+JvzyCRrqJCsm7dOm3atEk1a9Z07qtZs6ZefPFFNW/e3KvBAQCAwsHjhCQmJuaiD0DLzs5WdHS0V4ICAAAXR8vmd1OnTtWjjz6qLVu2OPdt2bJFjz/+uJ577jmvBgcAAFzlTmrN7+Zr3KqQlCxZ0iWbOn36tBo3bqwiRS68/fz58ypSpIjuvfdederUqUACBQAA/suthGT69OkFHAYAAHCHv7Zs3EpIEhISCjoOAADgBn99dPxVPxhNks6dO6fMzEyXfeHh4fkKCAAAFD4eJySnT5/WiBEjtGjRIh09ejTP8ezsbK8EBgAA8nIYhhz5bLnk9/0FweNVNsOHD9fq1as1e/ZsBQUFac6cOZowYYKio6M1b968gogRAAD8Lr8PRfPVh6N5XCFZtmyZ5s2bp5YtW6pfv35q3ry5YmNjValSJb3zzjvq1atXQcQJAADkv5NaPa6QHDt2TFWrVpV0Yb7IsWPHJEnNmjXT+vXrvRsdAAAoFDxOSKpWraq0tDRJUq1atbRo0SJJFyonuT+2BwAACoa/tmw8Tkj69eunb775RpL05JNPatasWQoODtbgwYP1xBNPeD1AAADwh9xJrfndfI3Hc0gGDx7s/HObNm30008/aevWrYqNjVW9evW8GhwAACgc8vUcEkmqVKmSKlWq5I1YAADAFXij5eKDBRL3EpKZM2e6PeBjjz121cEAAIDL89dVNm4lJNOmTXNrMMMwSEjyKe66cJ52C7/T7K5RVocAeJ2ZnXnlk+A2txKS3FU1AADAWg5dxYqUi4zha/I9hwQAAFw7hbplAwAAfINhSA4/nNTqi1UbAABQyFAhAQDARhxeqJDk9/0FgYQEAAAb8dc5JFfVstmwYYN69+6tJk2a6JdffpEkzZ8/Xxs3bvRqcAAAoHDwOCFZvHix2rVrp5CQEH399dfKyMiQJJ04cUKTJ0/2eoAAAOAPuS2b/G6+xuOE5Omnn9Yrr7yi119/XUWLFnXub9q0qb766iuvBgcAAFzxa7+/S0lJ0c0335xnf0REhI4fP+6NmAAAQCHjcUISFRWl1NTUPPs3btyoqlWreiUoAABwcQ7D8MrmazxOSAYOHKjHH39cX375pQzD0P79+/XOO+9o2LBhevDBBwsiRgAA8DuHlzZf4/Gy3yeffFI5OTlq3bq1zpw5o5tvvllBQUEaNmyYHn300YKIEQAA/M4bc0B8sEDieUJiGIZGjx6tJ554QqmpqTp16pRq166t4sWLF0R8AACgELjqB6MFBgaqdu3a3owFAABcgUP5nwPikO+VSDxOSFq1anXZJ7ytXr06XwEBAIBLo2Xzu/r167u8zsrK0rZt2/T9998rISHBW3EBAIBCxOOEZNq0aRfdP378eJ06dSrfAQEAgEvz1x/X89rKn969e+vNN9/01nAAAOAiDCP/zyLxxZaN1xKSzZs3Kzg42FvDAQCAQsTjlk3nzp1dXpumqQMHDmjLli0aM2aM1wIDAAB5Man1dxERES6vHQ6HatasqYkTJ6pt27ZeCwwAAOTlr3NIPEpIsrOz1a9fP9WtW1clS5YsqJgAAMAlGL//k98xfI1Hc0gCAgLUtm1bftUXAAB4lceTWuvUqaNdu3YVRCwAAOAKcls2+d18jccJydNPP61hw4Zp+fLlOnDggNLT0102AABQcPw1IXF7DsnEiRM1dOhQ3XrrrZKkO++80+UR8qZpyjAMZWdnez9KAADg19xOSCZMmKAHHnhAa9asKch4AADAZRiGcdnflHN3DF/jdkJimqYkqUWLFgUWDAAAuDx/Xfbr0RwSX8yoAACA/Xn0HJIaNWpcMSk5duxYvgICAACXxpNadWEeyV+f1AoAAK6d3B/Iy+8YvsajhKRHjx4qW7ZsQcUCAACuoNDPIWH+CAAAKCger7IBAAAW8sIcEh/8KRv3KyQ5OTm0awAAsJhDhlc2T6xfv1533HGHoqOjZRiGli5d6nLcNE2NHTtW5cuXV0hIiNq0aaMdO3Z4+LkAAAAu4/Tp04qPj9esWbMuenzKlCmaOXOmXnnlFX355ZcKDQ1Vu3btdO7cObev4dGkVgAAYC0rlv126NBBHTp0uOgx0zQ1ffp0/d///Z86duwoSZo3b57KlSunpUuXqkePHm5dgwoJAAA24s0f1/vrD+RmZGR4HE9aWpoOHjyoNm3aOPdFRESocePG2rx5s/ufy+MrAwAAvxATE6OIiAjnlpiY6PEYBw8elCSVK1fOZX+5cuWcx9xBywYAABvx5oPR9u7dq/DwcOf+oKCgfI2bH1RIAACwkdw5JPndJCk8PNxlu5qEJCoqSpJ06NAhl/2HDh1yHnMHCQkAADbikOGsklz15sUHkVSpUkVRUVFKTk527ktPT9eXX36pJk2auD0OLRsAAHBZp06dUmpqqvN1Wlqatm3bpsjISFWsWFGDBg3S008/rerVq6tKlSoaM2aMoqOj1alTJ7evQUICAICNWLHsd8uWLWrVqpXz9ZAhQyRJCQkJSkpK0vDhw3X69Gndd999On78uJo1a6YVK1YoODjY7WuQkAAAYCMO5X++hafvb9my5WV/QsYwDE2cOFETJ068ZjEBAAB4HRUSAABsxDAMGfns2eT3/QWBhAQAABsxlP8f6/W9dISWDQAA8AFUSAAAsBFvPqnVl5CQAABgM76XTuQfCQkAADZixXNIrgXmkAAAAMtRIQEAwEZY9gsAACxnxZNarwVfjAkAABQyVEgAALARWjYAAMByPKkVAACggFAhAQDARmjZAAAAy/nrKhsSEgAAbMRfKyS+mCQBAIBChgoJAAA24q+rbEhIAACwEX5cDwAAoIBQIQEAwEYcMuTIZ9Mlv+8vCCQkAADYCC0bAACAAkKFBAAAGzF+/ye/Y/gaEhIAAGzEX1s2JCQAANiI4YVJrb5YIWEOCQAAsBwVEgAAbISWDQAAsJy/JiS0bAAAgOWokAAAYCMs+wUAAJZzGBe2/I7ha2jZAAAAy1EhAQDARmjZAAAAy7HKBvADGzes192d7lCVitEKKWro44+WWh0S4LFh97bVxref0OGNz+nn5EQtemGgqlcqe8nzl770oM5+/ZLuaFnvGkaJgmLojyrJ1f/je0hIUKicPn1adevFa/rMWVaHAly15n+L1SsL16tFn+d0+4MvqUiRAC2f/YiKBQfmOffRXq1kmhYECXiIlg0KlXbtO6hd+w5WhwHkS8dHXnZ5fd+4t7V39TNqUDtGn3+107m/Xo3r9Pg9/1DTXlO0e1XitQ4TBcRfV9mQkACAzYUXD5Yk/XbijHNfSHBRJSX21aBnFunQ0ZNWhYYC4K+TWn2yZbN7924ZhqFt27ZZHQoA+DTDMDR1WBdt+nqnftx5wLl/ytC79cU3aVq+9jsLowPc55MJiR3NmjVLlStXVnBwsBo3bqz//Oc/VocEoBCYPrKbro8trz5PznXuu61FXbW8oYaemPqBhZGhoOSussnv5msKVUKSmZlZIOMuXLhQQ4YM0bhx4/TVV18pPj5e7dq10+HDhwvkegAgSdNGdNWtzeuo3cCZ+uXwcef+ln+voaoVSuvg+qk6+d8ZOvnfGZKkd58boM9ef9yiaOEthpc2X2NpQpKTk6MpU6YoNjZWQUFBqlixoiZNmuQ8vmvXLrVq1UrFihVTfHy8Nm/e7Dw2fvx41a9f32W86dOnq3Llys7Xffv2VadOnTRp0iRFR0erZs2aznbQhx9+eMmxPfXCCy9o4MCB6tevn2rXrq1XXnlFxYoV05tvvnnVYwLA5Uwb0VV3/iNe7e+fqZ/3H3U59tzcf+vv3RLVuMczzk2Shj+/WPeNe9uKcIErsnRS68iRI/X6669r2rRpatasmQ4cOKCffvrJeXz06NF67rnnVL16dY0ePVo9e/ZUamqqihRxP+zk5GSFh4dr5cqVLvsvN/aePXtUu3bty447atQojRo1SpmZmdq6datGjhzpPOZwONSmTZvLJjkZGRnKyMhwvk5PT3f7M+HqnTp1SjtTU52vd6el6Ztt21QyMlIVK1a0MDLAfdNHdlP3Do3UdfBrOnX6nMqVCpMknTh1TucysnTo6MmLTmTde+C3PMkL7MchQ4589lwcPlgjsSwhOXnypGbMmKGXXnpJCQkJkqRq1aqpWbNm2r17tyRp2LBhuu222yRJEyZM0PXXX6/U1FTVqlXL7euEhoZqzpw5Cgy8sD7fnbGjo6OvOKE2MjJSkvTrr78qOztb5cqVczlerlw5l+TqrxITEzVhwgS3Pwe846utW9SuTSvn6xFPDJEk9b4nQa+/mWRRVIBn7u92syRp5ZxBLvsHjp2vt5d9aUFEuJa80XLxvXTEwoRk+/btysjIUOvWrS95Tr16fzxVsHz58pKkw4cPe5SQ1K1b15mMuDt2kSJFFBsb6/Y1rsbIkSM1ZMgQ5+v09HTFxMQU6DUh3dyipc5m8ZQo2FtIg0euyXvgo/w0I7FsDklISMgVzylatKjzz8bv5amcnBxJF9oi5l8eP5iVlZVnjNDQUI/H3rNnj4oXL37ZbfLkyZKk0qVLKyAgQIcOHXIZ/9ChQ4qKirrkZwsKClJ4eLjLBgBAYWVZhaR69eoKCQlRcnKyBgwY4PH7y5Qpo4MHD8o0TWdC4a3nlnjSsgkMDFTDhg2VnJysTp06SbqQ2CQnJ+uRR/gvEgCAd/nrg9EsS0iCg4M1YsQIDR8+XIGBgWratKmOHDmiH3744bJtnFwtW7bUkSNHNGXKFHXp0kUrVqzQp59+6pVKg6ctmyFDhighIUGNGjXSDTfcoOnTp+v06dPq169fvmMBAMCFN54j4nv5iLWrbMaMGaMiRYpo7Nix2r9/v8qXL68HHnjArffGxcXp5Zdf1uTJk/XUU0/p7rvv1rBhw/Taa68VcNR5de/eXUeOHNHYsWN18OBB1a9fXytWrMgz0RUAAFycYf51IgYskZ6eroiICB06eoL5JPA7Jf9O+xL+x8zOVMZ3r+vEiWvzvZ3774nV2/aoeFj+rnfqZLr+Ub/iNYvdHfy4HgAAdsIqGwAAgIJBhQQAABthlQ0AALCcN36t1xd/7ZeEBAAAG/HTKSTMIQEAANajQgIAgJ34aYmEhAQAABvx10mttGwAAIDlqJAAAGAjrLIBAACW89MpJLRsAACA9aiQAABgJ35aIiEhAQDARvx1lQ0JCQAANuKvk1qZQwIAACxHhQQAABvx0ykkVEgAALAVw0ubm8aPHy/DMFy2WrVqee3j5KJCAgAALuv666/XqlWrnK+LFPF++kBCAgCAjVixyqZIkSKKiorK1zWvhJYNAAA2krvKJr+bJKWnp7tsGRkZF73mjh07FB0drapVq6pXr17as2eP1z8XCQkAAIVUTEyMIiIinFtiYmKecxo3bqykpCStWLFCs2fPVlpampo3b66TJ096NRZaNgAA2Ig3V9ns3btX4eHhzv1BQUF5zu3QoYPzz/Xq1VPjxo1VqVIlLVq0SP37989nJH8gIQEAwE68mJGEh4e7JCTuKFGihGrUqKHU1NR8BuGKlg0AADZieOmfq3Xq1Cnt3LlT5cuX9+KnIiEBAACXMWzYMK1bt067d+/Wpk2bdNdddykgIEA9e/b06nVo2QAAYCPX+rds9u3bp549e+ro0aMqU6aMmjVrpi+++EJlypTJXxB/QUICAICNXOtHx7/33nv5vJp7aNkAAADLUSEBAMBO/PTX9UhIAACwESseHX8t0LIBAACWo0ICAICdeGGVjQ8WSEhIAACwEz+dQkJCAgCArfhpRsIcEgAAYDkqJAAA2Ii/rrIhIQEAwEau9aPjrxVaNgAAwHJUSAAAsBE/ndNKQgIAgK34aUZCywYAAFiOCgkAADbCKhsAAGA5Q15YZeOVSLyLhAQAABvx0ykkzCEBAADWo0ICAICN+OuD0UhIAACwFf9s2tCyAQAAlqNCAgCAjdCyAQAAlvPPhg0tGwAA4AOokAAAYCO0bAAAgOV4dDwAALCen04iYQ4JAACwHBUSAABsxE8LJCQkAADYib9OaqVlAwAALEeFBAAAG2GVDQAAsJ6fTiKhZQMAACxHhQQAABvx0wIJCQkAAHbir6tsSEgAALCV/E9q9cUaCXNIAACA5aiQAABgI/7asqFCAgAALEdCAgAALEfLBgAAG/HXlg0JCQAANuKvj46nZQMAACxHhQQAABuhZQMAACzHo+MBAID1/DQjYQ4JAACwHBUSAABsxF9X2ZCQAABgI/46qZWWDQAAsBwVEgAAbMRP57SSkAAAYCt+mpHQsgEAAJajQgIAgI2wygYFyjRNSdLJ9HSLIwG8z8zOtDoEwOty7+vc7+9r5eTJ9Hyvkjl50vf+XUNC4iNOnjwpSYqtEmNxJAAAT5w8eVIREREFfp3AwEBFRUWpupf+PREVFaXAwECvjOUNhnmtUztcVE5Ojvbv36+wsDAZvrhA3M+kp6crJiZGe/fuVXh4uNXhAF7BfX1tmaapkydPKjo6Wg7HtZmSee7cOWVmeqfiGBgYqODgYK+M5Q1USHyEw+FQhQoVrA6j0AkPD+eLG36H+/rauRaVkT8LDg72qSTCm1hlAwAALEdCAgAALEdCgkIpKChI48aNU1BQkNWhAF7DfQ07Y1IrAACwHBUSAABgORISAABgORISAABgORISAABgORISAABgORISAABgORISACjk/vr0B54GASvwWzaAG3JycuRwOHT+/HkVKcL/beA/cu/tY8eO6eTJk8rJyVGVKlWcx03T5Ac/cU1QIQGuIPcLOyUlRePHj1daWprVIQFekXtvf/fdd2rSpIluu+02Va9eXX369NHSpUslSYZhUDHBNcF/6gFX4HA4tHPnTrVo0UKHDx/WwYMHNWHCBF133XVWhwbki8Ph0IEDB9ShQwd1795d99xzj3bt2qXXXntNTz/9tH755Rc9/PDDVEhwTZCQAFdw9uxZTZs2TW3btlXHjh31z3/+U1lZWZo8eTJJCWzvxx9/VGRkpP7v//5PJUuWVP369VWjRg29+uqrevnllxUcHKz+/ftbHSYKARIS4Aqys7P197//XYGBgbr77ru1Zs0atWrVSpJISmB7gYGB2rdvn1JSUnTjjTdKkurUqaPHHntMmZmZevvtt9WoUSPFx8dbHCn8HXNIgCsoXry4OnXqpJ49e0qSbrrpJiUnJ+u9997TyJEj9csvv0i60I//8ccfrQwV8Fh0dLSio6P12WefKSMjw7m/evXquu+++5SSkqItW7ZYGCEKCxIS4DJyJ/NFREQ4X+fk5KhZs2ZKTk7WwoULNWrUKKWlpWnw4MEaPHiwTp48aWXIgFty7+1q1appwIABmjhxohYuXOgygbVhw4Zq1KiRVq9ebVWYKERo2QCXcLElvoZhyDAMZWdnq1mzZlq9erXatWun1atX69ChQ/ryyy8VFhZmUcSAe3Lv7dxVNoMGDdLBgwc1cOBAnTlzRt26dVNkZKSkC4lLxYoVLY4YhYFhsp4LyOPPX9gPPvig+vTpo6ZNm7qck/tl3qlTJ23YsEHr1q1TnTp1LIoYcM9f7+3evXurefPmkqTx48fr2Wef1Z133qmyZcsqKytL7777rjZv3qzatWtbHDn8HRUS4C/+/IXdqFEjhYeHq3Hjxhc9d+TIkfr444/19ddfk4zA513s3m7SpInz+Pjx41W3bl2tXbtW27ZtU+XKlbVhwwaSEVwTVEiAP/nzF/YNN9ygyMhIffLJJypatKimTJmiBg0a6JZbbnGeu2zZMsXGxqpu3boWRw5c3pXu7Xr16ql9+/aS/phfkpWVpcDAQCvDRiFCQgL87q9f2CVKlNCnn36qokWLqm/fvkpOTtbnn39OPx224+m9zePiYQVW2QC68KyR3C/sxo0bu3xh33vvvfr888/12WefkYzAdq7m3iYZgRVISABJAQEBysnJUfXq1RUZGenyhb1u3Tp9/PHH9NFhS9zbsAsSEuB3O3bs0E033aRly5apaNGi6t+/v9atW6fly5crLi7O6vCAq8a9DTtgDgkKpSv1yLt27apvvvlGH330EV/YsBXubdgVy35R6OR+Ya9Zs0aff/65Tp8+raFDh6p06dKSpD179ujIkSP68MMP+cKGrXBvw86okKBQWr58ubp27aobb7xRqampkqS3335bzZs3l8PhUEZGhoKCgiyOEvAc9zbsijkkKHROnz6tFStW6OWXX1ZycrL+97//6YYbblDXrl21du1amabJFzZsiXsbdkZCgkLl66+/VlxcnL777jtVr15dDodDISEhWrx4sZo3b67u3bs7v7gBO+Heht2RkMDv5X4Br1+/XpGRkbr++uu1YcMG/fbbb5Iu/CaNJC1evFj/+Mc/1Lp1a23YsMGyeAF3cW/Dn5CQwO/lTvJr2bKlvv76ay1YsEAtWrTQoEGDtH37djkcDucX+8KFC3XPPfcoKirK4qiBK+Pehj9hUiv83q5duzR//nxFRERo0KBBkqTjx4/r9ttv1+HDh53LH3lcNuyGexv+hAoJ/NoPP/yg/v37KykpSRUqVJB04Xc9SpQooeXLl6ts2bLq0qWLvv/+e76wYSvc2/A3JCTwO38u+gUGBiomJkbHjh3T+vXrJUlFihRRdna2SpQooU8++USGYahv377KzMy0KmTALdzb8Ge0bOCXNm/erKioKFWpUkU///yzEhMTtWHDBg0cONBZ2s7OzlZAQIBOnDih3377TZUrV7Y0ZsAd3NvwVzypFX7n2LFjGj16tPbt26eVK1eqUqVKGj58uEzT1MKFC2UYhh5//HEFBAQoOztbERERioiIsDps4Iq4t+HPaNnA70RGRurJJ59U1apV1alTJ6Wlpalq1aoaPny46tWrpw8++EDPPPOMpAu/hArYBfc2/BkJCWwvt+t4/vx55762bdvqiSeeUKlSpdS5c2elpaWpWrVqGjFihCpWrKjVq1c7n9UA+CrubRQmzCGBX/jyyy81fvx4vfPOO4qMjHTuT05O1rhx45SRkaElS5aoQoUK2r17t4KDg3keA2yBexuFBRUS+IXU1FQdOHBA/fr1c/mvw9atW6tHjx7aunWrWrVqpd27d6ty5cp8YcM2uLdRWJCQwHYuVtTr3r27RowYoSNHjqh37946evSo81jt2rXVqlUr3Xjjjc5HaQO+iHsbhRktG9hO7lMnv/32W/36668KCgpS06ZNlZOTo0WLFunFF19UWFiY3n77bZUuXVrjxo3TkSNH9OyzzyosLMzq8IFL4t5GoWYCNjB58mRz1KhRZnZ2tmmapvnhhx+aoaGhZo0aNUzDMMzhw4ebmZmZZnZ2tvn++++bTZo0MYsUKWI2bdrULFasmPndd99Z/AmAi+PeBi7gOSSwheLFi2v06NEKDQ3VgAEDNHnyZL300ktq3ry5tm7dqt69e+v48eOaMWOGunTpoqZNm2rhwoWSpLlz56p69eoWfwLg4ri3gQtISODzTNPUo48+qpCQEN1///1KT09XvXr1dPfddyssLEzVqlVTeHi4OnbsKEmaOnWqypcv73xqJeCruLeBPzCHBD4t9/Y0DEOmaerdd99Vnz59VLp0aW3dulXXXXedcnJy5HA49Nlnn+nuu+/WXXfdpZkzZ6pkyZIWRw9cGvc24IpVNvB5hmFo1apVGjp0qOLj47VgwQIdOXJEr7/+uvML2zRNtWvXTu+++67+/e9/KyMjw+qwgSvi3gb+QMsGPs0wDH344Yfq3bu3Ro0apbNnz6pbt246ceKEHnjgARUtWlQjR450fnHfcccd2rVrl0JDQ60OHbgs7m3AFQkJfNr//vc/DRs2TM8//7wefPBB5/6BAwfKNE09+OCDcjgcGjFihByOCwU/vrBhB9zbgCsSEvi0PXv2qGjRorr11lud+3JL2ffdd59CQ0N1zz33qGjRoho2bJiFkQKe4d4GXJGQwKedOnVKZ8+edb7OycmRYRiSpLVr16phw4ZauHCh6tSpY1WIwFXh3gZcMakVPi0+Pl6//vqrXnvtNUmSw+Fwfml/9NFHWrBggTp37qy4uDgrwwQ8xr0NuKJCAp9WpUoVvfTSS3rggQeUlZWlPn36KCAgQElJSUpKStLmzZsVEBBgdZiAx7i3AVc8hwQ+LycnR4sXL9b999+v0NBQBQcHKyAgQO+++64aNGhgdXjAVePeBv5AQgLb2L9/v37++WcZhqEqVaqoXLlyVocEeAX3NkBCAgAAfACTWgEAgOVISAAAgOVISAAAgOVISAAAgOVISAAAgOVISAAAgOVISAAAgOVISAAAgOVISACob9++6tSpk/N1y5YtNWjQoGsex9q1a2UYho4fP37JcwzD0NKlS90ec/z48apfv36+4tq9e7cMw9C2bdvyNQ6ASyMhAXxU3759ZRiGDMNQYGCgYmNjNXHiRJ0/f77Ar/3hhx/qqaeecutcd5IIALgSfu0X8GHt27fX3LlzlZGRoX/96196+OGHVbRoUY0cOTLPuZmZmQoMDPTKdSMjI70yDgC4iwoJ4MOCgoIUFRWlSpUq6cEHH1SbNm308ccfS/qjzTJp0iRFR0erZs2akqS9e/eqW7duKlGihCIjI9WxY0ft3r3bOWZ2draGDBmiEiVKqFSpUho+fLj++pNWf23ZZGRkaMSIEYqJiVFQUJBiY2P1xhtvaPfu3WrVqpUkqWTJkjIMQ3379pV04ZdsExMTVaVKFYWEhCg+Pl4ffPCBy3X+9a9/qUaNGgoJCVGrVq1c4nTXiBEjVKNGDRUrVkxVq1bVmDFjlJWVlee8V199VTExMSpWrJi6deumEydOuByfM2eO4uLiFBwcrFq1aunll1/2OBYAV4+EBLCRkJAQZWZmOl8nJycrJSVFK1eu1PLly5WVlaV27dopLCxMGzZs0Oeff67ixYurffv2zvc9//zzSkpK0ptvvqmNGzfq2LFjWrJkyWWv26dPH7377ruaOXOmtm/frldffVXFixdXTEyMFi9eLElKSUnRgQMHNGPGDElSYmKi5s2bp1deeUU//PCDBg8erN69e2vdunWSLiROnTt31h133KFt27ZpwIABevLJJz3+OwkLC1NSUpJ+/PFHzZgxQ6+//rqmTZvmck5qaqoWLVqkZcuWacWKFfr666/10EMPOY+/8847Gjt2rCZNmqTt27dr8uTJGjNmjN566y2P4wFwlUwAPikhIcHs2LGjaZqmmZOTY65cudIMCgoyhw0b5jxerlw5MyMjw/me+fPnmzVr1jRzcnKc+zIyMsyQkBDzs88+M03TNMuXL29OmTLFeTwrK8usUKGC81qmaZotWrQwH3/8cdM0TTMlJcWUZK5cufKica5Zs8aUZP7222/OfefOnTOLFStmbtq0yeXc/v37mz179jRN0zRHjhxp1q5d2+X4iBEj8oz1V5LMJUuWXPL41KlTzYYNGzpfjxs3zgwICDD37dvn3Pfpp5+aDofDPHDggGmaplmtWjVzwYIFLuM89dRTZpMmTUzTNM20tDRTkvn1119f8roA8oc5JIAPW758uYoXL66srCzl5OTon//8p8aPH+88XrduXZd5I998841SU1MVFhbmMs65c+e0c+dOnThxQgcOHFDjxo2dx4oUKaJGjRrladvk2rZtmwICAtSiRQu3405NTdWZM2d0yy23uOzPzMxUgwYNJEnbt293iUOSmjRp4vY1ci1cuFAzZ87Uzp07derUKZ0/f17h4eEu51SsWFHXXXedy3VycnKUkpKisLAw7dy5U/3799fAgQOd55w/f14REREexwPg6pCQAD6sVatWmj17tgIDAxUdHa0iRVz/LxsaGury+tSpU2rYsKHeeeedPGOVKVPmqmIICQnx+D2nTp2SJH3yyScuiYB0YV6Mt2zevFm9evXShAkT1K5dO0VEROi9997T888/73Gsr7/+ep4EKSAgwGuxArg8EhLAh4WGhio2Ntbt8//2t79p4cKFKlu2bJ4qQa7y5cvryy+/1M033yzpQiVg69at+tvf/nbR8+vWraucnBytW7dObdq0yXM8t0KTnZ3t3Fe7dm0FBQVpz549l6ysxMXFOSfo5vriiy+u/CH/ZNOmTapUqZJGjx7t3Pfzzz/nOW/Pnj3av3+/oqOjnddxOByqWbOmypUrp+joaO3atUu9evXy6PoAvIdJrYAf6dWrl0qXLq2OHTtqw4YNSktL09q1a/XYY49p3759kqTHH39czzzzjJYuXaqffvpJDz300GWfIVK5cmUlJCTo3nvv1dKlS51jLlq0SJJUqVIlGYah5cuX68iRIzp16pTCwsI0bNgwDR48WG+99ZZ27typr776Si+++KJzougDDzygHTt26IknnlBKSooWLFigpKQkjz5v9erVtWfPHr333nvauXOnZs6cedEJusHBwUpISNA333yjDRs26LHHHlO3bt0UFRUlSZowYYISExM1c+ZM/e9//9N3332nuXPn6oUXXvAoHgBXj4QE8CPFihXT+vXrVbFiRXXu3FlxcXHq37+/zp0756yYDB06VPfcc48SEhLUpEkThYWF6a677rrsuLNnz1aXLl300EMPqVatWho4cKBOnz4tSbruuus0YcIEPfnkkypXrpweeeQRSdJTTz2lMWPGKDExUXFxcWrfvr0++eQTValSRdKFeR2LFy/W0qVLFR8fr1deeUWTJ0/26PPeeeedGjx4sB555BHVr19fmzZt0pgxY/KcFxsbq86dO+vWW29V27ZtVa9ePZdlvQMGDNCcOXM0d+5c1a1bVy1atFBSUpIzVgAFzzAvNZMNAADgGqFCAgAALEdCAgAALEdCAgAALEdCAgAALEdCAgAALEdCAgAALEdCAgAALEdCAgAALEdCAgAALEdCAgAALEdCAgAALPf/LUev8fM1Hn8AAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first row is for customers whose actual churn value in the test set is 1.\n",
        "As We can see, out of 40 customers, the churn value of 15 of them is 1.\n",
        "Out of these 15 cases, the classifier correctly predicted 6 of them as 1, and 9 of them as 0.\n",
        "\n",
        "This means, for 6 customers, the actual churn value was 1 in test set and classifier also correctly predicted those as 1. However, while the actual label of 9 customers was 1, the classifier predicted those as 0, which is not very good. We can consider it as the error of the model for first row.\n",
        "\n",
        "What about the customers with churn value 0? Lets look at the second row.\n",
        "It looks like  there were 25 customers whom their churn value were 0.\n",
        "\n",
        "\n",
        "The classifier correctly predicted 24 of them as 0, and one of them wrongly as 1. So, it has done a good job in predicting the customers with churn value 0."
      ],
      "metadata": {
        "id": "bNHM0wH5frZn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print (classification_report(y_test, yhat))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k8aXbwv_dLgT",
        "outputId": "336e00cc-6021-4e3a-a42a-a90e8b692cae"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.73      0.96      0.83        25\n",
            "           1       0.86      0.40      0.55        15\n",
            "\n",
            "    accuracy                           0.75        40\n",
            "   macro avg       0.79      0.68      0.69        40\n",
            "weighted avg       0.78      0.75      0.72        40\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Based on the count of each section, we can calculate precision and recall of each label:\n",
        "\n",
        "\n",
        "- __Precision__ : The measure of the accuracy provided that a class label has been predicted. It is defined by: precision = TP / (TP + FP)\n",
        "\n",
        "- __Recall__ : The true positive rate. It is defined as: Recall =  TP / (TP + FN)\n",
        "\n",
        "    \n",
        "So, we can calculate the precision and recall of each class.\n",
        "\n",
        "__F1 score:__\n",
        "Now we are in the position to calculate the F1 scores for each label based on the precision and recall of that label.\n",
        "\n",
        "The F1 score is the harmonic average of the precision and recall, where an F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.\n",
        "\n",
        "\n",
        "Finally, we can tell the average accuracy for this classifier is the average of the F1-score for both labels, which is 0.72 in our case.\n"
      ],
      "metadata": {
        "id": "HidQszVigEdZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Log loss"
      ],
      "metadata": {
        "id": "jIQvI0VRdMzn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import log_loss\n",
        "log_loss(y_test, yhat_prob)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nLa05KtidN0L",
        "outputId": "daed82e1-ba3b-48ab-b370-c0a13acd4ee4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.6017092478101185"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    }
  ]
}