Assignment-1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 2,
  "metadata": {
    "colab": {
      "name": "ML_Assignment-1.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "Jd1xOHPfOjtf",
        "LXth-jqQXM4p",
        "P_40AndKPez7",
        "q5JxBGeVYa3u"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3.9.0 64-bit"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.0",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "interpreter": {
      "hash": "1fa55b210fe6dc681b5d14c145252b97ca241050c73c1e9f8cb6c42409351a44"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **Import Datasets**"
      ],
      "metadata": {
        "id": "FZsxcMhBYleR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Fetching UCI Diabetes dataset\r\n",
        "\r\n",
        "I have used [Pima Indians Diabetes Dataset](https://www.kaggle.com/uciml/pima-indians-diabetes-database) for implementing the models. The dataset is available with this repository. You can also download from the link given above.\r\n",
        "\r\n",
        "**Context** : This dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective of the dataset is to diagnostically predict whether or not a patient has diabetes, based on certain diagnostic measurements included in the dataset.\r\n",
        "\r\n",
        "**Content** : The datasets consists of several medical predictor variables and one target variable ```Outcome```. Predictor variables (Features) includes,\r\n",
        "- ```Number of pregnancies```\r\n",
        "- ```Glucose```\r\n",
        "- ```Blood Pressure```\r\n",
        "- ```Skin thickness```\r\n",
        "- ```Insulin level```\r\n",
        "- ```BMI```\r\n",
        "- ```Diabetes Pedegree Function```\r\n",
        "- ```Age```\r\n",
        "\r\n",
        "**Classification task** : In each instance ${x_i}$, from the given feature values, the task is to predict if the patient instance has diabetes or not."
      ],
      "metadata": {
        "id": "ELS3uEQ0XVC6"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "source": [
        "import pandas as pd\r\n",
        "df = pd.read_csv('./datasets/diabetes.csv')\r\n",
        "df.head(5)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Pregnancies</th>\n",
              "      <th>Glucose</th>\n",
              "      <th>BloodPressure</th>\n",
              "      <th>SkinThickness</th>\n",
              "      <th>Insulin</th>\n",
              "      <th>BMI</th>\n",
              "      <th>DiabetesPedigreeFunction</th>\n",
              "      <th>Age</th>\n",
              "      <th>Outcome</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6</td>\n",
              "      <td>148</td>\n",
              "      <td>72</td>\n",
              "      <td>35</td>\n",
              "      <td>0</td>\n",
              "      <td>33.6</td>\n",
              "      <td>0.627</td>\n",
              "      <td>50</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>85</td>\n",
              "      <td>66</td>\n",
              "      <td>29</td>\n",
              "      <td>0</td>\n",
              "      <td>26.6</td>\n",
              "      <td>0.351</td>\n",
              "      <td>31</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>183</td>\n",
              "      <td>64</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>23.3</td>\n",
              "      <td>0.672</td>\n",
              "      <td>32</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>89</td>\n",
              "      <td>66</td>\n",
              "      <td>23</td>\n",
              "      <td>94</td>\n",
              "      <td>28.1</td>\n",
              "      <td>0.167</td>\n",
              "      <td>21</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>137</td>\n",
              "      <td>40</td>\n",
              "      <td>35</td>\n",
              "      <td>168</td>\n",
              "      <td>43.1</td>\n",
              "      <td>2.288</td>\n",
              "      <td>33</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
              "0            6      148             72             35        0  33.6   \n",
              "1            1       85             66             29        0  26.6   \n",
              "2            8      183             64              0        0  23.3   \n",
              "3            1       89             66             23       94  28.1   \n",
              "4            0      137             40             35      168  43.1   \n",
              "\n",
              "   DiabetesPedigreeFunction  Age  Outcome  \n",
              "0                     0.627   50        1  \n",
              "1                     0.351   31        0  \n",
              "2                     0.672   32        1  \n",
              "3                     0.167   21        0  \n",
              "4                     2.288   33        1  "
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Half Space using LP**"
      ],
      "metadata": {
        "id": "P_40AndKPez7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data Preparation"
      ],
      "metadata": {
        "id": "RGBXuZykXjdm"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "source": [
        "import pandas as pd\r\n",
        "import numpy as np\r\n",
        "df = pd.read_csv('./datasets/diabetes.csv')\r\n",
        "\r\n",
        "Y = np.array(df['Outcome'])\r\n",
        "X = np.array([list(df.loc[i][:-1]) for i in range(len(df))])"
      ],
      "outputs": [],
      "metadata": {
        "id": "Tt70jynYNVYf"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "source": [
        "for i in range(len(Y)):\r\n",
        "  if Y[i] == 0:\r\n",
        "    Y[i]=-1\r\n",
        "\r\n",
        "size = len(Y)\r\n",
        "random_indice = np.random.permutation(size)\r\n",
        "num_train = int(size*0.7)\r\n",
        "num_test = int(size*0.3)\r\n",
        "\r\n",
        "X_train = X[random_indice[:num_train]]\r\n",
        "y_train = Y[random_indice[:num_train]]\r\n",
        "X_test = X[random_indice[-num_test:]]\r\n",
        "y_test = Y[random_indice[-num_test:]]"
      ],
      "outputs": [],
      "metadata": {
        "id": "SY_Ad9Ec7TRO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Formulation of LP\n",
        "\n",
        "Maximize ${1}$ with subject to ${<\\underline{w}, y_i\\underline{x_i}> \\ge 1}$"
      ],
      "metadata": {
        "id": "3jMgHeGNXnYr"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "source": [
        "# yixi matrix (not inner product)\r\n",
        "res = np.array([[0 for a in range(len(X_train[0]))] for b in range(num_train)])\r\n",
        "for i in range(num_train):\r\n",
        "  for j in range(len(X_train[0])):\r\n",
        "    res[i][j] = y_train[i] * X_train[i][j]"
      ],
      "outputs": [],
      "metadata": {
        "id": "rn3zVik8-Fuv"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "source": [
        "import pulp as p\r\n",
        "\r\n",
        "Lp_prob = p.LpProblem('HSLP',p.LpMinimize)\r\n",
        "Lp_prob+=1\r\n",
        "\r\n",
        "w = np.array([p.LpVariable('w'+str(i)) for i in range(len(X_train[0]))])\r\n",
        "w = np.transpose(w)\r\n",
        "inner_prod = list(np.inner(w,res)) # <w, yixi>\r\n",
        "for i in inner_prod:\r\n",
        "  Lp_prob+=i>=1\r\n",
        "\r\n",
        "status = Lp_prob.solve()\r\n",
        "print(p.LpStatus[status])\r\n",
        "\r\n",
        "for i in range(len(X_train[0])):\r\n",
        "  print('w'+str(i)+' = '+str(p.value(w[i])))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Infeasible\n",
            "w0 = 0.026120028\n",
            "w1 = 0.0067761417\n",
            "w2 = -0.023588319\n",
            "w3 = -0.0029847554\n",
            "w4 = -0.00010776914\n",
            "w5 = -0.0051484147\n",
            "w6 = 0.054173608\n",
            "w7 = -0.00059618779\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UIOlGK-qCwZp",
        "outputId": "b28e5cfd-c64a-40b9-d2e4-46e2a53c75fd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Prediction using Test set\n",
        "\n",
        "Using the obtained weight vector $\\underline{w}$, the Test set is predicted and the obtained accuracy is around ${69.56\\%}$"
      ],
      "metadata": {
        "id": "5RKYyvYoX94C"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "source": [
        "w_val = [p.value(w[i]) for i in range(len(w))]\r\n",
        "y_pred = list()\r\n",
        "for i in range(num_test):\r\n",
        "  if np.inner(w_val, X_test[i])>0:\r\n",
        "    y_pred.append(1)\r\n",
        "  else :\r\n",
        "    y_pred.append(-1)\r\n",
        "\r\n",
        "cclf=0\r\n",
        "for i in range(num_test):\r\n",
        "  if y_pred[i]==y_test[i]:\r\n",
        "    cclf+=1\r\n",
        "print(\"Accuracy = \"+str(cclf/num_test))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy = 0.6521739130434783\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zxlhd8lpG7PB",
        "outputId": "f21c4af1-150f-4fe8-ff91-fb13636dc0e6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Half Space using Perceptron**\n",
        "\n",
        "The Perceptron model starts with some initial random $\\underline{w}$ and gets updated as it scans each instance of the Training set."
      ],
      "metadata": {
        "id": "q5JxBGeVYa3u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data Preparation"
      ],
      "metadata": {
        "id": "RQiYwZNZDwu2"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "source": [
        "import pandas as pd\r\n",
        "import numpy as np\r\n",
        "df = pd.read_csv('./datasets/diabetes.csv')\r\n",
        "\r\n",
        "Y = np.array(df['Outcome'])\r\n",
        "X = np.array([list(df.loc[i][:-1]) for i in range(len(df))])"
      ],
      "outputs": [],
      "metadata": {
        "id": "JNHEuwEvNYOg"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "source": [
        "for i in range(len(Y)):\r\n",
        "  if Y[i] == 0:\r\n",
        "    Y[i]=-1\r\n",
        "\r\n",
        "size = len(Y)\r\n",
        "random_indice = np.random.permutation(size)\r\n",
        "num_train = int(size*0.7)\r\n",
        "num_test = int(size*0.3)\r\n",
        "\r\n",
        "X_train = X[random_indice[:num_train]]\r\n",
        "y_train = Y[random_indice[:num_train]]\r\n",
        "X_test = X[random_indice[-num_test:]]\r\n",
        "y_test = Y[random_indice[-num_test:]]"
      ],
      "outputs": [],
      "metadata": {
        "id": "0c32T6VVYelt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Training iteratively using Train set"
      ],
      "metadata": {
        "id": "M2AiJIHfDzMp"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "source": [
        "import random\r\n",
        "w = np.array([random.random() for _ in range(len(X_train[0]))])\r\n",
        "print(\"Initial weight : \",w)\r\n",
        "for i in range(num_train):\r\n",
        "  inner_prod = np.inner(w, X_train[i])\r\n",
        "  if y_train[i]*inner_prod <= 0 :\r\n",
        "    w = np.add(w, np.dot(y_train[i], X_train[i]))\r\n",
        "print(\"Updated weight : \",w)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Initial weight :  [0.01457531 0.38077669 0.81243782 0.2058043  0.01818444 0.94797902\n",
            " 0.12066552 0.03133738]\n",
            "Updated weight :  [  76.01457531  -28.61922331 -337.18756218  -74.7941957    -7.98181556\n",
            "    7.04797902    7.54266552 -118.96866262]\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ofzNWoyE8M2b",
        "outputId": "30333757-d336-4c90-9ef2-6fb3931325da"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Prediction using Test set\n",
        "\n",
        "Using the updated weight vector $\\underline{w}$, the prediction is performed from Test set instances and the obtained accuracy is around ${63.47\\%}$"
      ],
      "metadata": {
        "id": "6Vf9O-nbD4PU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "source": [
        "y_pred=list()\r\n",
        "for i in range(num_test):\r\n",
        "  if np.inner(w, X_test[i]) > 0 :\r\n",
        "    y_pred.append(1)\r\n",
        "  else :\r\n",
        "    y_pred.append(-1)\r\n",
        "cclf=0\r\n",
        "for i in range(num_test):\r\n",
        "  if y_pred[i]==y_test[i]:\r\n",
        "    cclf+=1\r\n",
        "print(\"Accuracy = \"+str(cclf/num_test))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy = 0.6565217391304348\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XXJZByK6-4NV",
        "outputId": "55036800-3411-47bc-f1fc-6287f51f648a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Logistic Regression using Gradient Descent**\n",
        "\n",
        "It starts with some initial $\\underline{w}$ and applies gradient descent using the training set till convergence. Here, around ${10^6}$ iterations have been performed and the learning rate $\\alpha = 0.00015$"
      ],
      "metadata": {
        "id": "sAzWPihkxQ49"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data preparation"
      ],
      "metadata": {
        "id": "4nftUa_yLz5u"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "source": [
        "import pandas as pd\r\n",
        "import numpy as np\r\n",
        "df = pd.read_csv('./datasets/diabetes.csv')\r\n",
        "\r\n",
        "Y = np.array(df['Outcome'])\r\n",
        "X = np.array([list(df.loc[i][:-1]) for i in range(len(df))])"
      ],
      "outputs": [],
      "metadata": {
        "id": "RHIhW2fUN9Ue"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "source": [
        "import numpy as np\r\n",
        "size = len(Y)\r\n",
        "random_indice = np.random.permutation(size)\r\n",
        "num_train = int(size*0.7)\r\n",
        "num_test = int(size*0.3)\r\n",
        "\r\n",
        "X_train = X[random_indice[:num_train]]\r\n",
        "y_train = Y[random_indice[:num_train]]\r\n",
        "X_test = X[random_indice[-num_test:]]\r\n",
        "y_test = Y[random_indice[-num_test:]]"
      ],
      "outputs": [],
      "metadata": {
        "id": "sdNJtfrAxaB3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Applying Gradient descent using Training set"
      ],
      "metadata": {
        "id": "VRgMlqreL30b"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "source": [
        "def sigmoid(z):\r\n",
        "  return 1/(1+np.exp(-z))\r\n",
        "\r\n",
        "import random\r\n",
        "epoch = 1000000\r\n",
        "alpha = 0.00015\r\n",
        "w = [0 for _ in range(len(X_train[0]))]\r\n",
        "b=0\r\n",
        "print(\"Initial weight : \",w)\r\n",
        "num_features = len(X_train[0])\r\n",
        "for i in range(epoch):\r\n",
        "  z = np.dot(X_train, np.transpose(w))+b\r\n",
        "  h = sigmoid(z)\r\n",
        "  cost = -(1/num_features)*(y_train*np.log(h) + (1-y_train)*np.log(1-h))\r\n",
        "  d_cost = (1/num_features)*np.dot(np.transpose(X_train), (h-y_train))\r\n",
        "  d_b = (1/num_features)*(h-y_train)\r\n",
        "  w = w - alpha*np.transpose(d_cost)\r\n",
        "  b = b - alpha*b\r\n",
        "print(\"Updated weight : \",w)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Initial weight :  [0, 0, 0, 0, 0, 0, 0, 0]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "C:\\Users\\ayana\\AppData\\Local\\Temp/ipykernel_2444/3561230824.py:14: RuntimeWarning: divide by zero encountered in log\n",
            "  cost = -(1/num_features)*(y_train*np.log(h) + (1-y_train)*np.log(1-h))\n",
            "C:\\Users\\ayana\\AppData\\Local\\Temp/ipykernel_2444/3561230824.py:14: RuntimeWarning: invalid value encountered in multiply\n",
            "  cost = -(1/num_features)*(y_train*np.log(h) + (1-y_train)*np.log(1-h))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Updated weight :  [ 5.68597219 -0.08103908 -1.3511854  -0.2805684  -0.18509202 -0.16750063\n",
            " 13.336995   -0.04240825]\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QQQLFAAFoaea",
        "outputId": "d2924c21-995a-430d-aaa0-cc76186b7e91"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Predicting using Test set\n",
        "\n",
        "Using the obtained weight vector $\\underline{w}$, the Test set is predicted and the obtained accuracy is around ${63.47\\%}$"
      ],
      "metadata": {
        "id": "3OFu-8HWL7eA"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "source": [
        "z_pred = np.dot(X_test, np.transpose(w))+b\r\n",
        "h_pred = sigmoid(z_pred)\r\n",
        "\r\n",
        "cclf = 0\r\n",
        "for i in range(num_test) :\r\n",
        "  if y_test[i]==1 and h_pred[i]>0.5:\r\n",
        "    cclf+=1\r\n",
        "  if y_test[i]==0 and h_pred[i]<0.5:\r\n",
        "    cclf+=1\r\n",
        "\r\n",
        "print(\"Accuracy : \"+str(cclf/num_test))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy : 0.6304347826086957\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jg2bi3RnIWqI",
        "outputId": "7276ce24-a680-47e8-f958-76608e7a3209"
      }
    }
  ]
}