09_Neural_Networks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Lab 9: Neural Networks\n",
        "\n",
        "In this coding lab, we will explore the fundamentals of building a simple neural network using Python. We will use the well-known Iris dataset to classify the flowers based on their features.\n",
        "\n",
        "Throughout this lesson, you will learn how to:\n",
        "\n",
        "1. **Prepare the Data**: Load the Iris dataset, preprocess it by normalizing the features, and apply one-hot encoding to the target labels.\n",
        "2. **Build a Neural Network**: Implement a basic neural network from scratch, including forward propagation, loss calculation, backpropagation, and parameter updates.\n",
        "3. **Train the Model**: Train the neural network on the Iris dataset and monitor the loss to ensure convergence.\n",
        "3. **Evaluate Performance**: Assess the model's accuracy on unseen data and visualize the training process.\n",
        "\n",
        "By the end of this lab, you will have a solid understanding of how neural networks work and how to implement them, from scratch, for classification tasks!\n",
        "\n",
        "***Note***: This lab is structured like our Decision Tree lab. There won't be many \"<font color='red'>**TRY IT**</font> &#x1f9e0;s\", but rather, we will walk through this live together. If you are following along from outside of our course, try to fill in wherever you see \"XXXX\". If you get stuck, refer to the answer key that's also posted in the repository."
      ],
      "metadata": {
        "id": "J9G399G40Mee"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data Preparation\n",
        "\n",
        "This section of code is responsible for preparing the Iris dataset for training a neural network.\n",
        "\n",
        "1. **Loading the Data:** The `load_iris()` function retrieves the Iris dataset, which consists of features (measurements of iris flowers) and target labels (species of the flowers).\n",
        "\n",
        "2. **One-Hot Encoding:** The target labels are reshaped and converted to a one-hot encoded format using OneHotEncoder. This transformation allows the neural network to predict class probabilities rather than single integer labels, facilitating multi-class classification.\n",
        "\n",
        "3. **Feature Normalization:** The features are standardized using `StandardScaler`, which scales the data to have a mean of 0 and a standard deviation of 1. Normalization improves the model's convergence during training by ensuring that all features contribute equally.\n",
        "\n",
        "4. **Train-Test Split:** The dataset is split into training and testing sets using `train_test_split`. This separation is crucial for evaluating the model's performance on unseen data.\n",
        "\n",
        "Together, these ensure that the data is in the right format and scale for training the neural network.\n"
      ],
      "metadata": {
        "id": "5x8uQ-zyLj8F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.preprocessing import OneHotEncoder\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "# Load the Iris dataset\n",
        "iris = load_iris()\n",
        "X = iris.data\n",
        "y = iris.target\n",
        "\n",
        "# One-Hot Encoding the target labels\n",
        "y = y.reshape(-1, 1)\n",
        "encoder = OneHotEncoder(sparse_output=False)\n",
        "y_encoded = encoder.fit_transform(y)\n",
        "\n",
        "# Normalize the features\n",
        "scaler = StandardScaler()\n",
        "X_normalized = scaler.fit_transform(X)\n",
        "\n",
        "# Split the dataset into training and testing sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_normalized, y_encoded, test_size=0.3, random_state=42)"
      ],
      "metadata": {
        "id": "4smKIuDBnIvJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Define the Neural Network Structure\n",
        "We'll create a simple one-layer neural network with an input layer and an output layer. In this case, we’ll use a single layer to classify three species of Iris."
      ],
      "metadata": {
        "id": "bs-woLcSLqJE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class SimpleNeuralNetwork:\n",
        "    def __init__(self, input_size, output_size, learning_rate=0.01):\n",
        "        \"\"\"\n",
        "        Initialize the neural network with random weights and biases.\n",
        "\n",
        "        Parameters:\n",
        "        input_size (int): Number of input features.\n",
        "        output_size (int): Number of output classes.\n",
        "        learning_rate (float): Learning rate for weight updates.\n",
        "        \"\"\"\n",
        "        self.learning_rate = learning_rate\n",
        "        self.weights = XXXX\n",
        "        self.bias = XXXX\n",
        "\n",
        "    def softmax(self, z):\n",
        "        \"\"\"Compute softmax values for each class in z.\"\"\"\n",
        "        exp_z = np.exp(z - np.max(z))  # subtraction of np.max(z) is for stability improvement\n",
        "        return XXXX / XXXX\n",
        "\n",
        "    def forward(self, X):\n",
        "        \"\"\"Forward pass: computes predicted class probabilities.\"\"\"\n",
        "        z = np.dot(XXXX, XXXX) + XXXX  # Calculate the linear combination of inputs and weights\n",
        "        return XXXX # Apply the softmax function\n",
        "\n",
        "    def compute_loss(self, y_pred, y_true):\n",
        "        \"\"\"Compute the cross-entropy loss.\"\"\"\n",
        "        m = y_true.shape[0]  # Number of samples\n",
        "\n",
        "        # Get the index of the true class for each sample\n",
        "        true_class_indices = np.argmax(y_true, axis=1)\n",
        "\n",
        "        # Calculate log-likelihood using the true class indices\n",
        "        log_likelihood = -np.log(y_pred[range(m), true_class_indices])\n",
        "        loss = XXXX / XXXX  # Average loss\n",
        "\n",
        "        return loss\n",
        "\n",
        "    def backward(self, X, y_true, y_pred):\n",
        "        \"\"\"Backward pass: computes gradients.\"\"\"\n",
        "        m = y_true.shape[0]  # Number of samples\n",
        "\n",
        "        # Get the index of the true class for each sample\n",
        "        true_class_indices = np.argmax(y_true, axis=1)\n",
        "        y_pred[range(m), true_class_indices] -= 1  # Subtract 1 from predicted classes\n",
        "        y_pred /= m  # Normalize gradients by number of samples\n",
        "\n",
        "        dw = np.dot(X.T, y_pred)  # Gradient for weights\n",
        "        db = np.sum(y_pred, axis=0)  # Gradient for bias\n",
        "\n",
        "        return dw, db\n",
        "\n",
        "    def update_parameters(self, dw, db):\n",
        "        \"\"\"\n",
        "        Update weights and biases using computed gradients.\n",
        "\n",
        "        This is where the gradient descent step occurs:\n",
        "        The weights and biases are updated to reduce the loss.\n",
        "        \"\"\"\n",
        "        # Update weights and biases using gradient descent\n",
        "        self.weights -= XXXX  # Adjust weights\n",
        "        self.bias -= XXXX  # Adjust biases\n",
        "\n",
        "    def train(self, X, y, epochs=1000):\n",
        "        \"\"\"\n",
        "        Train the neural network using the provided data.\n",
        "        \"\"\"\n",
        "        loss_history = []  # To store loss values\n",
        "\n",
        "        for epoch in range(epochs):\n",
        "\n",
        "            # Forward pass: calculate predictions\n",
        "            y_pred = XXXX\n",
        "            # Compute the loss for the current predictions\n",
        "            loss = XXXX\n",
        "\n",
        "            # Backward pass: compute gradients\n",
        "            dw, db = XXXX\n",
        "            # Update parameters based on gradients (Gradient Descent)\n",
        "            XXXX\n",
        "\n",
        "            # track the loss for plotting purposes\n",
        "            loss_history.append(loss)\n",
        "\n",
        "            # Print loss every 100 epochs for monitoring\n",
        "            if epoch % 50 == 0:\n",
        "                print(f'Epoch {epoch}, Loss: {loss:.4f}')\n",
        "\n",
        "        # Plot the loss over epochs\n",
        "        plt.plot(loss_history)\n",
        "        plt.title('Loss over Epochs')\n",
        "        plt.xlabel('Epochs')\n",
        "        plt.ylabel('Loss')\n",
        "        plt.grid()\n",
        "        plt.show()"
      ],
      "metadata": {
        "id": "9MIsAjROiTDU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create and train the neural network\n",
        "input_size = X_train.shape[1] # this is the number of features\n",
        "output_size = y_train.shape[1] # this is the number of classes\n",
        "nn = SimpleNeuralNetwork(XXXX, XXXX, XXXX=XXXX)\n",
        "nn.train(X_train, y_train, epochs=XXXX)"
      ],
      "metadata": {
        "id": "CNFvnw5wiByl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 4: Making Predictions\n",
        "After training, we can use our neural network to make predictions. We just call the `forward` function we made to run the data through the neural network. Then we take the max of the output probabilities to get the most likely class each observation belongs to."
      ],
      "metadata": {
        "id": "aBNMzluVLs5M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluate the model on the test set\n",
        "y_pred = nn.forward(X_test)\n",
        "y_pred_classes = np.argmax(y_pred, axis=1)\n",
        "y_test_classes = np.argmax(y_test, axis=1)\n",
        "\n",
        "accuracy = accuracy_score(y_test_classes, y_pred_classes)\n",
        "print(f'Accuracy: {accuracy:.2f}')"
      ],
      "metadata": {
        "id": "qs-6tuWxnrRs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "How were your results? Not so good, right? That's because we've only done a few epochs!\n",
        "\n",
        "\"<font color='red'>**TRY IT**</font> &#x1f9e0;\": Increase that number and see what happens. Still not happy with your results? Change the learning rate and see what happens!"
      ],
      "metadata": {
        "id": "-d_eC7Ynpks0"
      }
    }
  ]
}