Assignment 7.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 7: Building Neural Networks from Scratch and with PyTorch\n",
    "\n",
    "## Objective:\n",
    "To design and implement a simple neural network from scratch using NumPy, and then transition to building more complex networks using PyTorch. This will help in understanding the foundational concepts and applying them using a deep learning framework.\n",
    "\n",
    "## Requirements:\n",
    "\n",
    "### Part 1: Simple Neural Network from Scratch using NumPy\n",
    "\n",
    "#### Neural Network Design:\n",
    "- Two input features\n",
    "- One hidden layer with one neuron (ReLU activation)\n",
    "- One output feature (Linear activation)\n",
    "\n",
    "### Mathematical Calculation Hints:\n",
    "\n",
    "#### Forward Pass:\n",
    "1. **Input to Hidden Layer**: \n",
    "   $$\n",
    "   z_1 = w_1 \\cdot x_1 + w_2 \\cdot x_2 + b\n",
    "   $$\n",
    "   where \\( w_1 \\) and \\( w_2 \\) are weights, \\( x_1 \\) and \\( x_2 \\) are inputs, and \\( b \\) is the bias.\n",
    "   \n",
    "2. **Activation (ReLU)**:\n",
    "   $$\n",
    "   a_1 = \\text{ReLU}(z_1) = \\max(0, z_1)\n",
    "   $$\n",
    "\n",
    "3. **Hidden to Output Layer**:\n",
    "   $$\n",
    "   y_{\\text{pred}} = w_3 \\cdot a_1 + b_2\n",
    "   $$\n",
    "   where \\( w_3 \\) is the weight from hidden to output, and \\( b_2 \\) is the output bias.\n",
    "\n",
    "#### Backward Pass:\n",
    "1. **Loss Calculation**:\n",
    "   $$\n",
    "   L = \\frac{1}{2}(y_{\\text{true}} - y_{\\text{pred}})^2\n",
    "   $$\n",
    "\n",
    "2. **Gradient Calculation (Output to Hidden)**:\n",
    "   $$\n",
    "   \\frac{\\partial L}{\\partial w_3} = \\frac{\\partial L}{\\partial y_{\\text{pred}}} \\cdot \\frac{\\partial y_{\\text{pred}}}{\\partial w_3}\n",
    "   $$\n",
    "   $$\n",
    "   \\frac{\\partial L}{\\partial b_2} = \\frac{\\partial L}{\\partial y_{\\text{pred}}} \\cdot \\frac{\\partial y_{\\text{pred}}}{\\partial b_2}\n",
    "   $$\n",
    "\n",
    "3. **Gradient Calculation (Hidden to Input)**:\n",
    "   $$\n",
    "   \\frac{\\partial L}{\\partial w_1} = \\frac{\\partial L}{\\partial y_{\\text{pred}}} \\cdot \\frac{\\partial y_{\\text{pred}}}{\\partial a_1} \\cdot \\frac{\\partial a_1}{\\partial z_1} \\cdot \\frac{\\partial z_1}{\\partial w_1}\n",
    "   $$\n",
    "   $$\n",
    "   \\frac{\\partial L}{\\partial w_2} = \\frac{\\partial L}{\\partial y_{\\text{pred}}} \\cdot \\frac{\\partial y_{\\text{pred}}}{\\partial a_1} \\cdot \\frac{\\partial a_1}{\\partial z_1} \\cdot \\frac{\\partial z_1}{\\partial w_2}\n",
    "   $$\n",
    "   $$\n",
    "   \\frac{\\partial L}{\\partial b_1} = \\frac{\\partial L}{\\partial y_{\\text{pred}}} \\cdot \\frac{\\partial y_{\\text{pred}}}{\\partial a_1} \\cdot \\frac{\\partial a_1}{\\partial z_1} \\cdot \\frac{\\partial z_1}{\\partial b_1}\n",
    "   $$\n",
    "\n",
    "### Implementation:\n",
    "\n",
    "- Implement the neural network using classes in Python.\n",
    "- Implement same network using PyTorch\n",
    "- Train the network on synthetic data.\n",
    "\n",
    "### Part 2: Basic Concepts in PyTorch ( This is not a task)\n",
    "\n",
    "#### PyTorch Concepts:\n",
    "- **Tensors**: Multidimensional arrays.\n",
    "- **Autograd**: Automatic differentiation.\n",
    "- **Modules**: Building blocks for neural networks.\n",
    "- **Optimizers**: Optimization algorithms.\n",
    "\n",
    "### Mini Code Tasks:\n",
    "\n",
    "1. **Building a Model**:\n",
    "   - Create a simple model using `nn.Module`.\n",
    "   \n",
    "2. **Forward Pass**:\n",
    "   - Perform a forward pass through the model.\n",
    "   \n",
    "3. **Loss Function**:\n",
    "   - Define and compute the loss using an appropriate loss function.\n",
    "   \n",
    "4. **Optimizer**:\n",
    "   - Define an optimizer for training the model.\n",
    "   \n",
    "5. **Backward Pass and Optimization Step**:\n",
    "   - Perform a backward pass and update the model parameters using the optimizer.\n",
    "\n",
    "### Part 3: Building Larger Neural Networks using PyTorch\n",
    "\n",
    "#### Regression Model:\n",
    "\n",
    "- **Input Layer**: 1 feature\n",
    "- **Hidden Layers**: 2 hidden layers, each with 10 neurons, ReLU activation\n",
    "- **Output Layer**: 1 neuron, Linear activation\n",
    "\n",
    "- Implement this model using PyTorch.\n",
    "- Train the model on synthetic data.\n",
    "\n",
    "#### Multiclass Classification Model:\n",
    "\n",
    "- **Input Layer**: 2 features\n",
    "- **Hidden Layers**: 2 hidden layers, first hidden layer with 50 neurons and second hidden layer with 25 neurons, ReLU activation\n",
    "- **Output Layer**: 5 neurons, Linear activation\n",
    "\n",
    "- Implement this model using PyTorch.\n",
    "- Train the model on synthetic data.\n",
    "\n",
    "## Submission:\n",
    "- Submit the Jupyter Notebook containing your implementation.\n",
    "- Ensure your code is well-documented and includes comments explaining the functionality of each method."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}