03 - Neural Networks/neural-networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Networks\n",
    "\n",
    "Neural networks can be constructed using the `torch.nn` package.\n",
    "\n",
    "Now that you had a glipse of `autograd`, `nn` depends on `autograd` to define models and differentiate them. An `nn.Module` contains layers, and a method `forward(input)` that returns the `output`.\n",
    "\n",
    "For example, look at a simple convolutional neural network that classifies images:\n",
    "\n",
    "![A Simple Convolutional Neural Network](../images/mnist.png)\n",
    "\n",
    "It is a simple convolutional neural network. It takes the input, feeds it through several layers one after the other and then finally returns the output.\n",
    "\n",
    "\n",
    "A typical training procedure for a neural network is as follows:\n",
    "\n",
    "- Define the neural network that has some learnable parameters (or weights)\n",
    "- Iterate over a dataset of inputs\n",
    "- Process input through the network\n",
    "- Compute the loss *(how far is the output from being correct)*\n",
    "- Propagate gradients back into the network's parameters\n",
    "- Update the weights of the network, typically using a simple update rule: `weight = weight - learning_rate * gradient`\n",
    "\n",
    "\n",
    "## Define the Network\n",
    "\n",
    "Let's define this network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PyTorch imports.\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constructing the `Net` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        \n",
    "        # in_channels: 1; out_channels: 6; kernel_size: 5x5.\n",
    "        # Defaults->stride:1; padding:0; dialation:1; groups:1; bias:True.\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        \n",
    "        # dense (or fully connected, or affine) layer [y = Wx + b]\n",
    "        # in_features: size of each input sample.\n",
    "        # out_features: size of each output sample.\n",
    "        # bias: Learn an additive bias. Default: `True`.\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # Max pooling over a 2x2 window.\n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), 2)\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n",
    "        \n",
    "        # Flatten layer.\n",
    "        x = x.view(-1, self._flatten(x))\n",
    "        \n",
    "        # Apply relu to the fully connected layers.\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        \n",
    "        # The output layer does not need ReLU activation.\n",
    "        # However, here we use softmax to squash outputs\n",
    "        # into proper probabilities that sum up to 1.\n",
    "        x = F.softmax(self.fc3(x), dim=1)\n",
    "        \n",
    "        return x\n",
    "    \n",
    "    def _flatten(self, x):\n",
    "        \"\"\"\n",
    "        Retrieve the flattened dimension excluding \n",
    "        the batch dimension.\n",
    "        \n",
    "        :param x:torch.Tensor\n",
    "            The tensor to extract it's flattened dim.\n",
    "        \n",
    "        :return num_features: int\n",
    "            Number of flattened dimension (except the batch dim).\n",
    "        \"\"\"\n",
    "        # Get all dimensions except for the batch dimension.\n",
    "        size = x.size()[1:]\n",
    "        \n",
    "        # Multiplying each dimension to get the number of features.\n",
    "        num_features = 1\n",
    "        for s in size:\n",
    "            num_features *= s\n",
    "        \n",
    "        return num_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d (1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d (6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120)\n",
      "  (fc2): Linear(in_features=120, out_features=84)\n",
      "  (fc3): Linear(in_features=84, out_features=10)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Create an instance of the Net class.\n",
    "net = Net()\n",
    "\n",
    "# Transfer the model to the GPU if there exists one.\n",
    "if torch.cuda.is_available():\n",
    "    net = net.cuda()\n",
    "\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have to define the `forward` function and the `backward` function *(where gradients are computed)* is automatically defined for you, thanks to the `autograd` package. You can also use the Tensor opeartions in the `forward` function.\n",
    "\n",
    "The learnable parameters of a model is returned by `net.parameters`\n",
    "\n",
    "**NOTE:** `net.parameters()` returns a generator, therefore you might want to convert it to a *regular Python list*. As show here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 1, 5, 5])\n"
     ]
    }
   ],
   "source": [
    "# Model's learnable parameters (weights & biases).\n",
    "params = list(net.parameters())\n",
    "\n",
    "# 1st layer (conv1) parameters.\n",
    "conv1_params = params[0]\n",
    "print(conv1_params.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv1_weights.size() = torch.Size([6, 1, 5, 5])\n",
      "conv2_bias.size()    = torch.Size([16])\n",
      "fc1_weights.size()   = torch.Size([120, 400])\n",
      "fc2_bias.size()      = torch.Size([84])\n"
     ]
    }
   ],
   "source": [
    "# Weights and bias are stored in `net.Module.weight` & `net.Module.bias`\n",
    "# e.g.\n",
    "conv1_weights = net.conv1.weight\n",
    "conv2_bias = net.conv2.bias\n",
    "\n",
    "fc1_weights = net.fc1.weight\n",
    "fc2_bias = net.fc2.bias\n",
    "\n",
    "print(f'conv1_weights.size() = {conv1_weights.size()}')\n",
    "print(f'conv2_bias.size()    = {conv2_bias.size()}')\n",
    "print(f'fc1_weights.size()   = {fc1_weights.size()}')\n",
    "print(f'fc2_bias.size()      = {fc2_bias.size()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input to the `forward` is a `autograd.Variable`, and so is the output.\n",
    "\n",
    "**NOTE:** Expected input size to this net (LeNet) is `32x32`. To use this net on MNIST, please resize the images from dataset to 32x32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 0.0953  0.1049  0.1029  0.0885  0.1068  0.1005  0.1015  0.1115  0.0955  0.0926\n",
      "[torch.FloatTensor of size 1x10]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Creating random input data.\n",
    "X_input = Variable(torch.randn(1, 1, 32, 32))\n",
    "\n",
    "# Making a prediction for the random data.\n",
    "out = net(X_input)\n",
    "\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zero the gradient buffers of all parameters and backprops with random gradients.\n",
    "\n",
    "**Recall:** If you want to compute the derivates, you can call the `.backward()` on a `Variable`. If `Variable` is a scalar *(i.e it holds a one element data)*, you don't need to specify any arguments to `backward()`, however if it has more elements, you need to specify a `grad_output` argument that is a tensor of matching shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Zero the gradient buffers.\n",
    "net.zero_grad()\n",
    "\n",
    "# `grad_output` must have same shape as `out`.\n",
    "grad_output = torch.randn(1, 10)\n",
    "\n",
    "# Backprop on out.\n",
    "out.backward(grad_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NOTE\n",
    "\n",
    "`torch.nn` only supports mini-batches. The entire `torch.nn` package only supports inputs that are a mini-batch of samples, and not a single sample.\n",
    "\n",
    "For example, `nn.Conv2d` will take in a 4D Tensor of `nSamples x nChannels x Height x Width`.\n",
    "\n",
    "If you have a single sample, just use `input.unsqueeze(0)` to add a fake batch dimension.\n",
    "\n",
    "\n",
    "**Recap:**\n",
    "\n",
    "- `torch.Tensor` – A *multi-dimensional* array.\n",
    "\n",
    "- `autograd.Variable` – Wraps a tensor and records the history of operations applied to it. Has the same API with `Tensor`, with some additions like `backward()`. Also holds the gradients w.r.t. the tensor.\n",
    "\n",
    "- `nn.Module` – Neural Network  module. *Convinent of encapsulating parameters*, with helpers for moving them to GPU, exporting, loading etc.\n",
    "\n",
    "- `nn.Parameter` – A kind of `Variable` that is *automatically registered as a parameter when assigned an attribute* to a `nn.Module`.\n",
    "\n",
    "- `autograd.Function` – Implements *forward and backward definitions of an autograd operation*. Every `Variable` operation creates at least a single `Function` node, that connects to the functions that created a `Variable` and encodes its history.\n",
    "\n",
    "At this point, we covered:\n",
    "- Defining a neural network\n",
    "- Processing inputs and calling backward.\n",
    "\n",
    "Still Left:\n",
    "- Computing the loss\n",
    "- Updating the weights of the network\n",
    "\n",
    "\n",
    "### Loss Function\n",
    "\n",
    "A loss function takes the (output, target) pair of inputs, and computes a value that estimates how far away the output is from the target.\n",
    "\n",
    "There are several different [loss function](http://pytorch.org/docs/nn.html#loss-functions) under the `nn` package. A simple loss is `nn.MSELoss` which computes the mean-squared error between the input and the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 37.4122\n",
      "[torch.FloatTensor of size 1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# A dummy example:\n",
    "\n",
    "# Inputs and lables (dummy dataset).\n",
    "X_input = torch.randn(1, 32, 32)          # NOTE: No batch dimension ...use .unsqueeze!\n",
    "X_input = Variable(X_input.unsqueeze(0))  # X_input.unsqueeze(0) adds an extra batch dim.\n",
    "target = Variable(torch.arange(1, 11))\n",
    "\n",
    "# Output prediction by the network.\n",
    "output = net(X_input)\n",
    "\n",
    "# Loss function criterion.\n",
    "loss_func = nn.MSELoss()\n",
    "\n",
    "# Calculate the loss.\n",
    "loss = loss_func(output, target)\n",
    "\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if you follow the `loss` in the backward direction, using it's `.grad_fn` attribute, you'll see a computation graph that looks like this:\n",
    "\n",
    "```\n",
    "input -> conv2d -> relu -> max_pool2d -> conv2d -> relu -> max_pool2d\n",
    "      -> view -> linear -> relu -> linear -> relu -> linear -> softmax\n",
    "      -> MSELoss\n",
    "      -> loss\n",
    "```\n",
    "\n",
    "So, when we call the `loss.backward()`, the whole graph is differentiated w.r.t. the loss, and all Variables in the graphs have their `.grad` Variable accumulated with the gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<MseLossBackward object at 0x10883c588>\n",
      "<SoftmaxBackward object at 0x10883c438>\n",
      "<AddmmBackward object at 0x10883c588>\n"
     ]
    }
   ],
   "source": [
    "# loss.grad_fn.next_functions\n",
    "# ((<SoftmaxBackward object at 0x1095a48d0>, 0),)\n",
    "\n",
    "print(loss.grad_fn)  # MSELoss\n",
    "print(loss.grad_fn.next_functions[0][0])  # Softmax\n",
    "print(loss.grad_fn.next_functions[0][0].next_functions[0][0])  # Linear"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backprop\n",
    "\n",
    "To backpropagate the error, all we need to do is `loss.backward()`. You need to clear or zero out the existing gradients, else the gradients will be accumulated to existing gradients\n",
    "\n",
    "Now, we call `loss.backward()` and we have a look at the 1st convolution bias before and after the call to backward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conv 1 bias before: Variable containing:\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 0\n",
      "[torch.FloatTensor of size 6]\n",
      "\n",
      "Conv 1 bias afer: Variable containing:\n",
      "1.00000e-03 *\n",
      "  0.7892\n",
      "  1.1216\n",
      " -4.2388\n",
      "  1.7667\n",
      " -0.8663\n",
      " -0.6962\n",
      "[torch.FloatTensor of size 6]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Zero out the gradient buffer for all params.\n",
    "net.zero_grad()\n",
    "\n",
    "# Bias gradient values::BEFORE.\n",
    "conv1_bias_before = net.conv1.bias.grad\n",
    "print('Conv 1 bias before: {}'.format(conv1_bias_before))\n",
    "\n",
    "# Back propagate: Calculates the gradients of each the model's\n",
    "# parameters w.r.t. the loss.\n",
    "loss.backward()\n",
    "\n",
    "# Bias gradient values::AFTER.\n",
    "conv1_bias_after = net.conv1.bias.grad\n",
    "\n",
    "print('Conv 1 bias afer: {conv1_bias_after}'.format(conv1_bias_after))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update the weights\n",
    "\n",
    "The simplest update rul used in practise is the Stochastic Gradient Descent (SGD):\n",
    "\n",
    "`weight = weight - learning_rate * gradient`\n",
    "\n",
    "We can implement this using small Python code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 1e-2\n",
    "\n",
    "for param in net.parameters():\n",
    "    # .sub_ method mutates the param variable\n",
    "    # by subtracting its arguments from it.\n",
    "    param.data.sub_(param.grad.data * learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, as you use neural networks, you want to use various different update rules such as *SGD*, *Nesterov-SGD*, *Adam*, *RMSProp*, etc. To enable this, the creators of Pytorch built a small package: `torch.optim` that implements all these methods. Using it is very simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "# Create SGD optimizer.\n",
    "optimizer = optim.SGD(net.parameters(), lr=1e-2)\n",
    "\n",
    "# In your training loop:\n",
    "\n",
    "# Zeros the gradient buffer.\n",
    "optimizer.zero_grad()\n",
    "\n",
    "output = net(X_input)\n",
    "\n",
    "loss = loss_func(output, target)\n",
    "loss.backward()\n",
    "\n",
    "# Does the SGD update to update all the\n",
    "# model's weights and biases.\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Observe how gradient buffers had to be manually set to zero using `optimizer.zero_grad()`. This is because gradients are accumulated as explained in [Backprop section](#Backprop)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}