02_Understanding_Computation_Graphs_and_Autograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cff61ec3-aa19-44b9-a11f-d12150b4f55c",
   "metadata": {},
   "source": [
    " PyTorch stands out as a versatile and powerful tool for building and training neural networks. At the heart of PyTorch lies its automatic differentiation functionality, which enables seamless computation of derivatives, crucial for optimizing parameters in complex models. Additionally, PyTorch's flexibility extends beyond neural networks, offering a wide range of applications, including curve fitting. Let's delve into the core concepts and practical examples that demonstrate the prowess of PyTorch.\n",
    "\n",
    "### Understanding PyTorch's Differentiation Mechanism\n",
    "\n",
    "PyTorch's differentiation capabilities revolve around the `torch.nn.Autograd.Function` class, which plays a pivotal role in computing gradients during both forward and backward passes. The forward function calculates outputs based on inputs, while the backward function propagates gradients backward through the network, employing the chain rule of calculus. This mechanism allows PyTorch to automatically compute gradients, simplifying the process of optimizing complex functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "68138a41-068e-42e9-a591-34e580380734",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# Example demonstrating forward and backward passes in PyTorch's differentiation mechanism\n",
    "class ExampleFunction(torch.autograd.Function):\n",
    "    @staticmethod\n",
    "    def forward(ctx, x, y):\n",
    "        # Forward pass computes the output\n",
    "        result = x + y\n",
    "        ctx.save_for_backward(x, y)\n",
    "        return result\n",
    "\n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):\n",
    "        # Backward pass computes gradients\n",
    "        x, y = ctx.saved_tensors\n",
    "        grad_x = grad_y = None\n",
    "        if ctx.needs_input_grad[0]:\n",
    "            grad_x = grad_output\n",
    "        if ctx.needs_input_grad[1]:\n",
    "            grad_y = grad_output\n",
    "        return grad_x, grad_y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b4c9ff9-6005-405b-b2c3-1b6ac7fc1282",
   "metadata": {},
   "source": [
    "In order to compute derivatives in our neural network, we generally call `backward` on the Tensor representing our loss. Then, we backtrack through the graph starting from node representing the grad_fn of our loss.\n",
    "\n",
    "As described above, the backward function is recursively called through out the graph as we backtrack. Once, we reach a leaf node, since the `grad_fn` is None, but stop backtracking through that path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5dc6fcb1-3abd-4d23-9dff-b415c84be3d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.)\n",
      "tensor(1.)\n"
     ]
    }
   ],
   "source": [
    "# Example usage\n",
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "y = torch.tensor(3.0, requires_grad=True)\n",
    "z = ExampleFunction.apply(x, y)\n",
    "z.backward()\n",
    "print(x.grad)  # Output: tensor(1.)\n",
    "print(y.grad)  # Output: tensor(1.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f30d85d9-f49a-4cf0-a852-8f2646d383e5",
   "metadata": {},
   "source": [
    "### Handling Vector-Valued Tensors\n",
    "\n",
    "It's important to note that PyTorch raises an error if backward is called on a vector-valued tensor. This means that backward can only be called on a scalar-valued tensor. In scenarios where a tensor is vector-valued, such as when dealing with multiple outputs or multi-class classification, additional processing may be required to obtain a scalar loss value before calling backward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "65b35b8a-5843-4fc5-9aeb-b4ae1a936fb4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 3])\n",
      "\n",
      " Can't call backward on vector valued tensor.\n"
     ]
    }
   ],
   "source": [
    "import torch \n",
    "\n",
    "a = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "w1 = torch.randn((3,3), requires_grad = True)\n",
    "w2 = torch.randn((3,3), requires_grad = True)\n",
    "w3 = torch.randn((3,3), requires_grad = True)\n",
    "w4 = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "b = w1*a \n",
    "c = w2*a\n",
    "\n",
    "d = w3*b + w4*c \n",
    "\n",
    "L = (10 - d)\n",
    "\n",
    "print(L.shape)\n",
    "\n",
    "try:\n",
    "    L.backward()\n",
    "except:\n",
    "    print(\"\\n Can't call backward on vector valued tensor.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1ef41c6-ad63-479b-91f2-4ccd862f8a40",
   "metadata": {},
   "source": [
    "### Exploring Automatic Differentiation\n",
    "\n",
    "The most important advantage of PyTorch over NumPy is its automatic differentiation functionality which is very useful in optimization applications such as optimizing parameters of a neural network. Let's try to understand it with an example.\n",
    "\n",
    "To grasp the essence of automatic differentiation in PyTorch, consider a composite function represented as a chain of two functions: $ g(u(x)) $. Leveraging PyTorch, we can compute derivatives analytically, exploiting the chain rule. By defining tensor operations and setting requires_grad to true, PyTorch facilitates the computation of derivatives effortlessly. A practical example involves defining quadratic and linear functions, ultimately demonstrating PyTorch's ability to compute derivatives accurately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "21ade761-19c2-466f-aa92-0949f00f641b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(-2.)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# Example demonstrating automatic differentiation in PyTorch\n",
    "x = torch.tensor(1.0, requires_grad=True)\n",
    "\n",
    "def u(x):\n",
    "    return x * x\n",
    "\n",
    "def g(u):\n",
    "    return -u\n",
    "\n",
    "dgdx = torch.autograd.grad(g(u(x)), x)[0]\n",
    "print(dgdx)  # Output: tensor(-2.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d8471a5-55a2-4602-a99f-ee50cb295dcf",
   "metadata": {},
   "source": [
    "### Curve Fitting\n",
    "\n",
    "The true power of PyTorch shines in scenarios like curve fitting, where we aim to estimate a function based on sampled data points. Suppose we have samples from a curve $ f(x) = 5x^2 + 3 $ and we intend to approximate $ f(x) $ using a parametric function $ g(x, w) = w_0x^2 + w_1x + w_2 $. Through stochastic gradient descent, PyTorch facilitates the minimization of a loss function, paving the way for efficient parameter estimation. By employing PyTorch's tensor operations and optimization modules, we can seamlessly train our model and achieve a close approximation to the true parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cbf92c42-4bb0-4510-97a2-bb8181880b94",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Hp\\AppData\\Local\\Temp\\ipykernel_57332\\4034046412.py:4: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  w = torch.tensor(torch.randn([3, 1]), requires_grad=True)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 4.9902859e+00]\n",
      " [-5.0420949e-04]\n",
      " [ 3.5755317e+00]]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# Example demonstrating curve fitting with PyTorch\n",
    "w = torch.tensor(torch.randn([3, 1]), requires_grad=True)\n",
    "opt = torch.optim.Adam([w], 0.1)\n",
    "\n",
    "def model(x):\n",
    "    f = torch.stack([x * x, x, torch.ones_like(x)], 1)\n",
    "    yhat = torch.squeeze(f @ w, 1)\n",
    "    return yhat\n",
    "\n",
    "def compute_loss(y, yhat):\n",
    "    loss = torch.nn.functional.mse_loss(yhat, y)\n",
    "    return loss\n",
    "\n",
    "def generate_data():\n",
    "    x = torch.rand(100) * 20 - 10\n",
    "    y = 5 * x * x + 3\n",
    "    return x, y\n",
    "\n",
    "def train_step():\n",
    "    x, y = generate_data()\n",
    "\n",
    "    yhat = model(x)\n",
    "    loss = compute_loss(y, yhat)\n",
    "\n",
    "    opt.zero_grad()\n",
    "    loss.backward()\n",
    "    opt.step()\n",
    "\n",
    "for _ in range(1000):\n",
    "    train_step()\n",
    "\n",
    "print(w.detach().numpy())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36fb4fdd-6112-4114-810d-4ca960950653",
   "metadata": {},
   "source": [
    "PyTorch emerges as a formidable tool for machine learning practitioners, offering not only automatic differentiation but also versatile functionalities like curve fitting. With its intuitive interface and robust optimization capabilities, PyTorch empowers users to tackle diverse optimization tasks, ranging from neural network training to curve approximation. As we navigate through the intricacies of PyTorch, it becomes evident that its utility transcends conventional boundaries, making it a cornerstone in the realm of computational science and artificial intelligence."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}