Ch06_Multilayer_Perceptrons/Numerical_Stability_and_Initialization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Numerical Stability and Initialization\n",
    "\n",
    ":label:`chapter_numerical_stability`\n",
    "\n",
    "\n",
    "In the past few sections, each model that we implemented\n",
    "required initializing our parameters according to some specified distribution.\n",
    "However, until now, we glossed over the details,\n",
    "taking the initialization hyperparameters for granted.\n",
    "You might even have gotten the impression that these choices\n",
    "are not especially important.\n",
    "However, the choice of initialization scheme\n",
    "plays a significant role in neural network learning,\n",
    "and can prove essentially to maintaining numerical stability.\n",
    "Moreover, these choices can be tied up in interesting ways\n",
    "with the choice of the activation function.\n",
    "Which nonlinear activation function we choose,\n",
    "and how we decide to initialize our parameters\n",
    "can play a crucial role in making\n",
    "the optimization algorithm converge rapidly.\n",
    "Failure to be mindful of these issues\n",
    "can lead to either exploding or vanishing gradients.\n",
    "In this section, we delve into these topics with greater detail\n",
    "and discuss some useful heuristics that you may use\n",
    "frequently throughout your career in deep learning.\n",
    "\n",
    "\n",
    "## Vanishing and Exploding Gradients\n",
    "\n",
    "Consider a deep network with $d$ layers,\n",
    "input $\\mathbf{x}$ and output $\\mathbf{o}$.\n",
    "Each layer satisfies:\n",
    "\n",
    "$$\\mathbf{h}^{t+1} = f_t (\\mathbf{h}^t) \\text{ and thus } \\mathbf{o} = f_d \\circ \\ldots \\circ f_1(\\mathbf{x})$$\n",
    "\n",
    "If all activations and inputs are vectors,\n",
    "we can write the gradient of $\\mathbf{o}$\n",
    "with respect to any set of parameters $\\mathbf{W}_t$\n",
    "associated with the function $f_t$ at layer $t$ simply as\n",
    "\n",
    "$$\\partial_{\\mathbf{W}_t} \\mathbf{o} = \\underbrace{\\partial_{\\mathbf{h}^{d-1}} \\mathbf{h}^d}_{:= \\mathbf{M}_d} \\cdot \\ldots \\cdot \\underbrace{\\partial_{\\mathbf{h}^{t}} \\mathbf{h}^{t+1}}_{:= \\mathbf{M}_t} \\underbrace{\\partial_{\\mathbf{W}_t} \\mathbf{h}^t}_{:= \\mathbf{v}_t}.$$\n",
    "\n",
    "In other words, it is the product of $d-t$ matrices\n",
    "$\\mathbf{M}_d \\cdot \\ldots \\cdot \\mathbf{M}_t$\n",
    "and the gradient vector $\\mathbf{v}_t$.\n",
    "What happens is similar to the situation\n",
    "when we experienced numerical underflow\n",
    "when multiplying too many probabilities.\n",
    "At the time, we were able to mitigate the problem\n",
    "by switching from into log-space, i.e. by shifting the problem\n",
    "from the mantissa to the exponent of the numerical representation. Unfortunately the problem outlined in the equation above is much more serious:\n",
    "initially the matrices $M_t$ may well have a wide variety of eigenvalues.\n",
    "They might be small, they might be large, and in particular,\n",
    "their product might well be *very large* or *very small*.\n",
    "This is not (only) a problem of numerical representation\n",
    "but it means that the optimization algorithm is bound to fail.\n",
    "It receives gradients that are either\n",
    "excessively large or excessively small.\n",
    "As a result the steps taken are either\n",
    "(i) excessively large (the *exploding* gradient problem),\n",
    "in which case the parameters blow up in magnitude rendering the model useless,\n",
    "or (ii) excessively small, (the *vanishing gradient problem*),\n",
    "in which case the parameters hardly move at all,\n",
    "and thus the learning process makes no progress.\n",
    "\n",
    "\n",
    "### Vanishing Gradients\n",
    "\n",
    "One major culprit in the vanishing gradient problem\n",
    "is the choices of the activation functions $\\sigma$\n",
    "that are interleaved with the linear operations in each layer.\n",
    "Historically, a the sigmoid function $\frac{1}{1 + e^{-x}}$",
    "(introduced in :numref:`chapter_mlp`)\n",
    "was a popular choice owing to its similarity to a thresholding function.\n",
    "Since early artificial neural networks were inspired\n",
    "by biological neural networks,\n",
    "the idea of neurons that either fire or do not fire\n",
    "(biological neurons do not partially fire) seemed appealing.\n",
    "Let's take a closer look at the function\n",
    "to see why picking it might be problematic\n",
    "vis-a-vis vanishing gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAD8CAYAAABNa2y4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8leX9//HXJyebJKwAmoQRZMmQDYoMJ+ICqXXbSq21tK7W2qL2Z9WvXY62al2luHcrOEFUFBeCMqSyITLDSliB7Jyc6/fHHSBGIAdIcp8k7+fjcR7nHlfO+dwkh/e513WZcw4RERGJHFF+FyAiIiLfpXAWERGJMApnERGRCKNwFhERiTAKZxERkQijcBYREYkwCmcREZEIo3AWERGJMApnERGRCBPt1xunpqa6Dh06+PX2IiIidW7+/PnbnHOtqmvnWzh36NCBefPm+fX2IiIidc7M1oXTToe1RUREIozCWUREJMIonEVERCKMb+ecD6SsrIzs7GyKi4v9LqXeiI+PJyMjg5iYGL9LERGRGhJR4ZydnU1ycjIdOnTAzPwuJ+I559i+fTvZ2dlkZmb6XY6IiNSQag9rm9lTZpZjZosPst7M7GEzyzKzb8ys35EWU1xcTMuWLRXMYTIzWrZsqSMNIiINTDjnnJ8BRh1i/dlA54rHtcDjR1OQgvnw6N9LRKThqfawtnPuUzPrcIgmY4DnnHMOmGNmzczsWOfc5hqqUUREIpxzjpCDsvIQ5SFHMOQIfmfaEQx582XlrmL59+edg1DFa4Wc2/e63vR3nw/WpvJ8aN/83mlv+d6anQMHFc/756lo88tTOxEfE6jzf8+aOOecDmyoNJ9dsex74Wxm1+LtXdOuXbsaeOu6cc0113DzzTfTvXv3WnuPc845h5deeolmzZp9Z/ldd91FUlISt9xyS629t4g0bMVl5ewqLCO/JEhhaZCCknLvubScwpIqz6VBikrLKQmGKA2GKAlWnvbm90+HKCkrp7Q8RFm583sza5wZXDO8Y70N5wMdVz3gb8k5NxGYCDBgwIB685ucNGlSrb/HtGnTav09RKT+Kw2GyM0vIWd3MTl7SsjdU0JeURm7CkvZVVjGrqIy8grL2FVUWrG8jJJgKKzXjo2OIjE2QEJMgLjoKOKiA8TFRHnTMVGkJMRULI8idu/6iumYQBTRUUYgYERHGdFRUUQHjEDUwef3tg1EGTGBKKLMm44yiDLDKp73T3un8rxl321T9XlvGztAW/CC17CKZ6+d7V0eAacLayKcs4G2leYzgE018Lq+KCgo4OKLLyY7O5vy8nLuuOMOHn/8cR544AEGDBjAk08+yb333ktaWhqdO3cmLi6ORx55hHHjxpGQkMDy5ctZt24dTz/9NM8++yyzZ89m8ODBPPPMMwC8/PLL/PnPf8Y5x7nnnsu9994L7O/ONDU1lT/96U8899xztG3bllatWtG/f38f/0VEpK4458jNL2HDjkI27Chi/Y5CNuwoZMvuYnJ2l5Czp5idhWUH/NmEmADNEmNomhBDs8QYMlOb0Cwh1ltWsTwpLpomsdEkxgVoEhtNk7gAibHRNImLJjE2QExAXV9EipoI57eA683sFWAwkFcT55vvfnsJSzftPuriKuuelsKd5/c4ZJvp06eTlpbG1KlTAcjLy+Pxx71r3DZt2sQ999zDggULSE5O5rTTTqN37977fnbnzp189NFHvPXWW5x//vnMmjWLSZMmMXDgQBYuXEjr1q2ZMGEC8+fPp3nz5owcOZI33niDCy64YN9rzJ8/n1deeYWvv/6aYDBIv379FM4iDcye4jJW5eSTtTWflVv3sHpbgRfIOwspLvvuXm7r5DiObZZAu5aJDOjQnNbJ8bROiaN1chxtUuJplRxHs8QY4qLr/tCr1J5qw9nMXgZOAVLNLBu4E4gBcM49AUwDzgGygELgJ7VVbF3o1asXt9xyCxMmTOC8885j2LBh+9Z99dVXjBgxghYtWgBw0UUXsXLlyn3rzz//fMyMXr160aZNG3r16gVAjx49WLt2LevWreOUU06hVStvQJIrrriCTz/99Dvh/NlnnzF27FgSExMBGD16dK1vs4jUnq27i1mUnceijXks3pjHss272ZS3//bHuOgoMlObkJnahBFdWtG2RSLtWiTStkUCGc0TfTnfKf4L52rty6pZ74DraqyiCtXt4daWLl26MH/+fKZNm8Ztt93GyJEj961z7tCnyePi4gCIioraN713PhgMEh0d3oGKSDjfISKHL1geYtnmPXy5ZjtfrdnB1xt2kbunBPDOl3ZqncSgzBZ0OSaZzq2T6dImiYzmiQSi9JmX74qoHsIiwaZNm2jRogVXXnklSUlJ+84VAwwaNIhf//rX7Ny5k+TkZCZPnrxv7zgcgwcP5qabbmLbtm00b96cl19+mRtuuOE7bYYPH864ceO49dZbCQaDvP322/z85z+vqc0TkRrknGPl1nxmrshh9rfbmb9uJ/klQQDat0xkWKdUemU0pVd6U7qnpZAYq/9yJTz6S6li0aJF/Pa3vyUqKoqYmBgef/zxfbcxpaenc/vttzN48GDS0tLo3r07TZs2Dfu1jz32WP7yl79w6qmn4pzjnHPOYcyYMd9p069fPy655BL69OlD+/btv3NYXUT8V1xWzhffbuOj5TnMXJ7Lxl1FAHRuncQFfdMYlNmSQR1acEzTeJ8rlfrMqjtUW1sGDBjg5s2b951ly5Yt4/jjj/elnnDl5+eTlJREMBhk7NixXH311YwdO9bXmurDv5tIfVZWHuKzVbm8uXATHyzdSmFpOYmxAU7ulMpp3VpzatfWCmMJi5nNd84NqK6d9pwP01133cWMGTMoLi5m5MiR37mYS0QaDuccc9fu5I2FG5m2aDO7CstolhjDBX3TGdXjGAZ3bKErpKXWKJwP0wMPPOB3CSJSi/JLgrz+9Uaen72WlVvzSYgJMLJHG8b0SWNop1bERuteYKl9CmcRESArJ58X5qzjtfnZ5JcE6Zmewn0/PIHzTjhWF3JJndNfnIg0ass27+bhD1fx7uItxAaiOPeEY/nRSe3p27aZbmsU3yicRaRRqhzKyXHR3HBaJ64a0oHUpLjqf1iklimcRaRRWbOtgPumL98Xyjee1omrh2bSLDHW79JE9tGVDbWsQ4cObNu2DYAhQ4Yc8es888wzbNpUb8cTEfFdQUmQe6cv56x/fMpnq7Zx42md+GzCqdw8squCWSKO9pyPwOF0xVnZF198ccTv+cwzz9CzZ0/S0tKO+DVEGiPnHO98s5k/TV3Glt3F/KBfOree3Y3WybovWSKXwvkA7rnnHl588UXatm1Lamoq/fv355133mHIkCHMmjWL0aNH06VLF/74xz9SWlpKy5YtefHFF2nTpg3bt2/nsssuIzc3l0GDBn2nP+6kpCTy8/MBuP/++/nPf/5DSUkJY8eO5e6772bt2rWcffbZDB06lC+++IL09HTefPNNpk6dyrx587jiiitISEhg9uzZJCQk+PXPI1JvfJubz+9fX8Sc1TvokZbCo1f0pX/7Fn6XJVKtyA3nd2+FLYtq9jWP6QVn//WQTebNm8fkyZMPOGTjrl27+OSTTwBveMg5c+ZgZkyaNIn77ruPv/3tb9x9990MHTqUP/zhD0ydOpWJEyd+7z3ef/99Vq1axVdffYVzjtGjR/Ppp5/Srl07Vq1axcsvv8y///1vLr74YiZPnsyVV17JI488sm9MaRE5tPKQ48nPV/PA+ytJiAnwp7E9uXRgOw0wIfVG5IazTz7//HPGjBmzb8/0/PPP37fukksu2TednZ3NJZdcwubNmyktLSUzMxOATz/9lClTpgBw7rnn0rx58++9x/vvv8/7779P3759Aa9L0FWrVtGuXTsyMzPp06cPAP3792ft2rW1sp0iDdWGHYX86tWFzF+3k5Hd2/DHsT11CFvqncgN52r2cGvLofoab9Kkyb7pG264gZtvvpnRo0fz8ccfc9ddd+1bV929kc45brvttu+NNrV27drvDDUZCAQoKio6zC0Qabze+t8mfj/FO+L24CV9GNMnTfcqS72kq7WrGDp0KG+//TbFxcXk5+czderUA7bLy8sjPT0dgGeffXbf8uHDh/Piiy8C8O6777Jz587v/exZZ53FU089te/888aNG8nJyTlkXcnJyezZs+eItkmkoSsJlnPblEXc+PLXdGqTxLSbhnFB33QFs9Rbkbvn7JOBAwcyevRoevfuTfv27RkwYMABh4W86667uOiii0hPT+fEE09kzZo1ANx5551cdtll9OvXjxEjRtCuXbvv/ezIkSNZtmwZJ510EuBdKPbCCy8QCBy8E/1x48Yxfvx4XRAmUsXGXUX88oX5/C87j/EjjuM3I7sQE9B+h9RvGjLyAPYOC1lYWMjw4cOZOHEi/fr187usg4qUfzeRujZ37Q5+/vx8SoMhHrioN6N6HuN3SSKHpCEjj8K1117L0qVLKS4u5qqrroroYBZprF7/OpsJry0ivXkCk64awHGtkvwuSaTGKJwP4KWXXvK7BBE5COccD324igdnrOLEji144sr+6uFLGpyIC2fnnC7iOAx+nZYQ8UN5yPGHNxfz4pfrubBfBn/5QS+NrywNUkSFc3x8PNu3b6dly5YK6DA459i+fTvx8bqHUxq+kmA5v3plIe8u3sL4EccxYVRX/T8hDVZEhXNGRgbZ2dnk5ub6XUq9ER8fT0ZGht9liNSq4rJyfvHCfGauyOX/nXs81wzr6HdJIrUqosI5JiZmX09bIiLgBfPPnpvHZ6u28eexvbh88PdvTxRpaCIqnEVEKisuK+eaZ+cx69tt3PfDE7h4QFu/SxKpEwpnEYlIZeUhrn9pAZ9nbeOBi3rzw/46fSONhy5zFJGIUx5y/OrVhcxYlsM9F/RUMEujo3AWkYjinOPOtxYz9ZvN3H5ON350Ynu/SxKpcwpnEYkoj87M4oU56xk/4jiuHX6c3+WI+ELhLCIR47/zNvDA+ysZ2zed353V1e9yRHwTVjib2SgzW2FmWWZ26wHWNzWzt83sf2a2xMx+UvOlikhDNvvb7dw2ZRFDO6Vy74UnEBWlDkak8ao2nM0sADwKnA10By4zs+5Vml0HLHXO9QZOAf5mZursVkTCsnZbAb94cT4dUpvw2JX91CWnNHrhfAIGAVnOudXOuVLgFWBMlTYOSDavL70kYAcQrNFKRaRByisq46fPzsWAJ68aQEp8jN8lifgunHBOBzZUms+uWFbZI8DxwCZgEXCTcy5UIxWKSIMVCjlufnUh67YX8viV/WnfsonfJYlEhHDC+UAnfqoOhXQWsBBIA/oAj5hZyvdeyOxaM5tnZvPUf7aIPPzRKj5cnsMfzu/OiR1b+l2OSMQIJ5yzgcp95mXg7SFX9hNgivNkAWuAblVfyDk30Tk3wDk3oFWrVkdas4g0AB8u28qDM1bxg37pupdZpIpwwnku0NnMMisu8roUeKtKm/XA6QBm1gboCqyuyUJFpOHYuKuIX7+6kB5pKfx5bC8N/ShSRbV9azvngmZ2PfAeEACecs4tMbPxFeufAO4BnjGzRXiHwSc457bVYt0iUk8Fy0P86pWvCTl47Ip+xMcE/C5JJOKENfCFc24aMK3KsicqTW8CRtZsaSLSEP3zoyzmrt3Jg5f00QVgIgehmwlFpM58tWYH//zIO898Qd+qN32IyF4KZxGpE7sKS/nVK1/TrkUi/zemp9/liEQ0jecsIrXOOcetkxeRs6eEKb8cQlKc/usRORTtOYtIrXvpq/VMX7KF343qygkZzfwuRyTiKZxFpFat2VbAPe8sZVjnVK4Z2tHvckTqBYWziNSaUMgxYfI3xASiuP+HvTXSlEiYFM4iUmte/HIdX63ZwR3ndeeYpvF+lyNSbyicRaRWbNhRyF/eXc6wzqlc1D/D73JE6hWFs4jUOOcct7++CAP+8gN1zylyuBTOIlLj/jsvm89WbePWc44no3mi3+WI1DsKZxGpUVvyirln6lIGZ7bgikHt/C5HpF5SOItIjXHO8f/eWERZeYh7LzxBV2eLHCGFs4jUmOmLtzBjWQ63jOxKh1QNaiFypBTOIlIjCkuD/N87Szn+2BTGDengdzki9ZrCWURqxMMfZrE5r5g/XtCD6ID+axE5GvoEichRy8rZw6TPVnNR/wz6t2/hdzki9Z7CWUSOinOOP7y5hMTYABPO7uZ3OSINgsJZRI7KO99s5otvt/Pbs7qSmhTndzkiDYLCWUSOWH5JkD9OXUrP9BQuH9ze73JEGgyNeC4iR+yhGSvJ2VPCE1f2J6B7mkVqjPacReSIrNq6h6dmreXSgW3p26653+WINCgKZxE5In+atozE2AC/PUsXgYnUNIWziBy2T1bm8vGKXG48rTMtmsT6XY5Ig6NwFpHDEiwP8aepS2nfMpEfD9FFYCK1QeEsIofllbkbWLk1n9vO7kZcdMDvckQaJIWziIRtd3EZ//hgJYMyW3BWj2P8LkekwdKtVCIStkdnZrGjsJRnzu2OmW6dEqkt2nMWkbBs2FHI05+v5Qd9M+iV0dTvckQaNIWziITlr9OXE4gyfntWV79LEWnwFM4iUq3563Yw9ZvN/HxER45pGu93OSINnsJZRA7JOcdf311O6+Q4rh3e0e9yRBqFsMLZzEaZ2QozyzKzWw/S5hQzW2hmS8zsk5otU0T88uGyHOau3clNZ3QmMVbXkIrUhWo/aWYWAB4FzgSygblm9pZzbmmlNs2Ax4BRzrn1Zta6tgoWkbpTHnLc995yOqY24eIBbf0uR6TRCGfPeRCQ5Zxb7ZwrBV4BxlRpczkwxTm3HsA5l1OzZYqIH6YsyGbl1nxuOasrMQGdBROpK+F82tKBDZXmsyuWVdYFaG5mH5vZfDP78YFeyMyuNbN5ZjYvNzf3yCoWkTpRXFbOPz5YSe+2zTi7pzocEalL4YTzgXoacFXmo4H+wLnAWcAdZtblez/k3ETn3ADn3IBWrVoddrEiUneen72OTXnF3DqqmzocEalj4VzdkQ1UPtmUAWw6QJttzrkCoMDMPgV6AytrpEoRqVN5RWU8MjOLEV1acdJxLf0uR6TRCWfPeS7Q2cwyzSwWuBR4q0qbN4FhZhZtZonAYGBZzZYqInXlX598y+7iMiaM0ljNIn6ods/ZORc0s+uB94AA8JRzbomZja9Y/4RzbpmZTQe+AULAJOfc4tosXERqx9bdxTw1aw1jeqfRPS3F73JEGqWwblp0zk0DplVZ9kSV+fuB+2uuNBHxw8MfrqI85PjNSHXTKeIX3RshIvus317Iq3M3cOnAdrRtkeh3OSKNlsJZRPZ58MOVBKKM60/r5HcpIo2awllEAMjK2cMbX2/kqiEdaJOiwS1E/KRwFhEA/v7BShJjoxk/4ji/SxFp9BTOIsLijXlMW7SFq4dm0qJJrN/liDR6CmcR4W/vr6BpQgzXDMv0uxQRQeEs0ujNX7eDmStyGT/iOFLiY/wuR0RQOIs0eg+8t5LUpDiuGtLe71JEpILCWaQRm5W1jdmrt3P9qceRGBtWn0QiUgcUziKNlHOO+99bQVrTeC4b3M7vckSkEoWzSCP14bIcFm7YxY2ndyYuOuB3OSJSicJZpBEKhRwPvL+CDi0TubB/ht/liEgVCmeRRmjqos0s37KHX5/ZhZiA/hsQiTT6VIo0MsHyEP/4YCVd2yRz/glpfpcjIgegcBZpZKZ8vZHV2wq4eWQXoqLM73JE5AAUziKNSGkwxEMzVtE7oykju7fxuxwROQiFs0gj8urc9WzcVcRvRnbFTHvNIpFK4SzSSBSVlvPPj7IY1KEFwzqn+l2OiByCwlmkkXh+zlpy9pRwy1naaxaJdApnkUZgT3EZj338LcO7tGJQZgu/yxGRaiicRRqBf3+2hl2FZfx2ZFe/SxGRMCicRRq47fklPPnZas7pdQy9Mpr6XY6IhEHhLNLAPfbxtxSVlXPzmdprFqkvFM4iDdimXUU8P2cdF/bLoFPrJL/LEZEwKZxFGrCHP1wFDm46o7PfpYjIYVA4izRQq3Pz+e/8bC4f3I6M5ol+lyMih0HhLNJA/f2DlcRFR3HdqZ38LkVEDpPCWaQBWrIpj3e+2czVJ2fSKjnO73JE5DApnEUaoAfeW0HThBh+Nryj36WIyBFQOIs0MHPX7mDmilzGjziOpgkxfpcjIkcgrHA2s1FmtsLMsszs1kO0G2hm5Wb2w5orUUTC5Zzj/ukraJUcx7ghHfwuR0SOULXhbGYB4FHgbKA7cJmZdT9Iu3uB92q6SBEJzycrc/lq7Q5uPK0TCbEBv8sRkSMUzp7zICDLObfaOVcKvAKMOUC7G4DJQE4N1iciYQqFHPe/t4KM5glcMrCd3+WIyFEIJ5zTgQ2V5rMrlu1jZunAWOCJmitNRA7H1EWbWbJpN78+owux0bqcRKQ+C+cTfKCBX12V+QeBCc658kO+kNm1ZjbPzObl5uaGW6OIVKMkWM597y2n2zHJXNA3vfofEJGIFh1Gm2ygbaX5DGBTlTYDgFcqBnBPBc4xs6Bz7o3KjZxzE4GJAAMGDKga8CJyhF6Ys54NO4p49upBBKIO9H1aROqTcMJ5LtDZzDKBjcClwOWVGzjnMvdOm9kzwDtVg1lEakdeURn//GgVQzulMrxzqt/liEgNqDacnXNBM7se7yrsAPCUc26JmY2vWK/zzCI+euzjLPKKyrjtnG5UHL0SkXounD1nnHPTgGlVlh0wlJ1z446+LBEJR/bOQp6etZaxfdPpkdbU73JEpIbokk6Reuzv768E4JaRXX2uRERqksJZpJ5avDGP1xdu5OqTM0lrluB3OSJSgxTOIvWQc46/vLuMZgkx/PLU4/wuR0RqmMJZpB76eGUus7K2c+PpnUmJ1+AWIg2NwlmknikNhrjnnaVkpjbhisHt/S5HRGqBwlmknnlu9lpW5xZwx3nHq5tOkQZKn2yReiR3TwkPzVjFKV1bcVq3Nn6XIyK1ROEsUo888N4KioPl3HHe90ZtFZEGROEsUk98k72L/8zfwE9OzuS4Vkl+lyMitUjhLFIPOOe4660ltGwSxw2ndfK7HBGpZQpnkXrgjYUbWbB+F78b1ZVk3Tol0uApnEUiXEFJkL++u5zeGU35Yb8Mv8sRkTqgcBaJcA9/tIqtu0u4c3QPojRWs0ijoHAWiWDLNu9m0mdruHhABv3aNfe7HBGpIwpnkQhVHnLcNmURzRJiuP2c4/0uR0TqkMJZJEK9+OU6Fm7YxR3ndadZYqzf5YhIHVI4i0SgLXnF3Dd9BcM6pzKmT5rf5YhIHVM4i0Sgu99eQll5iD9e0BMzXQQm0tgonEUizIylW3l38RZuPL0z7Vs28bscEfGBwlkkghSUBPnDm4vp2iaZa4d39LscEfFJtN8FiMh+97+3gk15xUy+vC8xAX13Fmms9OkXiRCzsrbxzBdrGTekA/3bt/C7HBHxkcJZJALkFZVxy3//R8dWTZgwqpvf5YiIz3RYWyQC3P3WEnL2lDD5F0NIiA3UzpuUFUNBDpQWQGkhRMdCbBNIaO49RCRiKJxFfPbuos1M+XojN57emT5tm9XMi5YWQvZXsH4ObPgKtq2CvA2AO3D7hObQ4jhI6wPtToL2J0PKsTVTi4gcNoWziI9y9hRz++uL6JXe9OjHaS4rguVTYembsOoDCBYBBm16QrvB0OJySEmDuGSISYTyUijNh4JtsGM1bM+C/70Ccyd5r5fWD7qPgZ4XQrO2R72tIhI+hbOIT5xz3DZ5EQWl5fzjkt5HfnX29m+9QF34EhTvgqQ20PcK6DIK2g6C+Kbhv1Z5ELYugm9nwrK3YMad8OHd0HkkDPgpdDoDonSpikhtUziL+OTVuRv4cHkOd5zXnU6tkw//BbYshs8egCVvQFQAjj8f+v8EOgw78gANRENaX+8x7GbYuRYWPA8LnoOV06F1dxj2G+gx1ntPEakV5txBzkHVsgEDBrh58+b58t4iflu1dQ+jH5lF33bNeOGngw9vnOYdq+HD/4Mlr0NsMgy6Bgb/ApLb1F7BwVJYMgU++ztsWwGpXeGMu6Dr2aDuRUXCZmbznXMDqmunPWeROlZUWs51Ly2gSVyABy/pE34wF++Gj/8CX/0bAjEw/Ldw0nV1c6V1dCz0vhR6Xewd7v7oHnjlMu/CsbPvg2N61n4NIo2Iwlmkjt311hJW5eTz3NWDaJ0SH94PLXsbpv0O9myGfj+CU27352rqqCjocQF0OxcWPAsz/wz/Gg5DrocRt0JsYt3XJNIAhXViysxGmdkKM8sys1sPsP4KM/um4vGFmfWu+VJF6r/X5mfz6rwN/PKU4xjWuVX1P5C3EV6+HF69EhJbwE8/gNH/9P82p0AMDLwGrp8HfS6DWQ/BYydC1gx/6xJpIKoNZzMLAI8CZwPdgcvMrHuVZmuAEc65E4B7gIk1XahIfbd4Yx6/f30RJ3Vsya/P6HLoxqEQfPkveHQQfPsRnHE3XPsxtB1YF6WGL7EFjHkUxk2FQCy8cCG89lPv9iwROWLh7DkPArKcc6udc6XAK8CYyg2cc18453ZWzM4BMmq2TJH6bWdBKT9/fj4tm8TyyOV9iT7UbVP5OfDSRfDu76DtYLhuDgz9lbe3Gqk6DIVfzIJTbvPOST92Eqx83++qROqtcMI5HdhQaT67YtnB/BR490ArzOxaM5tnZvNyc3PDr1KkHguWh7jh5a/J3VPC41f2p2VS3MEbZ30Ij58Maz+Hc/8OV06G5h3qrNajEh0Hp9wKP5sJTVp5XzCm/sbrrUxEDks44XygS0kPeP+VmZ2KF84TDrTeOTfROTfAOTegVaswzreJNAB3v72Uz7O28cexPel9sO45g6Xw/h3wwg8gsaUXcAN/Wj9vUzqmJ/zsIzjpeq9zlH8Nh01f+12VSL0STjhnA5X77ssANlVtZGYnAJOAMc657TVTnkj99tzstTw/Zx0/H96RiwccpAvMHavhqbPgi4dhwNVesLWpellHPRMTD2f9CX78pjfQxqQz4PN/eOfSRaRa4YTzXKCzmWWaWSxwKfBW5QZm1g6YAvzIObey5ssUqX8+XpHD3W8v5Yxanh1YAAASoElEQVTj2/C7gw0D+c1/4InhsONbuPh5OO8fDet2pI6neOeiu50HM+6CF8bCni0+FyUS+aoNZ+dcELgeeA9YBvzHObfEzMab2fiKZn8AWgKPmdlCM1PXX9Ko/W/DLn754gK6HZPMQ5f2IVC1o5GSPfD6eJjyM+8w8PhZ0H20P8XWtsQWcNEz3i1g67+Ex4foYjGRaqj7TpEatm57AT947AsSYgNM+eUQWidX6Whk09fe7UY718Dw33k9fQUaSX9AuSvgtath62I48ZdeF6DRh7hATqSBCbf7Tg0vI1KDtu4u5sdPfUXIOa8HsMrBHArB7Edh0pkQLIar3oZTb2s8wQzQqitc8yEM+jnMeQwmne6NNS0i36FwFqkh2/NLuGLSl2zbU8JT4wbSsVXS/pX5ufDSxfDe7dDlLBj/uXdvcGMUEw/n3AeXveL1gPav4fD1C+DTUTyRSKRwFqkBeYVl/OjJr9iwo5Anxw2kb7tKg1F8OxOeOBnWfArn/g0uecE7D9vYdT3bu1gsvT+8eR1M/ikU5/ldlUhEUDiLHKVdhaVc+eSXrMrZw79+1J8TO7b0VpSXwQd3wvNjvZGjrp3p9UddH+9dri0pad7tVqfd4Y1L/cRQ2DDX76pEfKdwFjkKOwpKuezfX7JiixfMp3RtXbFijXfv8qwHof84r1ORNj18rTViRQVg+C1w9XSve6OnzoLP/qZ7oqVRUziLHKGtu4u5bOIcVufm8++rBnBatzbeim/+C08Mg+1ZcNGzcP6DDeve5drSdhCM/8y7pezD/4PnL4Ddm/2uSsQXCmeRI7A6N58LH/+C7J2FPD1uICO6tIKind4tUlOu8faSx3/ujX0s4UtoBj982rsnOnuud65+xXS/qxKpcwpnkcP0TfYuLnpiNkWl5bx87YkM6ZQKqz/xBqxY+gac+v+8IRSbtfO71PrJDPr92BsiMzkNXr4E3p0AwRK/KxOpMwpnkcPw7qLNXPyv2STEBnjtF0M4oU08vPd7eG40xCTCTz+AEY2oU5Ha1KorXDMDBo+HL5/w7onOVe/A0jgonEXC4Jzj0ZlZ/OLFBXQ/NoU3rjuZzJLl8O9TYfYjMPBn8PNPIb2f36U2LDHxcPa9+++JnjgCvpyoi8WkwVM4i1QjvyTIdS8t4P73VjCmTxovjTuB1Fn/5420VLQLrngNzn1AF33Vpr33RLc7Ed79LTx9tvaipUHTsTeRQ8jKyWf8C/NZnZvP7ed042cZ2di/h8LOtdD/J3Dm3RDf1O8yG4eUNLhyCvzvZZh+m3ex2IgJcPJNEIjxuzqRGqU9Z5EDcM7x6tz1nP/Pz9lZUMorP+rCtbsewp4bDRYFV73j3SKlYK5bZtDncrjuK29v+qN7YOKpkD3f78pEapTCWaSKXYWl3PDy10yYvIh+bZOZecq3DHr7TK//5yE3eMM7Zg7zu8zGLbkNXPyc1xVqQS5MOs3rAjQ/1+/KRGqEDmuLVDJj6VZue30ROwtK+fuJhYzd8hfsw0XQYZh3YZJ6+Yosx58PmSPg0/tgzuOwtGKkr4HX6FC31Gsaz1kEyNldzJ+nLeONhZsY3rqIh1u/Q7Os1yElHUb+EXqMVZ/YkS53JUyfAN9+BK2O935vnU7X700iSrjjOWvPWRq10mCIp2et4eEPV5FSvpPXM2fSZ+sUrCAKht0Cw26G2CZ+lynhaNXFu2BsxTTvgrEXL4T2J8Ppd0K7wX5XJ3JYFM7SaH26Mpe73l7Cttwc7m0zk3MKXidqSyn0vRJG/A6aZvhdohwuM+h2LnQ6AxY8B5/cB0+NhM5nwel3wDG9/K5QJCw6rC2NzpJNefzjg5UsWJbFjckzucKmE1OaBz0vhFNuh9ROfpcoNaW0AL78lzc6WHEedB/j3XqV3t/vyqSRCvewtsJZGo0lm/J4aMYqli/7hl/GvcsPoz4hOlQC3c7z7pc99gS/S5TaUrTL68nty4lQkudd4Hfyr3ROWuqcwlmkwuKNeTw8YyU7ln/KNXEfMJIvsUA01vtSOOkG71ylNA4le2D+szD7UdizCVr3gJOu8y74Uw9vUgcUztKolQZDTF+yhddnLaLDxre5ImYmncjGxSZjA3/qDaaQcqzfZYpfgqWweDLMeghyl0FcU+h9idfrW5vuflcnDZjCWRqlLXnFvPzlGlZ+OZ0zSmdwXuAr4igleGw/ogdeDT1/oKuvZT/nYN0smP8MLH0TykshY5A3ZOXx53vjS4vUIIWzNBq7i8t4b9Emls39kHabpnNuYA6tLI9gTBKB3pdi/a/S+WSpXsF2r9/u+c/A9lUQiPWu+u55IXQZBXFJflcoDYDCWRq0gpIgny/byPKvptM0eyZn2FwybBtlFkdpxzNo0u9i6DxS5xHl8DkHmxbA4ineY88miE6ALiO9kO50JiS18rtKqacUztLgrNlWwJcLv6FgyXTabZ/FEPuGJlZC0GIpyBhKyoBLsK7nQHyK36VKQxEKwYY53vnpZW9D/lbAIK0vdDkLOp8Jx/SGgLqMkPAonKXe27SriMVLFpG37GOStsymR+ki2kV5AxvkxbahqMPptOp7PoHjTtEestS+UAi2fAOr3vce2fMAB3Ep3jjTHYZC+6FwrMJaDk7hLPVKcVk5K9dvZOuy2ZSun0eT7d/QJbiSNNsBQH5UMttTB5LS7RSa9zgDWnfX/anir4JtsPpjWPu5d1HZtpXe8tgkOLYPpPeFtH5ehyfN2unvVQCFs0Qo5xy5eQVsyFrC7vULKd+8lPhdK0kvXU2mbdnXLicmnT0tepFw3BDa9DqdQJvuEKURTiWC7dnqhfT62bBxgbeXXV7qrUts6Y1o1rp7pUc3iEv2t2apcwpn8Y1zjtxdu8nZsIo9G1dSmvstgV1rSMhfT8vSbNJcDrFWDkA5xtbodPJTOsGxfWje+URSu5yIJTb3eStEjlKwFHKWeEG96WvIWQo5y6GsYH+b5GOheSa06AgtOnjPe+d1G1eDVKOjUpnZKOAhIABMcs79tcp6q1h/DlAIjHPOLTjsqiXilZWVsW1bDru2baYgdwMlO7MJ7d5EdP4WYotySCzOoXn5Nlqzk9a2/4tfPgnkRqexp2lXVjQfRXxad1p27Evzdj1I0/liaYiiY70Lx9L67l8WCsGudZCzzAvrHathxxrImgH5W7778/HNICXNC/CUYyE5bf9z8jHeI6GF9z7S4FQbzmYWAB4FzgSygblm9pZzbmmlZmcDnSseg4HHK54lwrhQiOKiQgrzd1FSsJvigjxKC/MoKcijuGA3wYJdhIp2ECjcQaBkJ3FlecSX5dGkPI9kt5sUV8Cx5qjat9ZumrAjqiUFca3ZmtiZrc0yiEntREp6F1LbdSOpaRuSdM5NGruoKGiR6T26nfPddaUFsHOtF9Y7VnshvnuzdyvX1iVQkAMu9P3XjEuBxBbeofPElpCY6s3HN/UOm8eleM/xFc9xe5cnQ0yCzoVHqHD2nAcBWc651QBm9gowBqgczmOA55x3jHyOmTUzs2Odc5trvOII40IhQqEQzu1/9paVEwwGCQXLCAbLCJUHKQ+WEioPEiwrI1ReRmjv8vIyXHmQUDBIqDyIK69YXx7ElQdxoTIoLyNUVoILFuPKiiFYAsFirLwEC5Zg5SVElZcQKC8hENr7KCUQKiU6VEKcKybRFZFIMQlWTkI121VEHHmWQkFUCkXRKeQkHMPGuOaQ0ILopFTiUlqR2DKdlNbtad6mLSkJyegGJpGjENvEOy/dpseB15cHvYDeG9j5OVC4Awq373/k53h75YXboayw+veMioaYJhAT7wV1dELFdCJEVyyLSagynQCBmIpHrPcagVhvPirmEOui9y+zKIgKeM/7pgNhLA94XyYawReKcMI5HdhQaT6b7+8VH6hNOlAn4bx83ockTLsJcERVfLM0QhjOezi3f/oQjygcVLSPOsi6vcsDFYdsDe9Yvx/KXIBSYiixGMqIpdRiCVosZRZLMCqWEosnFNsUF4gjFJ1AeUwSLtZ7RMUlYfHJRCckE5XQlIQmTWmS3Iykpi1Iat6ahNiEagNcROpQINo7zJ2SBoQx5GV5mTfQR8nuiuc9ULy70rLd3nxZkRfkweKK6SJvungX7NkCwaL9y8uKoLyk1je1WhZ1gDCvHNxVni3q+8u+1+Yg666d6cuFe+GE84G+olS9iiycNpjZtcC1AO3atQvjrcMTl5jC9sRMHFFgtu8ZDLfvH9qL1f3Loyr9Evb/4tze+co/ZwZ7X7PiF217X7vyL73SvFnUvm+LFojGoqKh4tmbjyEqOgYLRBMVFY0FYogKRBMV7a0PVMwHYmK95VHRxMQnEBuXSEx8AnHxiURHxxBjhnqKFpHvCcRUHO5uUbOvGwpBxdE8ykshFKwyXVoxX1ap3d7pUgiVe4fn9z678irTocNb7kIV6/Yu83akvv9ceR2HWFfl2fzZ/QonnLOBtpXmM4BNR9AG59xEYCJ4V2sfVqWHkNl9IJnd366plxMRkYOJioKoOIiO87uSBi2cG0fnAp3NLNPMYoFLgbeqtHkL+LF5TgTyGsP5ZhERkdpQ7Z6zcy5oZtcD7+GdXn3KObfEzMZXrH8CmIZ3G1UW3q1UP6m9kkVERBq2sO5zds5NwwvgysueqDTtgOtqtjQREZHGSf0hioiIRBiFs4iISIRROIuIiEQYhbOIiEiEUTiLiIhEGN+GjDSzXGBdDb5kKrCtBl/Pbw1pe7QtkUnbEpm0LZGppralvXOuVXWNfAvnmmZm88IZI7O+aEjbo22JTNqWyKRtiUx1vS06rC0iIhJhFM4iIiIRpiGF80S/C6hhDWl7tC2RSdsSmbQtkalOt6XBnHMWERFpKBrSnrOIiEiD0KDC2cz6mNkcM1toZvPMbJDfNR0NM7vBzFaY2RIzu8/veo6Wmd1iZs7MUv2u5UiZ2f1mttzMvjGz182smd81HS4zG1Xxd5VlZrf6Xc+RMrO2ZjbTzJZVfEZu8rumo2VmATP72sze8buWo2FmzczstYrPyjIzO8nvmo6Umf264u9rsZm9bGbxdfG+DSqcgfuAu51zfYA/VMzXS2Z2KjAGOME51wN4wOeSjoqZtQXOBNb7XctR+gDo6Zw7AVgJ3OZzPYfFzALAo8DZQHfgMjPr7m9VRywI/MY5dzxwInBdPd6WvW4ClvldRA14CJjunOsG9KaebpOZpQM3AgOccz3xhk2+tC7eu6GFswNSKqabApt8rOVo/QL4q3OuBMA5l+NzPUfrH8Dv8H5H9ZZz7n3nXLBidg6Q4Wc9R2AQkOWcW+2cKwVewfsSWO845zY75xZUTO/BC4B0f6s6cmaWAZwLTPK7lqNhZinAcOBJAOdcqXNul79VHZVoIMHMooFE6ihXGlo4/wq438w24O1p1qu9miq6AMPM7Esz+8TMBvpd0JEys9HARufc//yupYZdDbzrdxGHKR3YUGk+m3ocaHuZWQegL/Clv5UclQfxvsCG/C7kKHUEcoGnKw7RTzKzJn4XdSSccxvxsmQ9sBnIc869XxfvHV0Xb1KTzGwGcMwBVv0eOB34tXNuspldjPfN7Yy6rO9wVLMt0UBzvMN1A4H/mFlHF6GX11ezLbcDI+u2oiN3qG1xzr1Z0eb3eIdVX6zL2mqAHWBZRP5NhcvMkoDJwK+cc7v9rudImNl5QI5zbr6ZneJ3PUcpGugH3OCc+9LMHgJuBe7wt6zDZ2bN8Y4sZQK7gP+a2ZXOuRdq+73rXTg75w4atmb2HN45G4D/EuGHh6rZll8AUyrC+CszC+H17ZpbV/UdjoNti5n1wvvD/p+ZgXcYeIGZDXLObanDEsN2qN8LgJldBZwHnB6pX5YOIRtoW2k+g3p8+sfMYvCC+UXn3BS/6zkKJwOjzewcIB5IMbMXnHNX+lzXkcgGsp1ze49ivIYXzvXRGcAa51wugJlNAYYAtR7ODe2w9iZgRMX0acAqH2s5Wm/gbQNm1gWIpR52IO+cW+Sca+2c6+Cc64D3we0XqcFcHTMbBUwARjvnCv2u5wjMBTqbWaaZxeJd3PKWzzUdEfO+7T0JLHPO/d3veo6Gc+4251xGxWfkUuCjehrMVHy2N5hZ14pFpwNLfSzpaKwHTjSzxIq/t9Opo4vb6t2eczV+BjxUceK+GLjW53qOxlPAU2a2GCgFrqqHe2kN0SNAHPBBxZGAOc658f6WFD7nXNDMrgfew7vy9Cnn3BKfyzpSJwM/AhaZ2cKKZbc756b5WJN4bgBerPgCuBr4ic/1HJGKw/KvAQvwTmN9TR31FKYewkRERCJMQzusLSIiUu8pnEVERCKMwllERCTCKJxFREQijMJZREQkwiicRUREIozCWUREJMIonEVERCLM/wfG86lw//sSvQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = torch.arange(-8.0,8.0,0.1)\n",
    "x.requires_grad_(True)\n",
    "y = torch.sigmoid(x)\n",
    "y.backward(torch.ones_like(x))\n",
    "\n",
    "with torch.no_grad():\n",
    "    plt.figure(figsize=(8,4))\n",
    "    plt.plot(x.detach().numpy(),y.detach().numpy())\n",
    "    plt.plot(x.detach().numpy(),x.grad.numpy())\n",
    "    plt.legend(['sigmoid','gradient'])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the gradient of the sigmoid vanishes\n",
    "both when its inputs are large and when they are small.\n",
    "Moreover, when we excecute backward propagation, due to the chain rule,\n",
    "this means that unless we are in the Goldilocks zone,\n",
    "where the inputs to most of the sigmoids are in the range of, say $[-4, 4]$,\n",
    "the gradients of the overall product may vanish.\n",
    "When we have many layers, unless we are especially careful,\n",
    "we are likely to find that our gradient is cut off at *some* layer.\n",
    "Before ReLUs ($\\max(0,x)$) were proposed\n",
    "as an alternative to squashing functions,\n",
    "this problem used to plague deep network training.\n",
    "As a consequence, ReLUs have become\n",
    "the default choice when designing activation functions in deep networks.\n",
    "\n",
    "\n",
    "### Exploding Gradients\n",
    "\n",
    "The opposite problem, when gradients explode,\n",
    "can be similarly vexing.\n",
    "To illustrate this a bit better,\n",
    "we draw $100$ Gaussian random matrices\n",
    "and multiply them with some initial matrix.\n",
    "For the scale that we picked\n",
    "(the choice of the variance $\\sigma^2=1$),\n",
    "the matrix product explodes.\n",
    "If this were to happen to us with a deep network,\n",
    "we would have no realistic chance of getting\n",
    "a gradient descent optimizer to converge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A single matrix \n",
      " tensor([[0.8210, 0.3891, 0.5661, 0.1407],\n",
      "        [0.5208, 0.7289, 0.4124, 0.7274],\n",
      "        [0.5456, 0.8456, 0.3011, 0.9407],\n",
      "        [0.4575, 0.4993, 0.4596, 0.8236]])\n",
      "After multiplying 100 matrices\n",
      " tensor([[4.1657e+26, 2.5340e+26, 1.4468e+27, 4.0063e+26],\n",
      "        [4.8251e+26, 2.9351e+26, 1.6759e+27, 4.6405e+26],\n",
      "        [4.7927e+26, 2.9154e+26, 1.6646e+27, 4.6093e+26],\n",
      "        [4.3570e+26, 2.6504e+26, 1.5133e+27, 4.1904e+26]])\n"
     ]
    }
   ],
   "source": [
    "M = torch.rand(4,4)\n",
    "print('A single matrix \\n',M)\n",
    "for i in range(100):\n",
    "    M = torch.mm(M,torch.randn(4,4))\n",
    "print('After multiplying 100 matrices\\n',M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Symmetry\n",
    "\n",
    "Another problem in deep network design\n",
    "is the symmetry inherent in their parametrization.\n",
    "Assume that we have a deep network\n",
    "with one hidden layer with two units, say $h_1$ and $h_2$.\n",
    "In this case, we could permute the weights $\\mathbf{W}_1$\n",
    "of the first layer and likewise permute the weights of the output layer\n",
    "to obtain the same function function.\n",
    "There is nothing special differentiating\n",
    "the first hidden unit vs the second hidden unit.\n",
    "In other words, we have permutation symmetry\n",
    "among the hidden units of each layer.\n",
    "\n",
    "This is more than just a theoretical nuisance.\n",
    "Imagine what would happen if we initialized\n",
    "all of the parameters of some layer as $\\mathbf{W}_l = c$\n",
    "for some constant $c$.\n",
    "In this case, the gradients for all dimensions are identical:\n",
    "thus not only would each unit take the same value,\n",
    "but it would receive the same update.\n",
    "Stochastic gradient descent would never break the symmetry on its own\n",
    "and we might never be able to realize the networks expressive power.\n",
    "The hidden layer would behave as if it had only a single unit.\n",
    "As an aside, note that while SGD would not break this symmetry,\n",
    "dropout regularization would!\n",
    "\n",
    "\n",
    "\n",
    "## Parameter Initialization\n",
    "\n",
    "One way of addressing, or at least mitigating the issues raised above\n",
    "is through careful initialization of the weight vectors.\n",
    "This way we can ensure that (at least initially) the gradients do not vanish a\n",
    "and that they maintain a reasonable scale\n",
    "where the network weights do not diverge.\n",
    "Additional care during optimization and suitable regularization\n",
    "ensures that things never get too bad.\n",
    "\n",
    "\n",
    "### Default Initialization\n",
    "\n",
    "Each Pytorch layer implements the method `reset_parameters`\n",
    "which is called at the end of the layer\n",
    "initialization to initialize the weights.\n",
    "\n",
    "\n",
    "If the user wishes to initialize manually, for Single\n",
    "Layer, functions from `torch.nn.init` \n",
    "are used. For instance: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[[[ 0.5463]],\n",
       "\n",
       "         [[ 0.0279]]],\n",
       "\n",
       "\n",
       "        [[[ 1.1137]],\n",
       "\n",
       "         [[-0.0143]]]], requires_grad=True)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conv1 = torch.nn.Conv2d(2,2,1)\n",
    "torch.nn.init.xavier_uniform_(conv1.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A function can be passed to `torch.nn.Module.apply()`. It \n",
    "will recursively initialize the weights of `nn.Module` ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=2, out_features=2, bias=True)\n",
       "  (1): Linear(in_features=2, out_features=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "def init_weights(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        torch.nn.init.xavier_uniform_(m.weight)\n",
    "        m.bias.data.fill_(0.01)\n",
    "\n",
    "net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))\n",
    "net.apply(init_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Xavier Initialization\n",
    "\n",
    "Let's look at the scale distribution of the activations of the hidden units $h_{i}$ for some layer. They are given by\n",
    "\n",
    "$$h_{i} = \\sum_{j=1}^{n_\\mathrm{in}} W_{ij} x_j$$\n",
    "\n",
    "The weights $W_{ij}$ are all drawn independently from the same distribution. Furthermore, let's assume that this distribution\n",
    "has zero mean and variance $\\sigma^2$\n",
    "(this doesn't mean that the distribution has to be Gaussian,\n",
    "just that mean and variance need to exist).\n",
    "We don't really have much control\n",
    "over the inputs into the layer $x_j$\n",
    "but let's proceed with the somewhat unrealistic assumption\n",
    "that they also have zero mean and variance $\\gamma^2$\n",
    "and that they're independent of $\\mathbf{W}$.\n",
    "In this case, we can compute mean and variance of $h_i$ as follows:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{E}[h_i] & = \\sum_{j=1}^{n_\\mathrm{in}} \\mathbf{E}[W_{ij} x_j] = 0 \\\\\n",
    "    \\mathbf{E}[h_i^2] & = \\sum_{j=1}^{n_\\mathrm{in}} \\mathbf{E}[W^2_{ij} x^2_j] \\\\\n",
    "        & = \\sum_{j=1}^{n_\\mathrm{in}} \\mathbf{E}[W^2_{ij}] \\mathbf{E}[x^2_j] \\\\\n",
    "        & = n_\\mathrm{in} \\sigma^2 \\gamma^2\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "One way to keep the variance fixed is to set $n_\\mathrm{in} \\sigma^2 = 1$.\n",
    "Now consider backpropagation.\n",
    "There we face a similar problem,\n",
    "albeit with gradients being propagated from the top layers.\n",
    "That is, instead of $\\mathbf{W} \\mathbf{w}$,\n",
    "we need to deal with $\\mathbf{W}^\\top \\mathbf{g}$,\n",
    "where $\\mathbf{g}$ is the incoming gradient from the layer above.\n",
    "Using the same reasoning as for forward propagation,\n",
    "we see that the gradients' variance can blow up\n",
    "unless $n_\\mathrm{out} \\sigma^2 = 1$.\n",
    "This leaves us in a dilemma:\n",
    "we cannot possibly satisfy both conditions simultaneously.\n",
    "Instead, we simply try to satisfy:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{1}{2} (n_\\mathrm{in} + n_\\mathrm{out}) \\sigma^2 = 1 \\text{ or equivalently }\n",
    "\\sigma = \\sqrt{\\frac{2}{n_\\mathrm{in} + n_\\mathrm{out}}}\n",
    "\\end{aligned}.\n",
    "$$\n",
    "\n",
    "This is the reasoning underlying the eponymous Xavier initialization,\n",
    "proposed by [Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) in 2010.\n",
    "It works well enough in practice.\n",
    "For Gaussian random variables, the Xavier initialization\n",
    "picks a normal distribution with zero mean\n",
    "and variance $\\sigma^2 = 2/(n_\\mathrm{in} + n_\\mathrm{out})$.\n",
    "For uniformly distributed random variables $U[-a, a]$,\n",
    "note that their variance is given by $a^2/3$.\n",
    "Plugging $a^2/3$ into the condition on $\\sigma^2$ yields\n",
    "that we should initialize uniformly with\n",
    "$U\\left[-\\sqrt{6/(n_\\mathrm{in} + n_\\mathrm{out})}, \\sqrt{6/(n_\\mathrm{in} + n_\\mathrm{out})}\\right]$.\n",
    "\n",
    "### Beyond\n",
    "\n",
    "The reasoning above barely scratches the surface\n",
    "of modern approaches to parameter initialization.\n",
    "In fact, Pytorch has an entire `torch.nn.init` module\n",
    "implementing over a dozen different heuristics.\n",
    "Moreover, intialization continues to be a hot area of inquiry\n",
    "within research into the fundamental theory of neural network optimization.\n",
    "Some of these heuristics are especially suited\n",
    "for when parameters are tied\n",
    "(i.e., when parameters of in different parts the network are shared),\n",
    "for superresolution, sequence models, and related problems.\n",
    "We recommend that the interested reader take a closer look\n",
    "at what is offered as part of this module,\n",
    "and investigate the recent research on parameter initialization.\n",
    "Perhaps you may come across a recent clever idea\n",
    "and contribute its implementation to Pytorch,\n",
    "or you may even invent your own scheme!\n",
    "\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Vanishing and exploding gradients are common issues in very deep networks, unless great care is taking to ensure that gradients and parameters remain well controlled.\n",
    "* Initialization heuristics are needed to ensure that at least the initial gradients are neither too large nor too small.\n",
    "* The ReLU addresses one of the vanishing gradient problems, namely that gradients vanish for very large inputs. This can accelerate convergence significantly.\n",
    "* Random initialization is key to ensure that symmetry is broken before optimization.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Can you design other cases of symmetry breaking besides the permutation symmetry?\n",
    "1. Can we initialize all weight parameters in linear regression or in softmax regression to the same value?\n",
    "1. Look up analytic bounds on the eigenvalues of the product of two matrices. What does this tell you about ensuring that gradients are well conditioned?\n",
    "1. If we know that some terms diverge, can we fix this after the fact? Look at the paper on LARS by [You, Gitman and Ginsburg, 2017](https://arxiv.org/pdf/1708.03888.pdf) for inspiration.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}