chapter_multilayer-perceptrons/mlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multilayer Perceptrons\n",
    "\n",
    ":label:`sec_mlp`\n",
    "\n",
    "\n",
    "In the previous chapter, we introduced \n",
    "softmax regression (:numref:`sec_softmax`), \n",
    "implementing the algorithm from scratch \n",
    "(:numref:`sec_softmax_scratch`) and in DJL \n",
    "(:numref:`sec_softmax_djl`)\n",
    "and training classifiers to recognize \n",
    "10 categories of clothing from low-resolution images.\n",
    "Along the way, we learned how to wrangle data,\n",
    "coerce our outputs into a valid probability distribution \n",
    "(via `softmax`),\n",
    "apply an appropriate loss function,\n",
    "and to minimize it with respect to our model's parameters.\n",
    "Now that we have mastered these mechanics\n",
    "in the context of simple linear models,\n",
    "we can launch our exploration of deep neural networks,\n",
    "the comparatively rich class of models \n",
    "with which this book is primarily concerned.\n",
    "\n",
    "## Hidden Layers\n",
    "\n",
    "To begin, recall the model architecture \n",
    "corresponding to our softmax regression example,\n",
    "illustrated in  :numref:`fig_singlelayer` below.\n",
    "This model mapped our inputs directly to our outputs\n",
    "via a single linear transformation:\n",
    "\n",
    "$$\n",
    "\\hat{\\mathbf{o}} = \\mathrm{softmax}(\\mathbf{W} \\mathbf{x} + \\mathbf{b}).\n",
    "$$\n",
    "\n",
    "If our labels truly were related \n",
    "to our input data by a linear function, \n",
    "then this approach would be sufficient.\n",
    "But linearity is a *strong assumption*.\n",
    "\n",
    "For example, linearity implies the *weaker*\n",
    "assumption of *monotonicity*: \n",
    "that any increase in our feature must\n",
    "either always cause an increase in our model's output \n",
    "(if the corresponding weight is positive),\n",
    "or always cause a decrease in our model's output\n",
    "(if the corresponding weight is negative).\n",
    "Sometimes that makes sense. \n",
    "For example, if we were trying to predict \n",
    "whether an individual will repay a loan,\n",
    "we might reasonably imagine that holding all else equal,\n",
    "an applicant with a higher income\n",
    "would always be more likely to repay \n",
    "than one with a lower income.\n",
    "While monotonic, this relationship likely\n",
    "is not linearly associated with the probability of \n",
    "repayment. An increase in income from $0 to $50k \n",
    "likely corresponds to a bigger increase \n",
    "in likelihood of repayment \n",
    "than an increase from $1M to $1.05M.\n",
    "One way to handle this might be to pre-process\n",
    "our data such that linearity becomes more plausible, \n",
    "say, by using the logarithm of income as our feature.\n",
    "\n",
    "\n",
    "Note that we can easily come up with examples\n",
    "that violate *monotonicity*.\n",
    "Say for example that we want to predict probability\n",
    "of death based on body temperature. \n",
    "For individuals with a body temperature \n",
    "above 37°C (98.6°F),\n",
    "higher termperatures indicate greater risk.\n",
    "However, for individuals with body termperatures\n",
    "below 37° C, higher temperatures indicate *lower* risk! \n",
    "In this case too, we might resolve the problem \n",
    "with some clever preprocessing. \n",
    "Namely, we might use the *distance* from 37°C as our feature. \n",
    "\n",
    "\n",
    "But what about classifying images of cats and dogs?\n",
    "Should increasing the intensity \n",
    "of the pixel at location (13, 17)\n",
    "always increase (or always decrease) \n",
    "the likelihood that the image depicts a dog?\n",
    "Reliance on a linear model corrsponds to the (implicit)\n",
    "assumption that the only requirement\n",
    "for differentiating cats vs. dogs is to assess \n",
    "the brightness of individual pixels.\n",
    "This approach is doomed to fail in a world\n",
    "where inverting an image preserves the category.\n",
    "\n",
    "\n",
    "And yet despite the apparent absurdity of linearity here,\n",
    "as compared to our previous examples, \n",
    "it is less obvious that we could address the problem\n",
    "with a simple preprocessing fix. \n",
    "That is because the significance of any pixel\n",
    "depends in complex ways on its context \n",
    "(the values of the surrounding pixels).\n",
    "While there might exist a representation of our data\n",
    "that would take into account \n",
    "the relevant interactions among our features \n",
    "(and on top of which a linear model would be suitable),\n",
    "we simply do not know how to calculate it by hand.\n",
    "With deep neural networks, we used observational data\n",
    "to jointly learn both a representation (via hidden layers)\n",
    "and a linear predictor that acts upon that representation.\n",
    "\n",
    "\n",
    "### Incorporating Hidden Layers\n",
    "\n",
    "We can overcome these limitations of linear models\n",
    "and handle a more general class of functions\n",
    "by incorporating one or more hidden layers.\n",
    "The easiest way to do this is to stack\n",
    "many fully-connected layers on top of each other.\n",
    "Each layer feeds into the layer above it, \n",
    "until we generate an output.\n",
    "We can think of the first $L-1$ layers \n",
    "as our representation and the final layer\n",
    "as our linear predictor. \n",
    "This architecture is commonly called \n",
    "a *multilayer perceptron*,\n",
    "often abbreviated as *MLP*.\n",
    "Below, we depict an MLP diagramtically (:numref:`fig_mlp`).\n",
    "\n",
    "![Multilayer perceptron with hidden layers. This example contains a hidden layer with 5 hidden units in it. ](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/mlp.svg)\n",
    "\n",
    ":label:`fig_mlp`\n",
    "\n",
    "\n",
    "This multilayer perceptron has 4 inputs, 3 outputs,\n",
    "and its hidden layer contains 5 hidden units.\n",
    "Since the input layer does not involve any calculations,\n",
    "producing outputs with this network\n",
    "requires implementing the computations \n",
    "for each of the 2 layers (hidden and output).\n",
    "Note that these layers are both fully connected.\n",
    "Every input influences every neuron in the hidden layer,\n",
    "and each of these in turn influences \n",
    "every neuron in the output layer.\n",
    "\n",
    "\n",
    "### From Linear to Nonlinear\n",
    "\n",
    "Formally, we calculate each layer \n",
    "in this one-hidden-layer MLP as follows:\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{h} & = \\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1, \\\\\n",
    "    \\mathbf{o} & = \\mathbf{W}_2 \\mathbf{h} + \\mathbf{b}_2, \\\\\n",
    "    \\hat{\\mathbf{y}} & = \\mathrm{softmax}(\\mathbf{o}).\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "Note that after adding this layer,\n",
    "our model now requires us to track and update\n",
    "two additional sets of parameters.\n",
    "So what have we gained in exchange?\n",
    "You might be surprised to find out \n",
    "that---in the model defined above---*we \n",
    "gain nothing for our troubles!*\n",
    "The reason is plain. \n",
    "The hidden units above are given by \n",
    "a linear function of the inputs,\n",
    "and the outputs (pre-softmax) are just \n",
    "a linear function of the hidden units.\n",
    "A linear function of a linear function \n",
    "is itself a linear function.\n",
    "Moreover, our linear model was already \n",
    "capable of representing any linear function.\n",
    "\n",
    "\n",
    "We can view the equivalence formally\n",
    "by proving that for any values of the weights,\n",
    "we can just collapse out the hidden layer,\n",
    "yielding an equivalent single-layer model with paramters\n",
    "$\\mathbf{W} = \\mathbf{W}_2 \\mathbf{W}_1$ and $\\mathbf{b} = \\mathbf{W}_2 \\mathbf{b}_1 + \\mathbf{b}_2$.\n",
    "\n",
    "$$\\mathbf{o} = \\mathbf{W}_2 \\mathbf{h} + \\mathbf{b}_2 = \\mathbf{W}_2 (\\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1) + \\mathbf{b}_2 = (\\mathbf{W}_2 \\mathbf{W}_1) \\mathbf{x} + (\\mathbf{W}_2 \\mathbf{b}_1 + \\mathbf{b}_2) = \\mathbf{W} \\mathbf{x} + \\mathbf{b}.$$\n",
    "\n",
    "In order to realize the potential of multilayer architectures,\n",
    "we need one more key ingredient---an \n",
    "elementwise *nonlinear activation function* $\\sigma$ \n",
    "to be applied to each hidden unit \n",
    "(following the linear transformation).\n",
    "The most popular choice for the nonlinearity \n",
    "these days is the rectified linear unit (ReLU) \n",
    "$\\mathrm{max}(x, 0)$.\n",
    "In general, with these activation functions in place,\n",
    "it is no longer possible to collapse our MLP into a linear model.\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{h} & = \\sigma(\\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1), \\\\\n",
    "    \\mathbf{o} & = \\mathbf{W}_2 \\mathbf{h} + \\mathbf{b}_2, \\\\\n",
    "    \\hat{\\mathbf{y}} & = \\mathrm{softmax}(\\mathbf{o}).\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "To build more general MLPs, we can continue stacking \n",
    "such hidden layers,\n",
    "e.g., $\\mathbf{h}_1 = \\sigma(\\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1)$\n",
    "and $\\mathbf{h}_2 = \\sigma(\\mathbf{W}_2 \\mathbf{h}_1 + \\mathbf{b}_2)$,\n",
    "one atop another, yielding ever more expressive models\n",
    "(assuming fixed width). \n",
    "\n",
    "MLPs can capture complex interactions \n",
    "among our inputs via their hidden neurons, \n",
    "which depend on the values of each of the inputs.\n",
    "We can easily design hidden nodes \n",
    "to perform arbitrary computation,\n",
    "for instance, basic logic operations on a pair of inputs.\n",
    "Moreover, for certain choices of the activation function,\n",
    "it is widely known that MLPs are universal approximators.\n",
    "Even with a single-hidden-layer network,\n",
    "given enough nodes (possibly absurdly many), \n",
    "and the right set of weights,\n",
    "we can model any function.\n",
    "*Actually learning that function is the hard part.*\n",
    "You might think of your neural network \n",
    "as being a bit like the C programming language. \n",
    "The language, like any other modern language,\n",
    "is capable of expressing any computable program.\n",
    "But actually coming up with a program \n",
    "that meets your specifications is the hard part.\n",
    "\n",
    "Moreover, just because a single-layer network \n",
    "*can* learn any function \n",
    "does not mean that you should try \n",
    "to solve all of your problems \n",
    "with single-layer networks.\n",
    "In fact, we can approximate many functions\n",
    "much more compactly by using deeper (vs wider) networks.\n",
    "We will touch upon more rigorous arguments in subsequent chapters,\n",
    "but first let us actually build an MLP in code.\n",
    "In this example, we’ll implement an MLP\n",
    "with two hidden layers and one output layer.\n",
    "\n",
    "### Vectorization and Minibatch\n",
    "\n",
    "As before, by the matrix $\\mathbf{X}$, \n",
    "we denote a minibatch of inputs.\n",
    "The calculations to produce outputs \n",
    "from an MLP with two hidden layers\n",
    "can thus be expressed:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{H}_1 & = \\sigma(\\mathbf{W}_1 \\mathbf{X} + \\mathbf{b}_1), \\\\\n",
    "    \\mathbf{H}_2 & = \\sigma(\\mathbf{W}_2 \\mathbf{H}_1 + \\mathbf{b}_2), \\\\\n",
    "    \\mathbf{O} & = \\mathrm{softmax}(\\mathbf{W}_3 \\mathbf{H}_2 + \\mathbf{b}_3).\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "With some abuse of notation, we define the nonlinearity \n",
    "$\\sigma$ to apply to its inputs in a row-wise fashion, \n",
    "i.e., one observation at a time.\n",
    "Note that we are also using the notation for *softmax* \n",
    "in the same way to denote a row-wise operation.\n",
    "Often, as in this section, the activation functions\n",
    "that we apply to hidden layers are not merely row-wise, \n",
    "but component wise.\n",
    "That means that after computing the linear portion of the layer,\n",
    "we can calculate each nodes activation \n",
    "without looking at the values taken by the other hidden units.\n",
    "This is true for most activation functions\n",
    "(the batch normalization operation to be introduced in :numref:`sec_batch_norm` is a notable exception to that rule). \n",
    "\n",
    "Now we will load the DJL and other relevant utility libraries to setup our environment which would enable us to run our code in the subsequent sections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load ../utils/djl-imports\n",
    "%load ../utils/plot-utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation Functions\n",
    "\n",
    "Activation functions decide whether a neuron should be activated or not by \n",
    "calculating the weighted sum and further adding bias with it. \n",
    "They are differentiable operators to transform input signals to outputs, \n",
    "while most of them add non-linearity.\n",
    "Because activation functions are fundamental to deep learning, \n",
    "let us briefly survey some common activation functions.\n",
    "\n",
    "### ReLU Function\n",
    "\n",
    "As stated above, the most popular choice,\n",
    "due to both simplicity of implementation\n",
    "its performance on a variety of predictive tasks \n",
    "is the rectified linear unit (ReLU).\n",
    "ReLU provides a very simple nonlinear transformation.\n",
    "Given the element $z$, the function is defined\n",
    "as the maximum of that element and 0.\n",
    "\n",
    "$$\\mathrm{ReLU}(z) = \\max(z, 0).$$\n",
    "\n",
    "Informally, the ReLU function retains only positive\n",
    "elements and discards all negative elements \n",
    "(setting the corresponding activations to 0).\n",
    "To gain some intuition, we can plot the function.\n",
    "Because it is used so commonly, `DJL` \n",
    "supports the `relu` function as a native operator.\n",
    "As you can see, the activation function is piecewise linear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "2"
    }
   },
   "outputs": [],
   "source": [
    "NDManager manager = NDManager.newBaseManager();\n",
    "NDArray x = manager.arange(-8.0f, 8.0f, 0.1f);\n",
    "x.setRequiresGradient(true);\n",
    "NDArray y = Activation.relu(x);\n",
    "\n",
    "// Converting the data into float arrays to render them in a plot.\n",
    "float[] X = x.toFloatArray();\n",
    "float[] Y = y.toFloatArray();\n",
    "\n",
    "Table data = Table.create(\"Data\").addColumns(\n",
    "    FloatColumn.create(\"X\", X), \n",
    "    FloatColumn.create(\"relu(x)\", Y)\n",
    ");\n",
    "render(LinePlot.create(\"\", data, \"x\", \"relu(X)\"), \"text/html\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the input is negative, \n",
    "the derivative of the ReLU function is 0,\n",
    "and when the input is positive, \n",
    "the derivative of the ReLU function is 1.\n",
    "Note that the ReLU function is not differentiable\n",
    "when the input takes value precisely equal to 0.\n",
    "In these cases, we default to the left-hand-side (LHS) \n",
    "derivative and say that the derivative is 0 when the input is 0.\n",
    "We can get away with this because \n",
    "the input may never actually be zero.\n",
    "There is an old adage that if subtle boundary conditions matter,\n",
    "we are probably doing (*real*) mathematics, not engineering.\n",
    "That conventional wisdom may apply here.\n",
    "We plot the derivative of the ReLU function plotted below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "9"
    }
   },
   "outputs": [],
   "source": [
    "try(GradientCollector collector = Engine.getInstance().newGradientCollector()){\n",
    "    y = Activation.relu(x);\n",
    "    collector.backward(y); \n",
    "}\n",
    "\n",
    "NDArray res = x.getGradient();\n",
    "float[] X = x.toFloatArray();\n",
    "float[] Y = res.toFloatArray();\n",
    "\n",
    "Table data = Table.create(\"Data\").addColumns(\n",
    "    FloatColumn.create(\"X\", X), \n",
    "    FloatColumn.create(\"grad of relu\", Y)\n",
    ");\n",
    "render(LinePlot.create(\"\", data, \"x\", \"grad of relu\"), \"text/html\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there are many variants to the ReLU function, \n",
    "including the parameterized ReLU (pReLU) of [He et al., 2015](https://arxiv.org/abs/1502.01852). \n",
    "This variation adds a linear term to the ReLU, \n",
    "so some information still gets through, \n",
    "even when the argument is negative.\n",
    "\n",
    "$$\\mathrm{pReLU}(x) = \\max(0, x) + \\alpha \\min(0, x).$$\n",
    "\n",
    "The reason for using the ReLU is that \n",
    "its derivatives are particularly well behaved: \n",
    "either they vanish or they just let the argument through. \n",
    "This makes optimization better behaved \n",
    "and it mitigated the well-documented problem \n",
    "of *vanishing gradients* that plagued\n",
    "previous versions of neural networks (more on this later).\n",
    "\n",
    "### Sigmoid Function\n",
    "\n",
    "The sigmoid function transforms its inputs,\n",
    "for which values lie in the domain $\\mathbb{R}$,\n",
    "to outputs that lie on the interval $(0, 1)$.\n",
    "For that reason, the sigmoid is \n",
    "often called a *squashing* function:\n",
    "it *squashes* any input in the range (-inf, inf)\n",
    "to some value in the range (0, 1).\n",
    "\n",
    "$$\\mathrm{sigmoid}(x) = \\frac{1}{1 + \\exp(-x)}.$$\n",
    "\n",
    "In the earliest neural networks, scientists\n",
    "were interested in modeling biological neurons\n",
    "which either *fire* or *do not fire*.\n",
    "Thus the pioneers of this field, \n",
    "going all the way back to McCulloch and Pitts, \n",
    "the inventors of the artificial neuron,\n",
    "focused on thresholding units.\n",
    "A thresholding activation takes value $0$\n",
    "when its input is below some threshold\n",
    "and value $1$ when the input exceeds the threshold.\n",
    "\n",
    "\n",
    "When attention shifted to gradient based learning,\n",
    "the sigmoid function was a natural choice\n",
    "because it is a smooth, differentiable \n",
    "approximation to a thresholding unit.\n",
    "Sigmoids are still widely used as \n",
    "activation functions on the output units,\n",
    "when we want to interpret the outputs as probabilities\n",
    "for binary classification problems\n",
    "(you can think of the sigmoid as a special case of the softmax).\n",
    "However, the sigmoid has mostly been replaced \n",
    "by the simpler and more easily trainable ReLU \n",
    "for most use in hidden layers.\n",
    "In the \"Recurrent Neural Network\" chapter \n",
    "(:numref:`sec_plain_rnn`), \n",
    "we will describe architectures that leverage sigmoid units \n",
    "to control the flow of information across time.\n",
    "\n",
    "Below, we plot the sigmoid function.\n",
    "Note that when the input is close to 0, \n",
    "the sigmoid function approaches \n",
    "a linear transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "4"
    }
   },
   "outputs": [],
   "source": [
    "NDArray y = Activation.sigmoid(x);\n",
    "float[] Y = y.toFloatArray();\n",
    "\n",
    "Table data = Table.create(\"Data\").addColumns(\n",
    "    FloatColumn.create(\"X\", X), \n",
    "    FloatColumn.create(\"sigmoid(x)\", Y)\n",
    ");\n",
    "render(LinePlot.create(\"\", data, \"x\", \"sigmoid(X)\"));    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivative of the sigmoid function is given by the following equation:\n",
    "\n",
    "$$\\frac{d}{dx} \\mathrm{sigmoid}(x) = \\frac{\\exp(-x)}{(1 + \\exp(-x))^2} = \\mathrm{sigmoid}(x)\\left(1-\\mathrm{sigmoid}(x)\\right).$$\n",
    "\n",
    "\n",
    "The derivative of the sigmoid function is plotted below.\n",
    "Note that when the input is 0, \n",
    "the derivative of the sigmoid function\n",
    "reaches a maximum of 0.25. \n",
    "As the input diverges from 0 in either direction, \n",
    "the derivative approaches 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "11"
    }
   },
   "outputs": [],
   "source": [
    "try(GradientCollector collector = Engine.getInstance().newGradientCollector()){\n",
    "    y = Activation.sigmoid(x);\n",
    "    collector.backward(y); \n",
    "}\n",
    "\n",
    "NDArray res = x.getGradient();\n",
    "float[] X = x.toFloatArray();\n",
    "float[] Y = res.toFloatArray();\n",
    "\n",
    "Table data = Table.create(\"Data\").addColumns(\n",
    "    FloatColumn.create(\"X\", X), \n",
    "    FloatColumn.create(\"grad of sigmoid\", Y)\n",
    ");\n",
    "render(LinePlot.create(\"\", data, \"x\", \"grad of sigmoid\"), \"text/html\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tanh Function\n",
    "\n",
    "Like the sigmoid function, the tanh (Hyperbolic Tangent)\n",
    "function also squashes its inputs,\n",
    "transforming them into elements on the interval between -1 and 1:\n",
    "\n",
    "$$\\text{tanh}(x) = \\frac{1 - \\exp(-2x)}{1 + \\exp(-2x)}.$$\n",
    "\n",
    "We plot the tanh function blow. Note that as the input nears 0, the tanh function approaches a linear transformation. Although the shape of the function is similar to the sigmoid function, the tanh function exhibits point symmetry about the origin of the coordinate system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "12"
    }
   },
   "outputs": [],
   "source": [
    "NDArray y = Activation.tanh(x);\n",
    "float[] Y = y.toFloatArray();\n",
    "\n",
    "Table data = Table.create(\"Data\").addColumns(\n",
    "    FloatColumn.create(\"X\", X), \n",
    "    FloatColumn.create(\"tanh(x)\", Y)\n",
    ");\n",
    "\n",
    "render(LinePlot.create(\"\", data, \"x\", \"tanh(X)\"));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivative of the Tanh function is:\n",
    "\n",
    "$$\\frac{d}{dx} \\mathrm{tanh}(x) = 1 - \\mathrm{tanh}^2(x).$$\n",
    "\n",
    "The derivative of tanh function is plotted below.\n",
    "As the input nears 0,\n",
    "the derivative of the tanh function approaches a maximum of 1.\n",
    "And as we saw with the sigmoid function,\n",
    "as the input moves away from 0 in either direction,\n",
    "the derivative of the tanh function approaches 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "13"
    }
   },
   "outputs": [],
   "source": [
    "try(GradientCollector collector = Engine.getInstance().newGradientCollector()){\n",
    "    y = Activation.tanh(x);\n",
    "    collector.backward(y);  \n",
    "}\n",
    "\n",
    "NDArray res = x.getGradient();\n",
    "float[] X = x.toFloatArray();\n",
    "float[] Y = res.toFloatArray();\n",
    "\n",
    "Table data = Table.create(\"Data\").addColumns(\n",
    "    FloatColumn.create(\"X\", X), \n",
    "    FloatColumn.create(\"grad of tanh\", Y)\n",
    ");\n",
    "render(LinePlot.create(\"\", data, \"x\", \"grad of tanh\"), \"text/html\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, we now know how to incorporate nonlinearities\n",
    "to build expressive multilayer neural network architectures.\n",
    "As a side note, your knowledge already\n",
    "puts you in command of a similar toolkit \n",
    "to a practitioner circa 1990.\n",
    "In some ways, you have an advantage \n",
    "over anyone working in the 1990s,\n",
    "because you can leverage powerful \n",
    "open-source deep learning frameworks\n",
    "to build models rapidly, using only a few lines of code.\n",
    "Previously, getting these nets training\n",
    "required researchers to code up \n",
    "thousands of lines of C and Fortran.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* The multilayer perceptron adds one or multiple fully-connected hidden layers between the output and input layers and transforms the output of the hidden layer via an activation function.\n",
    "* Commonly-used activation functions include the ReLU function, the sigmoid function, and the tanh function.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Compute the derivative of the tanh and the pReLU activation function.\n",
    "1. Show that a multilayer perceptron using only ReLU (or pReLU) constructs a continuous piecewise linear function.\n",
    "1. Show that $\\mathrm{tanh}(x) + 1 = 2 \\mathrm{sigmoid}(2x)$.\n",
    "1. Assume we have a multilayer perceptron *without* nonlinearities between the layers. In particular, assume that we have $d$ input dimensions, $d$ output dimensions and that one of the layers has only $d/2$ dimensions. Show that this network is less expressive (powerful) than a single layer perceptron.\n",
    "1. Assume that we have a nonlinearity that applies to one minibatch at a time. What kinds of problems do you expect this to cause?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "14.0.2+12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}