NLP_C2_W4_lecture_notebook_model_architecture.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dr-mushtaq/Natural-language-processing/blob/main/NLP_C2_W4_lecture_notebook_model_architecture.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v50v6lDf9NqA"
      },
      "source": [
        "# Word Embeddings: Intro to CBOW model, activation functions and working with Numpy\n",
        "\n",
        "In this lecture notebook you will be given an introduction to the continuous bag-of-words model, its activation functions and some considerations when working with Numpy.\n",
        "\n",
        "Let's dive into it!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MQwmJW8b9NqE"
      },
      "outputs": [],
      "source": [
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nTPPLH7T9NqG"
      },
      "source": [
        "# The continuous bag-of-words model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qXhfaWUR9NqG"
      },
      "source": [
        "The CBOW model is based on a neural network, the architecture of which looks like the figure below, as you'll recall from the lecture.\n",
        "\n",
        "<div style=\"width:image width px; font-size:100%; text-align:center;\"><img src='cbow_model_architecture.png' alt=\"alternate text\" width=\"width\" height=\"height\" style=\"width:917;height:337;\" /> Figure 1 </div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fVOD-xqh9NqH"
      },
      "source": [
        "## Activation functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Nyi7bTs9NqH"
      },
      "source": [
        "Let's start by implementing the activation functions, ReLU and softmax."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HC3vCtls9NqH"
      },
      "source": [
        "### ReLU"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BZp1-QP09NqI"
      },
      "source": [
        "ReLU is used to calculate the values of the hidden layer, in the following formulas:\n",
        "\n",
        "\\begin{align}\n",
        " \\mathbf{z_1} &= \\mathbf{W_1}\\mathbf{x} + \\mathbf{b_1}  \\tag{1} \\\\\n",
        " \\mathbf{h} &= \\mathrm{ReLU}(\\mathbf{z_1})  \\tag{2} \\\\\n",
        "\\end{align}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-DnpSc8K9NqI"
      },
      "source": [
        "Let's fix a value for $\\mathbf{z_1}$ as a working example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ecPlwAxm9NqJ",
        "outputId": "e8e3e49d-e514-4ddc-f090-d9aea359f9c2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[ 2.71320643],\n",
              "       [-4.79248051],\n",
              "       [ 1.33648235],\n",
              "       [ 2.48803883],\n",
              "       [-0.01492988]])"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Define a random seed so all random outcomes can be reproduced\n",
        "np.random.seed(10)\n",
        "\n",
        "# Define a 5X1 column vector using numpy\n",
        "z_1 = 10*np.random.rand(5, 1)-5\n",
        "\n",
        "# Print the vector\n",
        "z_1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wt45KJGm9NqK"
      },
      "source": [
        "Notice that using numpy's `random.rand` function returns a numpy array filled with values taken from a uniform distribution over [0, 1). Numpy allows vectorization so each value is multiplied by 10 and then substracted 5."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JknOP--F9NqK"
      },
      "source": [
        "To get the ReLU of this vector, you want all the negative values to become zeros.\n",
        "\n",
        "First create a copy of this vector."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vHlNVG1n9NqK"
      },
      "outputs": [],
      "source": [
        "# Create copy of vector and save it in the 'h' variable\n",
        "h = z_1.copy()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gmH4G7Gx9NqL"
      },
      "source": [
        "Now determine which of its values are negative."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "icRPJioX9NqL",
        "outputId": "b34e3c69-61e2-43db-9112-b308b0d57ffd"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[False],\n",
              "       [ True],\n",
              "       [False],\n",
              "       [False],\n",
              "       [ True]])"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Determine which values met the criteria (this is possible because of vectorization)\n",
        "h < 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lD8Zni759NqM"
      },
      "source": [
        "You can now simply set all of the values which are negative to 0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iJYKQZXo9NqM"
      },
      "outputs": [],
      "source": [
        "# Slice the array or vector. This is the same as applying ReLU to it\n",
        "h[h < 0] = 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WTGquqA_9NqM"
      },
      "source": [
        "And that's it: you have the ReLU of $\\mathbf{z_1}$!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oek4L3PJ9NqM",
        "outputId": "5599273b-c43f-451b-ba87-adffe828e5d2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[2.71320643],\n",
              "       [0.        ],\n",
              "       [1.33648235],\n",
              "       [2.48803883],\n",
              "       [0.        ]])"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Print the vector after ReLU\n",
        "h"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ek6vNvEc9NqN"
      },
      "source": [
        "**Now implement ReLU as a function.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h-csrPKA9NqN"
      },
      "outputs": [],
      "source": [
        "# Define the 'relu' function that will include the steps previously seen\n",
        "def relu(z):\n",
        "    result = z.copy()\n",
        "    result[result < 0] = 0\n",
        "    return result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fUAUN8fy9NqO"
      },
      "source": [
        "**And check that it's working.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SQphyI5D9NqO",
        "outputId": "f0fc6392-538a-4b83-dfce-8f0853e9318e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[0.        ],\n",
              "       [4.50714306],\n",
              "       [2.31993942],\n",
              "       [0.98658484],\n",
              "       [0.        ]])"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Define a new vector and save it in the 'z' variable\n",
        "z = np.array([[-1.25459881], [ 4.50714306], [ 2.31993942], [ 0.98658484], [-3.4398136 ]])\n",
        "\n",
        "# Apply ReLU to it\n",
        "relu(z)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LvYje2f-9NqP"
      },
      "source": [
        "Expected output:\n",
        "\n",
        "    array([[0.        ],\n",
        "           [4.50714306],\n",
        "           [2.31993942],\n",
        "           [0.98658484],\n",
        "           [0.        ]])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZL5VAHDN9NqP"
      },
      "source": [
        "### Softmax"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C8qkj_Lx9NqP"
      },
      "source": [
        "The second activation function that you need is softmax. This function is used to calculate the values of the output layer of the neural network, using the following formulas:\n",
        "\n",
        "\\begin{align}\n",
        " \\mathbf{z_2} &= \\mathbf{W_2}\\mathbf{h} + \\mathbf{b_2}   \\tag{3} \\\\\n",
        " \\mathbf{\\hat y} &= \\mathrm{softmax}(\\mathbf{z_2})   \\tag{4} \\\\\n",
        "\\end{align}\n",
        "\n",
        "To calculate softmax of a vector $\\mathbf{z}$, the $i$-th component of the resulting vector is given by:\n",
        "\n",
        "$$ \\textrm{softmax}(\\textbf{z})_i = \\frac{e^{z_i} }{\\sum\\limits_{j=1}^{V} e^{z_j} }  \\tag{5} $$\n",
        "\n",
        "Let's work through an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gHHvKoiQ9NqQ",
        "outputId": "9af55a8d-0297-4a6e-edab-b1d2af11d0f3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([ 9. ,  8. , 11. , 10. ,  8.5])"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Define a new vector and save it in the 'z' variable\n",
        "z = np.array([9, 8, 11, 10, 8.5])\n",
        "\n",
        "# Print the vector\n",
        "z"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sdtOPNFL9NqQ"
      },
      "source": [
        "You'll need to calculate the exponentials of each element, both for the numerator and for the denominator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5N83OMTJ9NqR",
        "outputId": "413d8e92-5191-4113-9679-15506033ebaf"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([ 8103.08392758,  2980.95798704, 59874.1417152 , 22026.46579481,\n",
              "        4914.7688403 ])"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Save exponentials of the values in a new vector\n",
        "e_z = np.exp(z)\n",
        "\n",
        "# Print the vector with the exponential values\n",
        "e_z"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BMuiW9sD9NqR"
      },
      "source": [
        "The denominator is equal to the sum of these exponentials."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p5NBrLBn9NqS",
        "outputId": "88bc616d-9e3c-415c-e714-a90adf07543e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "97899.41826492078"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Save the sum of the exponentials\n",
        "sum_e_z = np.sum(e_z)\n",
        "\n",
        "# Print sum of exponentials\n",
        "sum_e_z"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4fykEfQX9NqT"
      },
      "source": [
        "And the value of the first element of $\\textrm{softmax}(\\textbf{z})$ is given by:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iNbWCt4E9NqU",
        "outputId": "00f27ae9-218e-4f57-a6db-5a29f531abf1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.08276947985173956"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Print softmax value of the first element in the original vector\n",
        "e_z[0]/sum_e_z"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nibFwQNM9NqU"
      },
      "source": [
        "This is for one element. You can use numpy's vectorized operations to calculate the values of all the elements of the $\\textrm{softmax}(\\textbf{z})$ vector in one go.\n",
        "\n",
        "**Implement the softmax function.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xrrVKKJj9NqV"
      },
      "outputs": [],
      "source": [
        "# Define the 'softmax' function that will include the steps previously seen\n",
        "def softmax(z):\n",
        "    e_z = np.exp(z)\n",
        "    sum_e_z = np.sum(e_z)\n",
        "    return e_z / sum_e_z"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "csl_58OG9NqV"
      },
      "source": [
        "**Now check that it works.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WvZ88tW69NqV",
        "outputId": "505f745b-12e0-424b-894c-6c2df0d8a9c2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Print softmax values for original vector\n",
        "softmax([9, 8, 11, 10, 8.5])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cNzG_ai9NqW"
      },
      "source": [
        "Expected output:\n",
        "\n",
        "    array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TWQ3HNP39NqW"
      },
      "source": [
        "Notice that the sum of all these values is equal to 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zFpCMfjY9Nqm",
        "outputId": "b9642ae3-96d9-4383-ed59-98f55f14c5d3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Assert that the sum of the softmax values is equal to 1\n",
        "np.sum(softmax([9, 8, 11, 10, 8.5])) == 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ZQNmidZ9Nqn"
      },
      "source": [
        "## Dimensions: 1-D arrays vs 2-D column vectors\n",
        "\n",
        "Before moving on to implement forward propagation, backpropagation, and gradient descent in the next lecture notebook, let's have a look at the dimensions of the vectors you've been handling until now.\n",
        "\n",
        "Create a vector of length $V$ filled with zeros."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "QVVDmReZ9Nqn",
        "outputId": "68724476-a1b3-494e-f949-03e854a5a664"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0., 0., 0., 0., 0.])"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Define V. Remember this was the size of the vocabulary in the previous lecture notebook\n",
        "V = 5\n",
        "\n",
        "# Define vector of length V filled with zeros\n",
        "x_array = np.zeros(V)\n",
        "\n",
        "# Print vector\n",
        "x_array"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-3Wcf8Ty9Nqn"
      },
      "source": [
        "This is a 1-dimensional array, as revealed by the `.shape` property of the array."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SPOqy5ww9Nqn",
        "outputId": "5626a530-4da7-4d9d-9ce3-15c8e28de07a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(5,)"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Print vector's shape\n",
        "x_array.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_RCsfiXu9Nqo"
      },
      "source": [
        "To perform matrix multiplication in the next steps, you actually need your column vectors to be represented as a matrix with one column. In numpy, this matrix is represented as a 2-dimensional array.\n",
        "\n",
        "The easiest way to convert a 1D vector to a 2D column matrix is to set its `.shape` property to the number of rows and one column, as shown in the next cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uCIjZ5pF9Nqo",
        "outputId": "0a6d7669-98d2-43ac-a256-08d3bf211a1b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[0.],\n",
              "       [0.],\n",
              "       [0.],\n",
              "       [0.],\n",
              "       [0.]])"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Copy vector\n",
        "x_column_vector = x_array.copy()\n",
        "\n",
        "# Reshape copy of vector\n",
        "x_column_vector.shape = (V, 1)  # alternatively ... = (x_array.shape[0], 1)\n",
        "\n",
        "# Print vector\n",
        "x_column_vector"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w-xE5ohz9Nqo"
      },
      "source": [
        "The shape of the resulting \"vector\" is:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "glyYXwRS9Nqp",
        "outputId": "4e1c8786-f5f2-4b23-c92a-cacadaab5c9a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(5, 1)"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Print vector's shape\n",
        "x_column_vector.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DfFOP2eX9Nqp"
      },
      "source": [
        "So you now have a 5x1 matrix that you can use to perform standard matrix multiplication."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KADjbVl9Nqp"
      },
      "source": [
        "**Congratulations on finishing this lecture notebook!** Hopefully you now have a better understanding of the activation functions used in the continuous bag-of-words model, as well as a clearer idea of how to leverage Numpy's power for these types of mathematical computations.\n",
        "\n",
        "In the next lecture notebook you will get a comprehensive dive into:\n",
        "\n",
        "- Forward propagation.\n",
        "\n",
        "- Cross-entropy loss.\n",
        "\n",
        "- Backpropagation.\n",
        "\n",
        "- Gradient descent.\n",
        "\n",
        "**See you next time!**"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.1"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}