Text_Classification_Keras_LSTMs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Week 3: Exploring Overfitting in NLP\n",
    "\n",
    "Welcome to this assignment! During this week you saw different ways to handle sequence-like data. You saw how some Keras' layers such as `GRU`, `Conv` and `LSTM` can be used to tackle problems in this space. Now you will put this knowledge into practice by creating a model architecture that does not overfit.\n",
    "\n",
    "For this assignment you will be using a variation of the [Sentiment140 dataset](http://help.sentiment140.com/home), which contains 1.6 million tweets alongside their respective sentiment (0 for negative and 4 for positive).\n",
    "\n",
    "You will also need to create the helper functions very similar to the ones you coded in previous assignments pre-process data and to tokenize sentences. However the objective of the assignment is to find a model architecture that will not overfit.\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mGrader metadata detected! You can proceed with the lab!\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "# IMPORTANT: This will check your notebook's metadata for grading.\n",
    "# Please do not continue the lab unless the output of this cell tells you to proceed. \n",
    "!python add_metadata.py --filename C3W3_Assignment.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_**NOTE:** To prevent errors from the autograder, you are not allowed to edit or delete non-graded cells in this notebook . Please only put your solutions in between the `### START CODE HERE` and `### END CODE HERE` code comments, and also refrain from adding any new cells. **Once you have passed this assignment** and want to experiment with any of the non-graded code, you may follow the instructions at the bottom of this notebook._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "deletable": false,
    "editable": false,
    "id": "hmA6EzkQJ5jt",
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "import csv\n",
    "import random\n",
    "import pickle\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import linregress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining some useful global variables\n",
    "\n",
    "Next you will define some global variables that will be used throughout the assignment.\n",
    "\n",
    "- `EMBEDDING_DIM`: Dimension of the dense embedding, will be used in the embedding layer of the model. Defaults to 100.\n",
    "\n",
    "\n",
    "- `MAXLEN`: Maximum length of all sequences. Defaults to 16.\n",
    "\n",
    "\n",
    "- `TRUNCATING`: Truncating strategy (truncate either before or after each sequence.). Defaults to 'post'.\n",
    "\n",
    "\n",
    "- `PADDING`: Padding strategy (pad either before or after each sequence.). Defaults to 'post'.\n",
    "\n",
    "\n",
    "- `OOV_TOKEN`: Token to replace out-of-vocabulary words during text_to_sequence calls. Defaults to \\\"\\<OOV>\\\".\n",
    "    \n",
    "    \n",
    "- `MAX_EXAMPLES`: Max number of examples to use. Defaults to 160000 (10% of the original number of examples)\n",
    "    \n",
    "    \n",
    "- `TRAINING_SPLIT`: Proportion of data used for training. Defaults to 0.9\n",
    "    \n",
    "    \n",
    "**For now leave them unchanged but after submitting your assignment for grading you are encouraged to come back here and play with these parameters to see the impact they have in the classification process.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "EMBEDDING_DIM = 100\n",
    "MAXLEN = 16\n",
    "TRUNCATING = 'post'\n",
    "PADDING = 'post'\n",
    "OOV_TOKEN = \"<OOV>\"\n",
    "MAX_EXAMPLES = 160000\n",
    "TRAINING_SPLIT = 0.9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the dataset\n",
    "\n",
    "The dataset is provided in a csv file. \n",
    "\n",
    "Each row of this file contains the following values separated by commas:\n",
    "\n",
    "- target: the polarity of the tweet (0 = negative, 4 = positive)\n",
    "\n",
    "- ids: The id of the tweet\n",
    "\n",
    "- date: the date of the tweet\n",
    "\n",
    "- flag: The query. If there is no query, then this value is NO_QUERY.\n",
    "\n",
    "- user: the user that tweeted\n",
    "\n",
    "- text: the text of the tweet\n",
    "\n",
    "\n",
    "Take a look at the first two examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First data point looks like this:\n",
      "\n",
      "\"0\",\"1467810369\",\"Mon Apr 06 22:19:45 PDT 2009\",\"NO_QUERY\",\"_TheSpecialOne_\",\"@switchfoot http://twitpic.com/2y1zl - Awww, that's a bummer.  You shoulda got David Carr of Third Day to do it. ;D\"\n",
      "\n",
      "Second data point looks like this:\n",
      "\n",
      "\"0\",\"1467810672\",\"Mon Apr 06 22:19:49 PDT 2009\",\"NO_QUERY\",\"scotthamilton\",\"is upset that he can't update his Facebook by texting it... and might cry as a result  School today also. Blah!\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "SENTIMENT_CSV = \"./data/training_cleaned.csv\"\n",
    "\n",
    "with open(SENTIMENT_CSV, 'r') as csvfile:\n",
    "    print(f\"First data point looks like this:\\n\\n{csvfile.readline()}\")\n",
    "    print(f\"Second data point looks like this:\\n\\n{csvfile.readline()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice that this file does not have a header so you won't need to skip the first row when parsing the file.**\n",
    "\n",
    "For the task at hand you will only need the information of the target and the text, which are the first and last element of each row. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing the raw data\n",
    "\n",
    "Now you need to read the data from the csv file. To do so, complete the `parse_data_from_file` function.\n",
    "\n",
    "A couple of things to note:\n",
    "\n",
    "- You should NOT omit the first line as the file does not contain headers.\n",
    "- There is no need to save the data points as numpy arrays, regular lists is fine.\n",
    "- To read from csv files use `csv.reader` by passing the appropriate arguments.\n",
    "- `csv.reader` returns an iterable that returns each row in every iteration. So the label can be accessed via `row[0]` and the text via `row[5]`.\n",
    "- The labels are originally encoded as strings ('0' representing negative and '4' representing positive). **You need to change this so that the labels are integers and 0 is used for representing negative, while 1 should represent positive.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "deletable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# GRADED FUNCTION: parse_data_from_file\n",
    "def parse_data_from_file(filename):\n",
    "    \"\"\"\n",
    "    Extracts sentences and labels from a CSV file\n",
    "    \n",
    "    Args:\n",
    "        filename (string): path to the CSV file\n",
    "    \n",
    "    Returns:\n",
    "        sentences, labels (list of string, list of string): tuple containing lists of sentences and labels\n",
    "    \"\"\"\n",
    "    \n",
    "    sentences = []\n",
    "    labels = []\n",
    "    \n",
    "    with open(filename, 'r') as csvfile:\n",
    "        ### START CODE HERE\n",
    "        reader = csv.reader(csvfile, delimiter=',')\n",
    "        \n",
    "        # Iterate through each row in the CSV file\n",
    "        for row in reader:\n",
    "            # Convert label to integer (0 for negative, 1 for positive)\n",
    "            label = int(row[0])  # Original labels are 0 or 4\n",
    "            label = 0 if label == 0 else 1  # Convert 0 to 0, 4 to 1\n",
    "            \n",
    "            # Extract text from the correct column (assuming text is in column index 5)\n",
    "            text = row[5]\n",
    "            \n",
    "            # Append to lists\n",
    "            labels.append(label)\n",
    "            sentences.append(text)\n",
    "        ### END CODE HERE\n",
    "        \n",
    "    return sentences, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset contains 1600000 examples\n",
      "\n",
      "Text of second example should look like this:\n",
      "is upset that he can't update his Facebook by texting it... and might cry as a result  School today also. Blah!\n",
      "\n",
      "Text of fourth example should look like this:\n",
      "my whole body feels itchy and like its on fire \n",
      "\n",
      "Labels of last 5 examples should look like this:\n",
      "[1, 1, 1, 1, 1]\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Test your function\n",
    "sentences, labels = parse_data_from_file(SENTIMENT_CSV)\n",
    "\n",
    "print(f\"dataset contains {len(sentences)} examples\\n\")\n",
    "\n",
    "print(f\"Text of second example should look like this:\\n{sentences[1]}\\n\")\n",
    "print(f\"Text of fourth example should look like this:\\n{sentences[3]}\")\n",
    "\n",
    "print(f\"\\nLabels of last 5 examples should look like this:\\n{labels[-5:]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Expected Output:***\n",
    "\n",
    "```\n",
    "dataset contains 1600000 examples\n",
    "\n",
    "Text of second example should look like this:\n",
    "is upset that he can't update his Facebook by texting it... and might cry as a result  School today also. Blah!\n",
    "\n",
    "Text of fourth example should look like this:\n",
    "my whole body feels itchy and like its on fire \n",
    "\n",
    "Labels of last 5 examples should look like this:\n",
    "[1, 1, 1, 1, 1]\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might have noticed that this dataset contains a lot of examples. In order to keep a low execution time of this assignment you will be using only 10% of the original data. The next cell does this while also randomnizing the datapoints that will be used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 160000 sentences and 160000 labels after random sampling\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Bundle the two lists into a single one\n",
    "sentences_and_labels = list(zip(sentences, labels))\n",
    "\n",
    "# Perform random sampling\n",
    "random.seed(42)\n",
    "sentences_and_labels = random.sample(sentences_and_labels, MAX_EXAMPLES)\n",
    "\n",
    "# Unpack back into separate lists\n",
    "sentences, labels = zip(*sentences_and_labels)\n",
    "\n",
    "print(f\"There are {len(sentences)} sentences and {len(labels)} labels after random sampling\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Expected Output:***\n",
    "\n",
    "```\n",
    "There are 160000 sentences and 160000 labels after random sampling\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training - Validation Split\n",
    "\n",
    "Now you will code the `train_val_split`, which given the list of sentences, the list of labels and the proportion of data for the training set, should return the training and validation sentences and labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "deletable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# GRADED FUNCTION: train_val_split\n",
    "def train_val_split(sentences, labels, training_split):\n",
    "    \"\"\"\n",
    "    Splits the dataset into training and validation sets\n",
    "    \n",
    "    Args:\n",
    "        sentences (list of string): lower-cased sentences without stopwords\n",
    "        labels (list of string): list of labels\n",
    "        training split (float): proportion of the dataset to convert to include in the train set\n",
    "    \n",
    "    Returns:\n",
    "        train_sentences, validation_sentences, train_labels, validation_labels - lists containing the data splits\n",
    "    \"\"\"   \n",
    "    ### START CODE HERE\n",
    "    \n",
    "    # Compute the number of sentences that will be used for training (should be an integer)\n",
    "    train_size = int(len(sentences) * training_split)\n",
    "\n",
    "    # Split the sentences and labels into train/validation splits\n",
    "    train_sentences = sentences[:train_size]\n",
    "    train_labels = labels[:train_size]\n",
    "\n",
    "    validation_sentences = sentences[train_size:]\n",
    "    validation_labels = labels[train_size:]\n",
    "    \n",
    "    ### END CODE HERE\n",
    "    \n",
    "    return train_sentences, validation_sentences, train_labels, validation_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 144000 sentences for training.\n",
      "\n",
      "There are 144000 labels for training.\n",
      "\n",
      "There are 16000 sentences for validation.\n",
      "\n",
      "There are 16000 labels for validation.\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Test your function\n",
    "train_sentences, val_sentences, train_labels, val_labels = train_val_split(sentences, labels, TRAINING_SPLIT)\n",
    "\n",
    "print(f\"There are {len(train_sentences)} sentences for training.\\n\")\n",
    "print(f\"There are {len(train_labels)} labels for training.\\n\")\n",
    "print(f\"There are {len(val_sentences)} sentences for validation.\\n\")\n",
    "print(f\"There are {len(val_labels)} labels for validation.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Expected Output:***\n",
    "\n",
    "```\n",
    "There are 144000 sentences for training.\n",
    "\n",
    "There are 144000 labels for training.\n",
    "\n",
    "There are 16000 sentences for validation.\n",
    "\n",
    "There are 16000 labels for validation.\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization - Sequences, truncating and padding\n",
    "\n",
    "Now that you have sets for training and validation it is time for you to begin the tokenization process.\n",
    "\n",
    "Begin by completing the `fit_tokenizer` function below. This function should return a [Tokenizer](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer) that has been fitted to the training sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "deletable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# GRADED FUNCTION: fit_tokenizer\n",
    "def fit_tokenizer(train_sentences, oov_token):\n",
    "    \"\"\"\n",
    "    Instantiates the Tokenizer class on the training sentences\n",
    "    \n",
    "    Args:\n",
    "        train_sentences (list of string): lower-cased sentences without stopwords to be used for training\n",
    "        oov_token (string) - symbol for the out-of-vocabulary token\n",
    "    \n",
    "    Returns:\n",
    "        tokenizer (object): an instance of the Tokenizer class containing the word-index dictionary\n",
    "    \"\"\"    \n",
    "    ### START CODE HERE\n",
    "    \n",
    "    # Instantiate the Tokenizer class, passing in the correct values for oov_token\n",
    "    tokenizer = Tokenizer(oov_token=oov_token)\n",
    "    \n",
    "    # Fit the tokenizer to the training sentences\n",
    "    tokenizer.fit_on_texts(train_sentences)\n",
    "    \n",
    "    ### END CODE HERE\n",
    "    \n",
    "    return tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary contains 128293 words\n",
      "\n",
      "<OOV> token included in vocabulary\n",
      "\n",
      "index of word 'i' should be 2\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Test your function\n",
    "tokenizer = fit_tokenizer(train_sentences, OOV_TOKEN)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "VOCAB_SIZE = len(word_index)\n",
    "\n",
    "print(f\"Vocabulary contains {VOCAB_SIZE} words\\n\")\n",
    "print(\"<OOV> token included in vocabulary\" if \"<OOV>\" in word_index else \"<OOV> token NOT included in vocabulary\")\n",
    "print(f\"\\nindex of word 'i' should be {word_index['i']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Expected Output:***\n",
    "\n",
    "```\n",
    "Vocabulary contains 128293 words\n",
    "\n",
    "<OOV> token included in vocabulary\n",
    "\n",
    "index of word 'i' should be 2\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "deletable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# GRADED FUNCTION: seq_pad_and_trunc\n",
    "def seq_pad_and_trunc(sentences, tokenizer, padding, truncating, maxlen):\n",
    "    \"\"\"\n",
    "    Generates an array of token sequences and pads them to the same length\n",
    "    \n",
    "    Args:\n",
    "        sentences (list of string): list of sentences to tokenize and pad\n",
    "        tokenizer (object): Tokenizer instance containing the word-index dictionary\n",
    "        padding (string): type of padding to use\n",
    "        truncating (string): type of truncating to use\n",
    "        maxlen (int): maximum length of the token sequence\n",
    "    \n",
    "    Returns:\n",
    "        pad_trunc_sequences (array of int): tokenized sentences padded to the same length\n",
    "    \"\"\"     \n",
    "    ### START CODE HERE\n",
    "       \n",
    "    # Convert sentences to sequences\n",
    "    sequences = tokenizer.texts_to_sequences(sentences)\n",
    "    \n",
    "    # Pad the sequences using the correct padding, truncating and maxlen\n",
    "    pad_trunc_sequences = pad_sequences(sequences, padding=padding, truncating=truncating, maxlen=maxlen)\n",
    "    \n",
    "    ### END CODE HERE\n",
    "    \n",
    "    return pad_trunc_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Padded and truncated training sequences have shape: (144000, 16)\n",
      "\n",
      "Padded and truncated validation sequences have shape: (16000, 16)\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Test your function\n",
    "train_pad_trunc_seq = seq_pad_and_trunc(train_sentences, tokenizer, PADDING, TRUNCATING, MAXLEN)\n",
    "val_pad_trunc_seq = seq_pad_and_trunc(val_sentences, tokenizer, PADDING, TRUNCATING, MAXLEN)\n",
    "\n",
    "print(f\"Padded and truncated training sequences have shape: {train_pad_trunc_seq.shape}\\n\")\n",
    "print(f\"Padded and truncated validation sequences have shape: {val_pad_trunc_seq.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Expected Output:***\n",
    "\n",
    "```\n",
    "Padded and truncated training sequences have shape: (144000, 16)\n",
    "\n",
    "Padded and truncated validation sequences have shape: (16000, 16)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that the `pad_sequences` function returns numpy arrays, so your training and validation sequences are already in this format.\n",
    "\n",
    "However the labels are still Python lists. Before going forward you should convert them numpy arrays as well. You can do this by running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "train_labels = np.array(train_labels)\n",
    "val_labels = np.array(val_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using pre-defined Embeddings\n",
    "\n",
    "This time you will not be learning embeddings from your data but you will be using pre-trained word vectors.\n",
    "\n",
    "In particular you will be using the 100 dimension version of [GloVe](https://nlp.stanford.edu/projects/glove/) from Stanford."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Define path to file containing the embeddings\n",
    "GLOVE_FILE = './data/glove.6B.100d.txt'\n",
    "\n",
    "# Initialize an empty embeddings index dictionary\n",
    "GLOVE_EMBEDDINGS = {}\n",
    "\n",
    "# Read file and fill GLOVE_EMBEDDINGS with its contents\n",
    "with open(GLOVE_FILE) as f:\n",
    "    for line in f:\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        coefs = np.asarray(values[1:], dtype='float32')\n",
    "        GLOVE_EMBEDDINGS[word] = coefs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have access to GloVe's pre-trained word vectors. Isn't that cool?\n",
    "\n",
    "Let's take a look at the vector for the word **dog**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vector representation of word dog looks like this:\n",
      "\n",
      "[ 0.30817    0.30938    0.52803   -0.92543   -0.73671    0.63475\n",
      "  0.44197    0.10262   -0.09142   -0.56607   -0.5327     0.2013\n",
      "  0.7704    -0.13983    0.13727    1.1128     0.89301   -0.17869\n",
      " -0.0019722  0.57289    0.59479    0.50428   -0.28991   -1.3491\n",
      "  0.42756    1.2748    -1.1613    -0.41084    0.042804   0.54866\n",
      "  0.18897    0.3759     0.58035    0.66975    0.81156    0.93864\n",
      " -0.51005   -0.070079   0.82819   -0.35346    0.21086   -0.24412\n",
      " -0.16554   -0.78358   -0.48482    0.38968   -0.86356   -0.016391\n",
      "  0.31984   -0.49246   -0.069363   0.018869  -0.098286   1.3126\n",
      " -0.12116   -1.2399    -0.091429   0.35294    0.64645    0.089642\n",
      "  0.70294    1.1244     0.38639    0.52084    0.98787    0.79952\n",
      " -0.34625    0.14095    0.80167    0.20987   -0.86007   -0.15308\n",
      "  0.074523   0.40816    0.019208   0.51587   -0.34428   -0.24525\n",
      " -0.77984    0.27425    0.22418    0.20164    0.017431  -0.014697\n",
      " -1.0235    -0.39695   -0.0056188  0.30569    0.31748    0.021404\n",
      "  0.11837   -0.11319    0.42456    0.53405   -0.16717   -0.27185\n",
      " -0.6255     0.12883    0.62529   -0.52086  ]\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "test_word = 'dog'\n",
    "\n",
    "test_vector = GLOVE_EMBEDDINGS[test_word]\n",
    "\n",
    "print(f\"Vector representation of word {test_word} looks like this:\\n\\n{test_vector}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to change the `test_word` to see the vector representation of any word you can think of.\n",
    "\n",
    "Also, notice that the dimension of each vector is 100. You can easily double check this by running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Each word vector has shape: (100,)\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "print(f\"Each word vector has shape: {test_vector.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Represent the words in your vocabulary using the embeddings\n",
    "\n",
    "Save the vector representation of each word in the vocabulary in a numpy array.\n",
    "\n",
    "A couple of things to notice:\n",
    "- If a word in your vocabulary is not present in `GLOVE_EMBEDDINGS` the representation for that word is left as a column of zeros.\n",
    "- `word_index` starts counting at 1, because of this you will need to add an extra column at the left-most side of the `EMBEDDINGS_MATRIX` array. This is the reason why you add 1 to `VOCAB_SIZE` in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "deletable": false,
    "editable": false,
    "id": "C1zdgJkusRh0",
    "outputId": "538df576-bbfc-4590-c3a3-0559dab5f176",
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Initialize an empty numpy array with the appropriate size\n",
    "EMBEDDINGS_MATRIX = np.zeros((VOCAB_SIZE+1, EMBEDDING_DIM))\n",
    "\n",
    "# Iterate all of the words in the vocabulary and if the vector representation for \n",
    "# each word exists within GloVe's representations, save it in the EMBEDDINGS_MATRIX array\n",
    "for word, i in word_index.items():\n",
    "    embedding_vector = GLOVE_EMBEDDINGS.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        EMBEDDINGS_MATRIX[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have the pre-trained embeddings ready to use!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a model that does not overfit\n",
    "\n",
    "Now you need to define a model that will handle the problem at hand while not overfitting.\n",
    "\n",
    "A couple of things to note / hints:\n",
    "\n",
    "- The first layer is provided so you can see how the Embedding layer is configured when using pre-trained embeddings\n",
    "\n",
    "\n",
    "- You can try different combinations of layers covered in previous ungraded labs such as:\n",
    "    - `Conv1D`\n",
    "    - `Dropout`\n",
    "    - `GlobalMaxPooling1D`    \n",
    "    - `MaxPooling1D`    \n",
    "    - `LSTM`    \n",
    "    - `Bidirectional(LSTM)`\n",
    "\n",
    "\n",
    "- The last two layers should be `Dense` layers.\n",
    "\n",
    "\n",
    "- There multiple ways of solving this problem. So try an architecture that you think will not overfit.\n",
    "\n",
    "\n",
    "- Try simpler architectures first to avoid long training times. Architectures that are able to solve this problem usually have around 3-4 layers (excluding the last two `Dense` ones)\n",
    "\n",
    "\n",
    "- Include at least one `Dropout` layer to mitigate overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "deletable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# GRADED FUNCTION: create_model\n",
    "def create_model(vocab_size, embedding_dim, maxlen, embeddings_matrix):\n",
    "    \"\"\"\n",
    "    Creates a binary sentiment classifier model\n",
    "    \n",
    "    Args:\n",
    "        vocab_size (int): size of the vocabulary for the Embedding layer input\n",
    "        embedding_dim (int): dimensionality of the Embedding layer output\n",
    "        maxlen (int): length of the input sequences\n",
    "        embeddings_matrix (array): predefined weights of the embeddings\n",
    "    \n",
    "    Returns:\n",
    "        model (tf.keras Model): the sentiment classifier model\n",
    "    \"\"\"\n",
    "    ### START CODE HERE\n",
    "    \n",
    "    model = tf.keras.Sequential([\n",
    "        # Embedding layer with pre-trained weights\n",
    "         tf.keras.layers.Embedding(vocab_size+1, embedding_dim, input_length=maxlen, weights=[embeddings_matrix], trainable=False),\n",
    "        \n",
    "        # Add a Convolutional layer\n",
    "        tf.keras.layers.Conv1D(filters=64, kernel_size=5, activation='relu'),\n",
    "        \n",
    "        # Add a MaxPooling layer\n",
    "        tf.keras.layers.MaxPooling1D(pool_size=4),\n",
    "        \n",
    "        # Add a Bidirectional LSTM layer\n",
    "        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32, return_sequences=True)),\n",
    "        \n",
    "        # Add a GlobalMaxPooling1D layer to aggregate features\n",
    "        tf.keras.layers.GlobalMaxPooling1D(),\n",
    "        \n",
    "        # Add a Dropout layer to mitigate overfitting\n",
    "        tf.keras.layers.Dropout(0.6),\n",
    "        \n",
    "        # Add Dense layers\n",
    "        tf.keras.layers.Dense(32, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01)),\n",
    "        tf.keras.layers.Dense(1, activation='sigmoid')\n",
    "    ])\n",
    "    \n",
    "    optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)\n",
    "    \n",
    "    model.compile(loss='binary_crossentropy',\n",
    "                  optimizer=optimizer,\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model\n",
    "\n",
    "    ### END CODE HERE\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "4500/4500 [==============================] - 52s 11ms/step - loss: 0.7287 - accuracy: 0.6560 - val_loss: 0.5921 - val_accuracy: 0.7058\n",
      "Epoch 2/20\n",
      "4500/4500 [==============================] - 49s 11ms/step - loss: 0.5762 - accuracy: 0.7111 - val_loss: 0.5557 - val_accuracy: 0.7212\n",
      "Epoch 3/20\n",
      "4500/4500 [==============================] - 49s 11ms/step - loss: 0.5469 - accuracy: 0.7286 - val_loss: 0.5386 - val_accuracy: 0.7302\n",
      "Epoch 4/20\n",
      "4500/4500 [==============================] - 49s 11ms/step - loss: 0.5283 - accuracy: 0.7415 - val_loss: 0.5303 - val_accuracy: 0.7368\n",
      "Epoch 5/20\n",
      "4500/4500 [==============================] - 49s 11ms/step - loss: 0.5151 - accuracy: 0.7510 - val_loss: 0.5244 - val_accuracy: 0.7409\n",
      "Epoch 6/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.5033 - accuracy: 0.7577 - val_loss: 0.5350 - val_accuracy: 0.7326\n",
      "Epoch 7/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4935 - accuracy: 0.7647 - val_loss: 0.5233 - val_accuracy: 0.7380\n",
      "Epoch 8/20\n",
      "4500/4500 [==============================] - 47s 10ms/step - loss: 0.4847 - accuracy: 0.7717 - val_loss: 0.5208 - val_accuracy: 0.7453\n",
      "Epoch 9/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4763 - accuracy: 0.7768 - val_loss: 0.5190 - val_accuracy: 0.7479\n",
      "Epoch 10/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4688 - accuracy: 0.7825 - val_loss: 0.5223 - val_accuracy: 0.7477\n",
      "Epoch 11/20\n",
      "4500/4500 [==============================] - 47s 11ms/step - loss: 0.4627 - accuracy: 0.7862 - val_loss: 0.5242 - val_accuracy: 0.7441\n",
      "Epoch 12/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4568 - accuracy: 0.7906 - val_loss: 0.5242 - val_accuracy: 0.7461\n",
      "Epoch 13/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4508 - accuracy: 0.7943 - val_loss: 0.5268 - val_accuracy: 0.7433\n",
      "Epoch 14/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4445 - accuracy: 0.7983 - val_loss: 0.5313 - val_accuracy: 0.7423\n",
      "Epoch 15/20\n",
      "4500/4500 [==============================] - 47s 10ms/step - loss: 0.4393 - accuracy: 0.8024 - val_loss: 0.5395 - val_accuracy: 0.7447\n",
      "Epoch 16/20\n",
      "4500/4500 [==============================] - 47s 11ms/step - loss: 0.4349 - accuracy: 0.8052 - val_loss: 0.5394 - val_accuracy: 0.7427\n",
      "Epoch 17/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4298 - accuracy: 0.8083 - val_loss: 0.5408 - val_accuracy: 0.7443\n",
      "Epoch 18/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4260 - accuracy: 0.8112 - val_loss: 0.5442 - val_accuracy: 0.7351\n",
      "Epoch 19/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4208 - accuracy: 0.8152 - val_loss: 0.5476 - val_accuracy: 0.7406\n",
      "Epoch 20/20\n",
      "4500/4500 [==============================] - 48s 11ms/step - loss: 0.4164 - accuracy: 0.8172 - val_loss: 0.5429 - val_accuracy: 0.7398\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Create your untrained model\n",
    "model = create_model(VOCAB_SIZE, EMBEDDING_DIM, MAXLEN, EMBEDDINGS_MATRIX)\n",
    "\n",
    "# Train the model and save the training history\n",
    "history = model.fit(train_pad_trunc_seq, train_labels, epochs=20, validation_data=(val_pad_trunc_seq, val_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**To pass this assignment your `val_loss` (validation loss) should either be flat or decreasing.** \n",
    "\n",
    "Although a flat `val_loss` and a lowering `train_loss` (or just `loss`) also indicate some overfitting what you really want to avoid is having a lowering `train_loss` and an increasing `val_loss`.\n",
    "\n",
    "With this in mind, the following three curves will be acceptable solutions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table><tr><td><img src='images/valid-1.png'></td><td><img src='images/valid-2.jpg'></td><td><img src='images/valid-3.jpg'></td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the following would not be able to pass the grading:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table><tr><td><img src='images/invalid-1.jpg'></td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to check your loss curves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU5dn/8c/FrqxhEUFAoIKaCLK7gIJaFYWCilXRKmirxVZbq7X42FYsatU+PtUflmrVutQNFS1FxSLuuJZFQBZBQFRQkX0pe3L9/rjPkEmYhMkymSTzfb9e5zUz59znzJWTZK65l3Mfc3dEREQKq5HuAEREpHJSghARkYSUIEREJCElCBERSUgJQkREElKCEBGRhJQgpEKY2StmNqK8y6aTma0ws++n4LhuZodFz+83s98nU7YU73ORmb1a2jiLOe4AM1tZ3seVilcr3QFI5WVmW+NeHgjsBHKj1z919yeTPZa7n5GKstWdu48qj+OYWXvgc6C2u++Jjv0kkPTvUDKPEoQUyd0bxJ6b2QrgJ+7+WuFyZlYr9qEjItWHmpikxGJNCGY22sy+BR4xsywze8nM1pjZhuh5m7h93jKzn0TPR5rZu2Z2V1T2czM7o5RlO5jZO2a2xcxeM7PxZvZEEXEnE+MtZvZedLxXzax53PaLzewLM1tnZr8t5vwcY2bfmlnNuHVnm9m86HkfM/vAzDaa2Tdm9hczq1PEsR41s1vjXl8f7fO1mV1WqOwgM/vYzDab2VdmdnPc5neix41mttXMjoud27j9jzezGWa2KXo8PtlzUxwzOzLaf6OZLTCzIXHbzjSzhdExV5nZr6P1zaPfz0YzW29m081Mn1cVTCdcSutgoClwKHAF4W/pkeh1O2A78Jdi9j8GWAw0B/4E/N3MrBRlnwL+AzQDbgYuLuY9k4nxQuBS4CCgDhD7wMoG7ouO3zp6vzYk4O4fAf8FTi503Kei57nAr6Kf5zjgFOBnxcRNFMPAKJ5TgU5A4f6P/wKXAE2AQcCVZnZWtO3E6LGJuzdw9w8KHbsp8DIwLvrZ/gy8bGbNCv0M+5yb/cRcG3gReDXa72rgSTM7PCryd0JzZUPgKOCNaP11wEqgBdASuBHQvEAVTAlCSisPGOPuO919u7uvc/fn3X2bu28BbgP6F7P/F+7+oLvnAo8BrQgfBEmXNbN2QG/gJnff5e7vApOLesMkY3zE3Ze4+3bgWaBbtP5c4CV3f8fddwK/j85BUZ4GhgOYWUPgzGgd7j7L3T909z3uvgL4W4I4Ejkvim++u/+XkBDjf7633P0Td89z93nR+yVzXAgJ5TN3fzyK62ngU+AHcWWKOjfFORZoANwR/Y7eAF4iOjfAbiDbzBq5+wZ3nx23vhVwqLvvdvfpronjKpwShJTWGnffEXthZgea2d+iJpjNhCaNJvHNLIV8G3vi7tuipw1KWLY1sD5uHcBXRQWcZIzfxj3fFhdT6/hjRx/Q64p6L0Jt4RwzqwucA8x29y+iODpHzSffRnH8kVCb2J8CMQBfFPr5jjGzN6MmtE3AqCSPGzv2F4XWfQEcEve6qHOz35jdPT6Zxh93GCF5fmFmb5vZcdH6/wWWAq+a2XIzuyG5H0PKkxKElFbhb3PXAYcDx7h7I/KbNIpqNioP3wBNzezAuHVtiylflhi/iT929J7Niirs7gsJH4RnULB5CUJT1adApyiOG0sTA6GZLN5ThBpUW3dvDNwfd9z9ffv+mtD0Fq8dsCqJuPZ33LaF+g/2HtfdZ7j7UELz0yRCzQR33+Lu17l7R2AIcK2ZnVLGWKSElCCkvDQktOlvjNqzx6T6DaNv5DOBm82sTvTt8wfF7FKWGCcCg82sX9ShPJb9//88BfySkIieKxTHZmCrmR0BXJlkDM8CI80sO0pQheNvSKhR7TCzPoTEFLOG0CTWsYhjTwE6m9mFZlbLzM4HsgnNQWXxEaG28Rszq21mAwi/ownR7+wiM2vs7rsJ5yQPwMwGm9lhUV/TJkK/TXFNepICShBSXu4BDgDWAh8C/66g972I0NG7DrgVeIZwvUYipY7R3RcAPyd86H8DbCB0ohYn1gfwhruvjVv/a8KH9xbgwSjmZGJ4JfoZ3iA0v7xRqMjPgLFmtgW4iejbeLTvNkKfy3vRyKBjCx17HTCYUMtaB/wGGFwo7hJz912EhHAG4bz/FbjE3T+NilwMrIia2kYRfp8QOuFfA7YCHwB/dfc3yxKLlJyp30eqEzN7BvjU3VNegxGp7lSDkCrNzHqb2ffMrEY0DHQooS1bRMpIV1JLVXcw8AKhw3glcKW7f5zekESqBzUxiYhIQmpiEhGRhKpNE1Pz5s29ffv26Q5DRKRKmTVr1lp3b5FoW7VJEO3bt2fmzJnpDkNEpEoxs8JX0O+lJiYREUlICUJERBJSghARkYSqTR+EiFSc3bt3s3LlSnbs2LH/wlIp1KtXjzZt2lC7du2k91GCEJESW7lyJQ0bNqR9+/YUfZ8nqSzcnXXr1rFy5Uo6dOiQ9H5qYhKREtuxYwfNmjVTcqgizIxmzZqVuManBCEipaLkULWU5velBLFhA4wdC7qGQkSkACWIGjVgzBh47bV0RyIiJdCgQTJ3PJWyUIJo3BjatIEFC9IdiYhIpaIEAZCdDQsXpjsKESmjOXPmcOyxx9K1a1fOPvtsNmzYAMC4cePIzs6ma9euXHDBBQC8/fbbdOvWjW7dutG9e3e2bNmSztArJQ1zBcjJgfvvh7y80OQkIsm75hqYM6d8j9mtG9xzT4l3u+SSS7j33nvp378/N910E3/4wx+45557uOOOO/j888+pW7cuGzduBOCuu+5i/Pjx9O3bl61bt1KvXr3y/RmqAX0aQqhBbN8OK1akOxIRKaVNmzaxceNG+vfvD8CIESN45513AOjatSsXXXQRTzzxBLVqhe/Fffv25dprr2XcuHFs3Lhx73rJpzMCoQYBoR+iY8f0xiJS1ZTim35Fe/nll3nnnXd48cUXue222/jkk0+44YYbGDRoEFOmTKFv375MnTqVI444It2hViqqQQAceWR4VD+ESJXVuHFjsrKymD59OgCPP/44/fv3Jy8vj6+++oqTTjqJO++8k02bNrF161aWLVtGly5dGD16NL179+bTTz9N809Q+agGAdCkCRxyiEYyiVQh27Zto02bNntfX3vttTz22GOMGjWKbdu20bFjRx555BFyc3P50Y9+xKZNm3B3fvGLX9CkSRN+//vf8+abb1KjRg1ycnI444wz0vjTVE5KEDEaySRSpeTl5SVc/+GHH+6z7t13391n3b333lvuMVU3amKKycmBRYvCSCYREVGC2Cs7G7Ztgy+KvPueiEhGUYKIiY1kUjOTiAigBJEvOzs8qqNaRARQgsjXpAm0bq0ahIhIRAkiXna2ahAiIpGUJggzG2hmi81sqZndkGD73WY2J1qWmNnGuG0jzOyzaBmRyjj30kgmkSrhpJNOYurUqQXW3XPPPVx55ZVF7jNgwABmRvd9OfPMM/fOyRTv5ptv5q677ir2vSdNmsTCuJaGm266idfK4XYBb731FoMHDy7zccpTyhKEmdUExgNnANnAcDPLji/j7r9y927u3g24F3gh2rcpMAY4BugDjDGzrFTFuld2Nvz3v/Dllyl/KxEpveHDhzNhwoQC6yZMmMDw4cOT2n/KlCk0adKkVO9dOEGMHTuW73//+6U6VmWXyhpEH2Cpuy93913ABGBoMeWHA09Hz08Hprn7enffAEwDBqYw1kAjmUSqhHPPPZeXX36ZXbt2AbBixQq+/vprTjjhBK688kp69epFTk4OY8aMSbh/+/btWbt2LQC33XYbnTt3pl+/fixevHhvmQcffJDevXtz9NFHM2zYMLZt28b777/P5MmTuf766+nWrRvLli1j5MiRTJw4EYDXX3+d7t2706VLFy677DJ27ty59/3GjBlDjx496NKlS4mm9Xj66afp0qULRx11FKNHjwYgNzeXkSNHctRRR9GlSxfuvvtuIPG05mWRyiupDwG+inu9klAj2IeZHQp0AN4oZt9DEux3BXAFQLt27coecfxIpjPPLPvxRDJAOmb7btq0KX369OGVV15h6NChTJgwgfPOOw8z47bbbqNp06bk5uZyyimnMG/ePLp27ZrwOLNmzWLChAnMmTOHPXv20KNHD3r27AnAOeecw+WXXw7A7373O/7+979z9dVXM2TIEAYPHsy5555b4Fg7duxg5MiRvP7663Tu3JlLLrmE++67j2uuuQaA5s2bM3v2bP76179y11138dBDD+33PHz99deMHj2aWbNmkZWVxWmnncakSZNo27Ytq1atYv78+QB7m8sSTWteFpWlk/oCYKK755ZkJ3d/wN17uXuvFi1alD2KrCxo1Uo1CJEqIL6ZKb556dlnn6VHjx50796dBQsWFGgOKmz69OmcffbZHHjggTRq1IghQ4bs3TZ//nxOOOEEunTpwpNPPsmC/QxgWbx4MR06dKBz585AwenGISQcgJ49e7IiyVsLzJgxgwEDBtCiRQtq1arFRRddxDvvvEPHjh1Zvnw5V199Nf/+979p1KgRkHha87JIZQ1iFdA27nWbaF0iFwA/L7TvgEL7vlWOsRVNI5lESiRds30PHTqUX/3qV8yePZtt27bRs2dPPv/8c+666y5mzJhBVlYWI0eOZMeOHaU6/siRI5k0aRJHH300jz76KG+99VaZ4q1bty4ANWvWZM+ePWU6VlZWFnPnzmXq1Kncf//9PPvsszz88MMJpzUvS6JIZQ1iBtDJzDqYWR1CEphcuJCZHQFkAR/ErZ4KnGZmWVHn9GnRutTLyQk1CPcKeTsRKZ0GDRpw0kkncdlll+2tPWzevJn69evTuHFjVq9ezSuvvFLsMU488UQmTZrE9u3b2bJlCy+++OLebVu2bKFVq1bs3r2bJ598cu/6hg0bJrw96eGHH86KFStYunQpkD/deFn06dOHt99+m7Vr15Kbm8vTTz9N//79Wbt2LXl5eQwbNoxbb72V2bNnFzmteVmkrAbh7nvM7CrCB3tN4GF3X2BmY4GZ7h5LFhcAE9zzP5Hdfb2Z3UJIMgBj3X19qmItIH4k06GHVshbikjpDB8+nLPPPntvU9PRRx9N9+7dOeKII2jbti19+/Ytdv8ePXpw/vnnc/TRR3PQQQfRu3fvvdtuueUWjjnmGFq0aMExxxyzNylccMEFXH755YwbN25v5zRAvXr1eOSRR/jhD3/Inj176N27N6NGjSrRz/P6668XmML8ueee44477uCkk07C3Rk0aBBDhw5l7ty5XHrppXtntL399tuLnNa8LMyryTflXr16eWyMc5m8+y6ccAJMmQKaH14koUWLFnFk7EZbUmUk+r2Z2Sx375WofGXppK48YidP/RAikuGUIApr1gxattRIJhHJeEoQieTkqAYhsh/VpXk6U5Tm96UEkUjs9qP6BxBJqF69eqxbt05Joopwd9atW0e9evVKtJ/uSZ1ITg5s3QpffQXlcYW2SDXTpk0bVq5cyZo1a9IdiiSpXr16BUZIJUMJIpHYlBsLFypBiCRQu3ZtOnTokO4wJMXUxJRIbNI+9UOISAZTgkikWTM46CCNZBKRjKYEURSNZBKRDKcEURSNZBKRDKcEUZScHNiyBVauTHckIiJpoQRRlPiRTCIiGUgJoigaySQiGU4JoijNm0OLFqpBiEjGUoIoTuzmQSIiGUgJojix249qJJOIZCAliOLk5MDmzfD11+mORESkwilBFCc2kkkd1SKSgZQgihMbyaR+CBHJQEoQxWnRIoxmUg1CRDKQEsT+aCSTiGQoJYj90UgmEclQShD7k5MDmzbBN9+kOxIRkQqV0gRhZgPNbLGZLTWzG4ooc56ZLTSzBWb2VNz6XDObEy2TUxlnsTSSSUQyVMpuOWpmNYHxwKnASmCGmU1294VxZToB/wP0dfcNZnZQ3CG2u3u3VMWXtPiRTKeemt5YREQqUCprEH2Ape6+3N13AROAoYXKXA6Md/cNAO7+XQrjKZ0WLcId5lSDEJEMk8oEcQjwVdzrldG6eJ2Bzmb2npl9aGYD47bVM7OZ0fqzEr2BmV0RlZm5Zs2a8o0+/000kklEMlK6O6lrAZ2AAcBw4EEzaxJtO9TdewEXAveY2fcK7+zuD7h7L3fv1aJFi9RFqZFMIpKBUpkgVgFt4163idbFWwlMdvfd7v45sISQMHD3VdHjcuAtoHsKYy1edjZs3Ajffpu2EEREKloqE8QMoJOZdTCzOsAFQOHRSJMItQfMrDmhyWm5mWWZWd249X2B9LXx6OZBIpKBUpYg3H0PcBUwFVgEPOvuC8xsrJkNiYpNBdaZ2ULgTeB6d18HHAnMNLO50fo74kc/VTjdflREMlDKhrkCuPsUYEqhdTfFPXfg2miJL/M+0CWVsZVIy5bQtKlqECKSUdLdSV01mIVahGoQIpJBlCCSlZOjkUwiklGUIJKVnQ0bNsDq1emORESkQihBJEsjmUQkwyhBJEsjmUQkwyhBJOvggyErSzUIEckYShDJ0kgmEckwShAloZFMIpJBlCBKIjsb1q+H7yrfrOQiIuVNCaIk4m8eJCJSzSlBlIRuPyoiGUQJoiRatYImTVSDEJGMoARRErGRTKpBiEgGUIIoKd1+VEQyhBJESWVnw9q1GskkItWeEkRJaSSTiGQIJYiS0kgmEckQShAl1bo1NG6sGoSIVHtKECWlkUwikiGUIEpDI5lEJAMoQZRGdjasWRMWEZFqSgmiNDSSSUQygBJEaWgkk4hkgJQmCDMbaGaLzWypmd1QRJnzzGyhmS0ws6fi1o8ws8+iZUQq4yyxQw6BRo1UgxCRaq1Wqg5sZjWB8cCpwEpghplNdveFcWU6Af8D9HX3DWZ2ULS+KTAG6AU4MCvad0Oq4i0RjWQSkQyQyhpEH2Cpuy93913ABGBooTKXA+NjH/zuHpu/4nRgmruvj7ZNAwamMNaS0+1HRaSaS2WCOAT4Ku71ymhdvM5AZzN7z8w+NLOBJdgXM7vCzGaa2cw1FT2iKCcnzMe0dm3Fvq+ISAVJdyd1LaATMAAYDjxoZk2S3dndH3D3Xu7eq0WLFikKsQixjmrVIkSkmkplglgFtI173SZaF28lMNndd7v758ASQsJIZt/0ig11VT+EiFRTqUwQM4BOZtbBzOoAFwCTC5WZRKg9YGbNCU1Oy4GpwGlmlmVmWcBp0bqU+OILyMsr4U5t2kDDhqpBiEi1lbIE4e57gKsIH+yLgGfdfYGZjTWzIVGxqcA6M1sIvAlc7+7r3H09cAshycwAxkbryt3ixaG1aNy4Eu6okUwiUs2Zu6c7hnLRq1cvnzlzZon3c4ehQ+HVV2HWrPyWo6RcdhlMmQLfflvi9xURqQzMbJa790q0Ld2d1GlnBg8+GK57+9GPYNeuEuyckwOrV8O6dSmLT0QkXTI+QQC0bBmSxJw5cPPNJdhRI5lEpBpTgogMHQo//jHceSe8916SO2kkk4hUY0klCDOrb2Y1ouedzWyImdVObWgV7+674dBD4ZJLYMuWJHZo2xYaNFANQkSqpWRrEO8A9czsEOBV4GLg0VQFlS4NG8Ljj8OKFXDttUnsEBvJpAQhItVQsgnC3H0bcA7wV3f/IVCS8T5VRt++MHo0PPQQTC581UYiGuoqItVU0gnCzI4DLgJejtbVTE1I6XfzzdCtG/zkJ2G6pWLl5IRhrutTcpmGiEjaJJsgriFMy/3P6GK3joQL26qlOnVCU9PmzXD55eFaiSJpJJOIVFNJJQh3f9vdh7j7nVFn9Vp3/0WKY0uro46C228PzUyPPFJMwdhIpmnTKiQuEZGKkuwopqfMrJGZ1QfmAwvN7PrUhpZ+v/wlnHRSeFy+vIhC7drBOefALbck2WkhIlI1JNvElO3um4GzgFeADoSRTNVajRrw6KNQs2YY+pqbm6CQGfzjH9CzJwwfHubrEBGpBpJNELWj6x7OIpqem3Ar0GqvXTsYPz5cPPe//1tEofr14cUXoUULGDw4TA8rIlLFJZsg/gasAOoD75jZocDmVAVV2Vx4IZx3Htx0E3z8cRGFDj4YXn4Ztm+HQYNg06YKjVFEpLyVejZXM6sVTeldKZR2NtdkrV8fOq6zskIrUr16RRR8/XUYOBAGDAgzvdaudheci0g1UubZXM2ssZn9OXb/ZzP7P0JtImM0bRpGMy1cCDfeWEzBU06BBx6A116DK6/czxhZEZHKK9kmpoeBLcB50bIZKG7wZ7V0+unw85+HOZveeKOYgpdeCr/9Lfz972H2PxGRKiipJiYzm+Pu3fa3Lp1S3cQUs20bdO8euhrmzYMmTYoo6A4XXQRPPw0TJsD556c8NhGRkiqPGwZtN7N+cQfsC2wvj+CqmgMPhCeegK+/hquvLqagWWiT6tcPRowowRziIiKVQ7IJYhQw3sxWmNkK4C/AT1MWVSXXu3cY0fTEE/Dss8UUrFsXJk0KY2WHDoWlSyssRhGRskp2qo257n400BXo6u7dgZNTGlkld+ON0KcPjBoFq1YVU7BZszCaCeDMM3V7UhGpMkp0Rzl33xxdUQ2QzB0Tqq1atcKEfjt2wGWX7Wew0mGHwb/+BV9+CWefDTt3VlicIiKlVZZbjlq5RVFFde4M//d/8OqrYWrwYu9C17cvPPYYTJ+eREYREUm/siSI/X7CmdlAM1tsZkvN7IYE20ea2RozmxMtP4nblhu3vtLOgjdqFNxwQ+iP7toV3ixuEvTzz4c//hGeeip0YoiIVGLFJggz22JmmxMsW4DW+9m3JjAeOAPIBoabWXaCos+4e7doeShu/fa49UNK+HNVGLMwLfi774aLpk8+OYxu+u9/i9jhhhvgxz+GW2/dzzziIiLpVWyCcPeG7t4owdLQ3Wvt59h9gKXuvtzddwETgKHlFXhlc/zxMGdOmBr8L3+Bo48OSWMfZnDffXDqqXDFFWFqDhGRSqgsTUz7cwjwVdzrldG6woaZ2Twzm2hmbePW14um9fjQzM5K9AZmdkVs+o81a9aUY+ilc+CBcM898NZbkJcHJ54I110XLqoroHZteO45OOIIGDZMd6MTkUqp1JP17ffAZucCA939J9Hri4Fj3P2quDLNgK3uvtPMfgqc7+4nR9sOcfdV0e1N3wBOcfdlRb1fRV1JnaytW+E3vwmVhcMPD/3TxxxTqNCXX4aVdeuGC+kOSZQ/RaQ62L07/MsvWxaW5cvzH2vWhG7dQstDt26hP7PIWRrKWXFXUqcyQRwH3Ozup0ev/wfA3W8vonxNYL27N06w7VHgJXefWNT7VbYEEfPaa6HLYeXKkDBuvjnkg71mzYL+/cM9Jf7xjzDhk4hUSVu25CeA+CSwbFlIDvE3HatbFzp2hO99L4x8nzMH4htC2rfPTxix5NG+fWilLk/pShC1gCXAKcAqYAZwobsviCvTyt2/iZ6fDYx292PNLAvYFtUsmgMfAEPdvci2mMqaIAA2b4Zrrw1z9+XkhNpEz55xBRYuDCOc5s+H668PHdh16qQtXhFJLC8vTLOzfHnBGkAsCaxdW7B88+b5SSC2xF63ahXuWhnjDt9+GxLF3Ln5j4sX54+Kb9w41C7iaxs5OcXcfiAJaUkQ0RufCdwD1AQedvfbzGwsMNPdJ5vZ7cAQYA+wHrjS3T81s+MJNynKI/ST3OPufy/uvSpzgoh55ZVwvcTq1WGy19/+Ni4PbN8essj994e5PCZMCH9JIlKhtm2Dzz/ftxlo+fKwPv4615o1w0w68Ukg9rxjx/CBXh7xzJ9fMHHMmxeasWMxnH56uF9ZaaQtQVSkqpAgADZsCCOdHn88ZP/HHgvfCPZ6/vmQRXJz4W9/C/e5FpESy8sLH6KbN4emn0SPseexWsGyZeFbfLyGDQt++McngHbt0nNPsLy8EG8sYdSvH0bQl4YSRCX0r3+FUa4bNsCYMfDrX8f1TXzxRbjP6fvvh6uux40LfwEi1cimTfDSS/DPf4b5zMzylxo1Cr4ubn2NGuFbffwH/ubN+d+w96d2bWjZMnENoGPHMJ1aebf7VyZKEJXU2rVw1VXwzDPQqBH84Adw7rmhunhA7T2hR/uPfwzDoCZMCI2OIlXYunXhy9Hzz4cBHLt2hbb4o44K7eyFl7y8/a/Lywtfrho1CkvDhokfi9pWYNBIBlKCqORefz3MvjFpUrj3df36MHhwSBZnHPAW9S+/MGy4665wS7vq/HVGqp1vvw1/2xMnhmuEcnPh0EPDJUDDhsGxxxbsrJWKpQRRRezeHf6BJk4M1e41a+CAA+DMU3Zy7sp7GDTnVhoOPQUefjjcJDvN1q8Ps4U8+SQceWQYxqtKjgB89RW88EKoKbz7bvim37lzflLo0UPfcyoLJYgqaM+eMPHrxInhH+3bb6FurT0MzJ3CuU1e4wdPXkDjM45PS2yzZ8P48eFuqtu3Q69e8Omnoc339NNh9GgYMEAfAJlm2bKQEJ5/Hv7zn7CuS5f8pJCTo7+JykgJoorLzQ391RMnwvMTdrHquzrUZhendVrBuTccxpCzaqS8QrFzZ5gdZPx4+PDDMK3Ij34EP/tZqDVs2BCuGv9//w+++y6M1B09Gs46KwzDk+plzx5YsgQ++SSMpJkyJTxC+MIwbBicc06oNUjlpgRRjeTlwUdv/Jfnf/k2Exdm8wXtqVXL6dXL6Ncv3Haib19o0aJ83u/LL8OlGQ89FJq8OncOSWHEiMRTAWzfHobu3nVX+EbZuXMYoXXJJeoMrIrcw3U78+aFZDBvXlgWLcq/HqBWrXB3xVhSaN8+rSFLCSlBVFP+2D+YNepBXuAc3mlzITO+PIhdu0IdvnPnkChiSaNz5+Sr93l5oeN8/Hh48cWw7gc/CP3jp5ySXIdibm5oGrvzzjCbyMEHwzXXhPtnlMfFQ1L+tm0LF/UXTgbxVwe3bh2u2+nSJTx27RrmnNSF/1WXEkR1tmRJ+Dr/4Yfs6NSFWRffw3t1BvDe+zV47738W2A3b55fu+jbN0z1Ufgb/caN8OijoaloyZJQC/nJT+CnPw2jTkrDHd54IySKadPC0MJRo0KyaNWqTD+5lIB7aNqiEa0AABSmSURBVAb8+utwzUHhx08/hc8+y5/S4cADw9DT+ETQpUu4JkCqFyWI6s49DC7/7W/DV8AePeC22/DTTmfxEuPdd8Nkse+9Fz4EICSH3r3zk8W0aWE00rZtYdjhz38OP/xh+TYLzZ4Nf/pT6MuoVSs0O11/fWa3U7uHi7p27Sq4Pr62l8zzDRvCB32iD//Y444d+75/06ZhEuHvfS/0JcUSQseO6jvKFEoQmSI3N1xQMWZMmDTmhBPChXb9+u0tsnp16PB+770w/HD27DC8tl69cPH2z38e8ksqLVsW7uX9yCOhHfvss+G000KNonXr8NiyZUgiVcmePWHo79q1oeZWeEm0fv36sF95q1cvfPDHltat931s3bpsk7xJ9aAEkWl27QpTx95yC3zzDZxxBtx2G3Tvvk/R7dvD6JPOnSv+0orvvguziIwfH5q34pmFJq5YwogthV8ffHDFdX5v2RLybmzitviJ3FavDlNHFKVu3dA8U3hp3jyc9/gP6vh/yWSfN2lS8MO/SRMNKZXkKEFkqm3bwv1P77gjtEGcdx6MHRum7qhE9uwJ13l8801Yvv46/3n869WrQwd6Yc2ahWTStGnBpVmzfdfFlsaN9/0Azc0NTTHxCSB+KXzTwsaNQ9NMhw7hQzn2gZ8oEdSvrw9sqZyUIDLdpk2hTefPfw5VhpEjQzNUu3bpjqxEcnNDrSNR8ohvsoktxU3WVrMmZGWFZJGVFcqvWBGa2+LLxKZyLjyLZ8eOYT+Rqk4JQoLvvoPbb4e//jW8vvJKuPFGOOig9MaVIrt2hYrT+vX7Jo/YElvfpMm+SaBt2/RM5SxSkZQgpKAvvwz9E488Ehq/r7kGrrtOX4lFMlBxCUJzKGaidu3gwQdhwYIwbextt4V1110Xbp4tIoISRGaL3Wdi7lwYOjRMpNSxY7hJ0aJF6Y5ORNJMCULClVFPPAFLl4bLpidMgOzsMNPeBx+kOzoRSRMlCMnXvj3ce2+45elNN4X5xo8/Hvr3D9N1VpP+KhFJjhKE7KtFC/jDH0KiuPvucHXYoEFhLoYnnig4FlREqi0lCClagwZhhNOyZWEO77w8uPhi6NQp1DS2bUt3hCKSQkoQsn+1a4eZ9ebNg8mToU0b+MUvwsinP/whf8pYEalWUpogzGygmS02s6VmdkOC7SPNbI2ZzYmWn8RtG2Fmn0XLiFTGKUmqUSPcGOLdd8Ny/PFw880hUfz0p/DRR+qnEKlGUpYgzKwmMB44A8gGhptZdoKiz7h7t2h5KNq3KTAGOAboA4wxM13FVZn07RtqE598AuefH/omjj02zBd99937TlwkIlVOKmsQfYCl7r7c3XcBE4ChSe57OjDN3de7+wZgGjAwRXFKWRx1FDz8cJgQ6YEHQr/FtdeGaUXPPRdeeSVMoiQiVU4qE8QhwFdxr1dG6wobZmbzzGyimbUtyb5mdoWZzTSzmWv0jTW9GjWCyy+HDz+E+fPh6qvh7bfhzDPD7eh+97swJaqIVBnp7qR+EWjv7l0JtYTHSrKzuz/g7r3cvVeLFi1SEqCUQk5OmD121SqYODEMj7399jAL3kknheao7dvTHaWI7EcqE8QqoG3c6zbRur3cfZ2774xePgT0THZfqQLq1IFhw+Dll8M1FbfeGiYKvPjicLefn/0MZs5Ux7ZIJZXKBDED6GRmHcysDnABMDm+gJnF37Z+CBCbAGgqcJqZZUWd06dF66SqatMm3DP7s8/gzTfDaKhHHgk3xu7WLdQ4NFGgSKWSsgTh7nuAqwgf7IuAZ919gZmNNbMhUbFfmNkCM5sL/AIYGe27HriFkGRmAGOjdVLV1agBAwbA44+H28jdd1+4H+evfx2Gyw4YAH/7m66tEKkEdD8IqRw++yxMEvjUU/Dpp1CrFpx2Glx4YZhptkGDdEcoUi3pfhBS+XXqBL//PSxcCB9/HIbKzp8PP/pRuOPd+efDpEmwc+f+jyUi5UIJQioXs9AnceedYZLAd9+FSy8N/RZnnw0tW8KPfwyvvabrK0RSTAlCKq8aNcIV2+PHw9dfw7//HZqbnnsOTj01XIz3y1+Gay+qSVOpSGWiBCFVQ61acPrpYVbZ1avD9RX9+oUO7eOOgw4dQkf3f/6jZCFSTpQgpOo54IBwfcXEiSFZPPpomPJj3Dg45pj8ZKHJA0XKRAlCqrbGjWHECHjpJfjuu4LJ4thjw13yrrtOyUKkFJQgpPpo0qRgsnjssTC77L33FkwW6rMQSYoShFRPTZqEmxzFJ4uuXUOyOO64MIHgtdcqWYgUQxfKSWbZuDHcx+K55+DVV2HXLmjbFs46CwYPhv79w5XdIhmiuAvllCAkc23alJ8spk2DHTugfv1wBffgwWGq8oMPTneUIimlBCGyP9u2hYvxXnopLLGJA3v3Dsli8GDo3j1cyCdSjShBiJSEO8ybl58sYiOgWreGQYNCsjjllFDbEKnilCBEyuK778KtU196CaZOhS1bQj/FySeHZDFoUOj0FqmClCBEysuuXTB9en7tYunSsL5zZzjhhPylQwc1R0mVoAQhkipLloRE8dZbYWLBDRvC+tatQ6I48cTwmJMT5pYSqWSUIEQqQl5emK58+vSwvPNOuC83QFZWmHgwVsPo2TPcklUkzZQgRNLBHVasyE8Y06fD4sVh2wEHhHmjYrWMY4/VTZEkLZQgRCqL774LTVGxhPHxx6HmUbNmuA9G375hltq+fUMzlUiKKUGIVFabN8MHH8B774XE8dFH4ZoMCB3dsWTRrx8ceaT6MaTcKUGIVBW7d8OcOSFZxJLG6tVhW1YWHH98fsLo3Rvq1UtvvFLlKUGIVFXusGxZfrJ47z1YtChsq1MndHbHOr/79YOmTdMbr1Q5ShAi1cnatfD++/lJY+bMcH0GhOG0saG1J5wAbdqkN1ap9NKWIMxsIPD/gJrAQ+5+RxHlhgETgd7uPtPM2gOLgGjIBx+6+6ji3ksJQjLWjh0wY0YYVjt9ekgcW7eGbR06FLyAr3NnXcAnBRSXIGql8E1rAuOBU4GVwAwzm+zuCwuVawj8Evio0CGWuXu3VMUnUm3Uq5efAAD27IG5c/NHSk2ZAv/4R9h20EH5ZU88Mdwjo2bN9MUulVrKEgTQB1jq7ssBzGwCMBRYWKjcLcCdwPUpjEUkc9SqFfomevaEa64J/RiLF+dfvDd9Ojz/fCjbqFHo+D7xxHAvjF69dAGf7JXKBHEI8FXc65XAMfEFzKwH0NbdXzazwgmig5l9DGwGfufu01MYq0j1ZQZHHBGWyy8P6778suAV3zfeGNYfcEBIGP37h6VPH42UymCpTBDFMrMawJ+BkQk2fwO0c/d1ZtYTmGRmOe6+udAxrgCuAGjXrl2KIxapRtq1g4suCgvAmjUhUbz9dljGjAk1j7p1w1XesYRx7LFw4IHpjV0qTMo6qc3sOOBmdz89ev0/AO5+e/S6MbAMiHrTOBhYDwxx95mFjvUW8OvC6+Opk1qkHK1fH0ZIxRJG7Irv2rVDrSKWMI4/XlOEVHFpGcVkZrWAJcApwCpgBnChuy8oovxbREnAzFoA690918w6AtOBLu6+vqj3U4IQSaFNm8LoqFjCmDkTcnNDB3evXiFR9O4dlu99TyOlqpC0jGJy9z1mdhUwlTDM9WF3X2BmY4GZ7j65mN1PBMaa2W4gDxhVXHIQkRRr3Djco/vMM8PrrVvDFCGxhHHffXD33WFbkyYhafTunf/Ypo2SRhWkC+VEpOz27IEFC8L1GDNnhsd588J6gJYt900aBx2U3pgF0JXUIpIOO3aEJDFjRn7iWLgwdH5D6CiPJYvu3aFLF2jVSjWNCpaWJiYRyXD16oUO7T598tdt3QqzZ+fXMmbMgBdeyN/etCkcdVRIFvGPjRtXfPyiBCEiFahBg3BR3okn5q9bvx4++SR/mT8fHn88TIUe07ZtSBbxieOII8IwXEkZJQgRSa+mTfOHzca4w1dfFUwan3wC06aFKdEhjKDq3Dkki6OPDs1VPXtCs2bp+TmqISUIEal8zEIfRbt2MGhQ/vrdu+GzzwomjZkz4dln88u0b5+fLGKPWVkV/iNUB+qkFpGqb+PG0Lcxa1ZIGLNmhftoxHTsWDBp9OgRhuOKOqlFpJpr0gROPjksMevXF0waM2YUrGkcdlh+0ujaFQ4/PPR16Laue6kGISKZY926kDDiaxpffJG//YADoFOnkCwKL40apS/uFNJ1ECIiRVmzJlzkt3hxWJYsCY/Ll4f5p2JatkycODp0CFOsV1FqYhIRKUqLFjBgQFji7doV+jFiiSO2vPBCqInE1K4dhtzGXyXepUu1GIKrBCEikkidOnDkkWEpbN26gklj7lz417/g4Yfz9+3atWDSOPLIKlfTUBOTiEh5cA/9GfFTi8ycCVu2hO0HHhimFIlPGocdlvZOcfVBiIikQ15euG4jPml8/DFs3x62N24cRlHF5qLq0iXUNA44oMJCVIIQEaks9uwJkxbGksaMGaGTfOfOsL1GjVCzOOqogvNRHXZYSpqolCBERCqzPXtg6dL8q8Njj0uX5s9+W7duqF3EEkYseZTxXhtKECIiVdG2bbBoUUgY8clj1ar8Mo0bw8CBMGFCqd5Cw1xFRKqiAw8MfRQ9exZcv359aJaKJYwUTRuiBCEiUtU0bQonnBCWFNKkIyIikpAShIiIJKQEISIiCSlBiIhIQkoQIiKSkBKEiIgkpAQhIiIJKUGIiEhC1WaqDTNbA3yx34JFaw6sLadwUkHxlY3iKxvFVzaVOb5D3b1Fog3VJkGUlZnNLGo+kspA8ZWN4isbxVc2lT2+oqiJSUREElKCEBGRhJQg8j2Q7gD2Q/GVjeIrG8VXNpU9voTUByEiIgmpBiEiIgkpQYiISEIZlSDMbKCZLTazpWZ2Q4Ltdc3smWj7R2bWvgJja2tmb5rZQjNbYGa/TFBmgJltMrM50XJTRcUXF8MKM/skev997vFqwbjoHM4zsx4VGNvhcedmjpltNrNrCpWp0HNoZg+b2XdmNj9uXVMzm2Zmn0WPWUXsOyIq85mZjajA+P7XzD6Nfn//NLOEtyvb399CCuO72cxWxf0Ozyxi32L/31MY3zNxsa0wszlF7Jvy81dm7p4RC1ATWAZ0BOoAc4HsQmV+BtwfPb8AeKYC42sF9IieNwSWJIhvAPBSms/jCqB5MdvPBF4BDDgW+CiNv+9vCRcBpe0cAicCPYD5cev+BNwQPb8BuDPBfk2B5dFjVvQ8q4LiOw2oFT2/M1F8yfwtpDC+m4FfJ/H7L/b/PVXxFdr+f8BN6Tp/ZV0yqQbRB1jq7svdfRcwARhaqMxQ4LHo+UTgFDOzigjO3b9x99nR8y3AIuCQinjvcjYU+IcHHwJNzKxVGuI4BVjm7mW5ur7M3P0dYH2h1fF/Z48BZyXY9XRgmruvd/cNwDRgYEXE5+6vuvue6OWHQJvyft9kFXH+kpHM/3uZFRdf9NlxHvB0eb9vRcmkBHEI8FXc65Xs+wG8t0z0D7IJaFYh0cWJmra6Ax8l2Hycmc01s1fMLKdCAwsceNXMZpnZFQm2J3OeK8IFFP2Pme5z2NLdv4mefwu0TFCmspzHywg1wkT297eQSldFTWAPF9FEVxnO3wnAanf/rIjt6Tx/ScmkBFElmFkD4HngGnffXGjzbEKTydHAvcCkio4P6OfuPYAzgJ+b2YlpiKFYZlYHGAI8l2BzZTiHe3loa6iUY83N7LfAHuDJIoqk62/hPuB7QDfgG0IzTmU0nOJrD5X+fymTEsQqoG3c6zbRuoRlzKwW0BhYVyHRhfesTUgOT7r7C4W3u/tmd98aPZ8C1Daz5hUVX/S+q6LH74B/Eqry8ZI5z6l2BjDb3VcX3lAZziGwOtbsFj1+l6BMWs+jmY0EBgMXRUlsH0n8LaSEu69291x3zwMeLOJ9033+agHnAM8UVSZd568kMilBzAA6mVmH6BvmBcDkQmUmA7HRIucCbxT1z1HeovbKvwOL3P3PRZQ5ONYnYmZ9CL+/ikxg9c2sYew5oTNzfqFik4FLotFMxwKb4ppTKkqR39zSfQ4j8X9nI4B/JSgzFTjNzLKiJpTTonUpZ2YDgd8AQ9x9WxFlkvlbSFV88X1aZxfxvsn8v6fS94FP3X1loo3pPH8lku5e8opcCCNslhBGN/w2WjeW8I8AUI/QLLEU+A/QsQJj60doapgHzImWM4FRwKiozFXAAsKIjA+B4yv4/HWM3ntuFEfsHMbHaMD46Bx/AvSq4BjrEz7wG8etS9s5JCSqb4DdhHbwHxP6tV4HPgNeA5pGZXsBD8Xte1n0t7gUuLQC41tKaL+P/R3GRva1BqYU97dQQfE9Hv1tzSN86LcqHF/0ep//94qIL1r/aOxvLq5shZ+/si6aakNERBLKpCYmEREpASUIERFJSAlCREQSUoIQEZGElCBERCQhJQiR/TCz3EKzxJbbzKBm1j5+JlCRyqRWugMQqQK2u3u3dAchUtFUgxAppWg+/z9Fc/r/x8wOi9a3N7M3osnkXjezdtH6ltH9FeZGy/HRoWqa2YMW7gPyqpkdEJX/hYX7g8wzswlp+jElgylBiOzfAYWamM6P27bJ3bsAfwHuidbdCzzm7l0JE92Ni9aPA972MFFgD8IVtACdgPHungNsBIZF628AukfHGZWqH06kKLqSWmQ/zGyruzdIsH4FcLK7L48mWvzW3ZuZ2VrC9A+7o/XfuHtzM1sDtHH3nXHHaE+470On6PVooLa732pm/wa2EmacneTRJIMiFUU1CJGy8SKel8TOuOe55PcNDiLMa9UDmBHNECpSYZQgRMrm/LjHD6Ln7xNmDwW4CJgePX8duBLAzGqaWeOiDmpmNYC27v4mMJow9fw+tRiRVNI3EpH9O6DQjef/7e6xoa5ZZjaPUAsYHq27GnjEzK4H1gCXRut/CTxgZj8m1BSuJMwEmkhN4IkoiRgwzt03lttPJJIE9UGIlFLUB9HL3demOxaRVFATk4iIJKQahIiIJKQahIiIJKQEISIiCSlBiIhIQkoQIiKSkBKEiIgk9P8BxxJWPnUFjkYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "#-----------------------------------------------------------\n",
    "# Retrieve a list of list results on training and test data\n",
    "# sets for each training epoch\n",
    "#-----------------------------------------------------------\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = [*range(20)]\n",
    "\n",
    "#------------------------------------------------\n",
    "# Plot training and validation loss per epoch\n",
    "#------------------------------------------------\n",
    "plt.plot(epochs, loss, 'r')\n",
    "plt.plot(epochs, val_loss, 'b')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend([\"Loss\", \"Validation Loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wish so, you can also check the training and validation accuracies of your model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3iUVfbA8e+hN+kuIEVQ6QsRCE1BQCxgAQELqCjqgg0Lq66sroio+7N3xUXsBVBcEKUtiEpQEAIiVZBOEJXeQoAk5/fHfYdMhkkyIXkzKefzPPPMzNvmzJvJnLn3vvdeUVWMMcaYUMWiHYAxxpj8yRKEMcaYsCxBGGOMCcsShDHGmLAsQRhjjAnLEoQxxpiwLEGYiInIdBG5Mbe3jSYR2SQiF/hwXBWRs7zHb4rII5FsexKvc52I/O9k4zQmM2L9IAo3ETkY9LQccARI8Z7fqqof531U+YeIbAL+pqqzc/m4CjRU1XW5ta2I1Ac2AiVVNTk34jQmMyWiHYDxl6pWCDzO7MtQRErYl47JL+zzmD9YFVMRJSJdRSRBRB4Ukd+Bd0Wkioh8JSI7RGSP97hO0D7fisjfvMeDRGSeiDznbbtRRHqe5LYNRGSuiBwQkdki8rqIfJRB3JHE+LiIfO8d738iUj1o/UAR2Swiu0Tk4UzOT3sR+V1Eigct6yMiy7zH7URkvojsFZHtIvKaiJTK4FjvicgTQc8f8Pb5TURuDtn2UhH5SUT2i8hWERkZtHqud79XRA6KSMfAuQ3a/xwRWSQi+7z7cyI9N9k8z1VF5F3vPewRkclB63qLyFLvPawXkR7e8nTVeSIyMvB3FpH6XlXbLSKyBZjjLf/M+zvs8z4jzYP2Lysiz3t/z33eZ6ysiEwVkbtC3s8yEekT7r2ajFmCKNpqAlWB04EhuM/Du97zesBh4LVM9m8PrAGqA88Ab4uInMS2nwALgWrASGBgJq8ZSYzXAjcBfwFKAfcDiEgzYLR3/NO816tDGKr6I3AIOD/kuJ94j1OAYd776Qh0B+7IJG68GHp48VwINARC2z8OATcAlYFLgdtF5Apv3XnefWVVraCq80OOXRWYCrzivbcXgKkiUi3kPZxwbsLI6jx/iKuybO4d60UvhnbAB8AD3ns4D9iU0fkIowvQFLjYez4dd57+AiwBgqtEnwPaAOfgPsf/AFKB94HrAxuJSAxQG3duTHaoqt2KyA33j3qB97grcBQok8n2ZwN7gp5/i6uiAhgErAtaVw5QoGZ2tsV9+SQD5YLWfwR8FOF7Chfjv4Ke3wHM8B6PAMYHrSvvnYMLMjj2E8A73uNTcF/ep2ew7b3ApKDnCpzlPX4PeMJ7/A7wVNB2jYK3DXPcl4AXvcf1vW1LBK0fBMzzHg8EFobsPx8YlNW5yc55BmrhvoirhNnuP4F4M/v8ec9HBv7OQe/tjExiqOxtUwmXwA4DMWG2KwPswbXrgEskb+T1/1thuFkJomjboapJgSciUk5E/uMV2ffjqjQqB1ezhPg98EBVE72HFbK57WnA7qBlAFszCjjCGH8PepwYFNNpwcdW1UPAroxeC1da6CsipYG+wBJV3ezF0cirdvndi+PfuNJEVtLFAGwOeX/tReQbr2pnH3BbhMcNHHtzyLLNuF/PARmdm3SyOM91cX+zPWF2rQusjzDecI6fGxEpLiJPedVU+0kriVT3bmXCvZb3mZ4AXC8ixYABuBKPySZLEEVb6CVs9wGNgfaqWpG0Ko2Mqo1yw3agqoiUC1pWN5PtcxLj9uBje69ZLaONVXUV7gu2J+mrl8BVVf2C+5VaEXjoZGLAlaCCfQJMAeqqaiXgzaDjZnXJ4W+4KqFg9YBtEcQVKrPzvBX3N6scZr+twJkZHPMQrvQYUDPMNsHv8VqgN64arhKulBGIYSeQlMlrvQ9ch6v6S9SQ6jgTGUsQJtgpuGL7Xq8++1G/X9D7RR4PjBSRUiLSEbjcpxgnApeJSCevQXkUWf8PfALcg/uC/Cwkjv3AQRFpAtweYQyfAoNEpJmXoELjPwX36zzJq8+/NmjdDlzVzhkZHHsa0EhErhWREiJyDdAM+CrC2ELjCHueVXU7rm3gDa8xu6SIBBLI28BNItJdRIqJSG3v/AAsBfp728cCV0YQwxFcKa8crpQWiCEVV133goic5pU2OnqlPbyEkAo8j5UeTpolCBPsJaAs7tfZAmBGHr3udbiG3l24ev8JuC+GcE46RlVdCdyJ+9LfjqunTshit3G4htM5qrozaPn9uC/vA8BbXsyRxDDdew9zgHXefbA7gFEicgDXZvJp0L6JwJPA9+KunuoQcuxdwGW4X/+7cI22l4XEHamszvNA4BiuFPUnrg0GVV2IawR/EdgHfEdaqeYR3C/+PcBjpC+RhfMBrgS3DVjlxRHsfmA5sAjYDTxN+u+0D4AWuDYtcxKso5zJd0RkAvCLqvpegjGFl4jcAAxR1U7RjqWgshKEiToRaSsiZ3pVEj1w9c6Ts9rPmIx41Xd3AGOiHUtBZgnC5Ac1cZdgHsRdw3+7qv4U1YhMgSUiF+Paa/4g62oskwmrYjLGGBOWlSCMMcaEVWgG66tevbrWr18/2mEYY0yBsnjx4p2qemq4dYUmQdSvX5/4+Phoh2GMMQWKiIT2vj/OqpiMMcaEZQnCGGNMWL4mCBHpISJrRGSdiAwPs76eNzDZT9547Zd4yy8UkcUisty7P//EoxtjjPGTb20Q3qiPr+PGvU8AFonIFG8AtIB/AZ+q6mhvrP5puAG5dgKXq+pvIvJXYCbpR6SMyLFjx0hISCApKSnrjU2RUaZMGerUqUPJkiWjHYox+ZqfjdTtcHMAbAAQkfG4HrLBCUKBit7jSrjRKAnpJLUSKCsipVU1o/F5wkpISOCUU06hfv36ZDyPjSlKVJVdu3aRkJBAgwYNoh2OMfman1VMtUk/7n0CJ5YCRuLGbE/AlR7u4kT9cOPwn5AcRGSIiMSLSPyOHTtO2DEpKYlq1apZcjDHiQjVqlWzUqUxEYh2I/UA4D1VrQNcAnzoTfABgDf/7NPAreF2VtUxqhqrqrGnnhr2Ml5LDuYE9pkwJjJ+VjFtI/3EKHU4ceKSW4Ae4MZvF5EyuJmi/hQ3Qfok4AZVzckMVcYYU7ikpsLmzbBmDfzyC5QtC7eG/R2dI34miEVAQxFpgEsM/Uk/+QnAFtyMT++JSFPcFII7vJmqpgLDVfV7H2PMM5MnT6ZPnz6sXr2aJk2aZL2DMcYcPOiSQCAR/PKLe7x2LQRXk3bsWLAShKomi8hQ3BVIxXGTv68UkVFAvKpOwU1s8paIDMM1WA9SVfX2OwsYISIjvENepKp/+hWv38aNG0enTp0YN24cjz32mC+vkZKSQvHiGU0fbYzJl1Rh69a0JBCcDLYFVboUKwZnnAGNG8OFF7r7Jk3cfQZV7LkQmxaKW5s2bTTUqlWrTlgWDQcOHNDTTjtN16xZo40aNVJV1eTkZL3vvvu0efPm2qJFC33llVdUVXXhwoXasWNHbdmypbZt21b379+v7777rt55553Hj3fppZfqN998o6qq5cuX17///e/asmVLjYuL08cee0xjY2O1efPmOnjwYE1NTVVV1V9//VW7d++uLVu21FatWum6det04MCBOmnSpOPHvfbaa3Xy5Ml5dFaiK798NkwRdfiw6tSpqoMHq9asqerShLtVrKjarp3qDTeoPvmk6uefq65YoZqU5EsouB/sYb9XC81YTFm6915YujR3j3n22fDSS1lu9sUXX9CjRw8aNWpEtWrVWLx4MQsXLmTTpk0sXbqUEiVKsHv3bo4ePco111zDhAkTaNu2Lfv376ds2bKZHvvQoUO0b9+e559/HoBmzZoxYoQrdA0cOJCvvvqKyy+/nOuuu47hw4fTp08fkpKSSE1N5ZZbbuHFF1/kiiuuYN++ffzwww+8//77OT8vxpgT7d4NU6fCF1/AjBlw6BBUqAA9e8L557vSQJMmUKMG5JMLKYpOgoiicePGcc899wDQv39/xo0bx8aNG7ntttsoUcL9CapWrcry5cupVasWbdu2BaBixYoZHjOgePHi9OvX7/jzb775hmeeeYbExER2795N8+bN6dq1K9u2baNPnz6A6ygG0KVLF+644w527NjB559/Tr9+/Y7HY4zJBRs3uoTwxRcQFwcpKVCrFlx/PVxxBXTrBqVLRzvKDBWdb4MIfun7Yffu3cyZM4fly5cjIqSkpCAix5NAJEqUKEFqaurx58HX8JcpU+Z4u0NSUhJ33HEH8fHx1K1bl5EjR2Z5vf8NN9zARx99xPjx43n33Xez+e6MMemowpIlaUlh2TK3vHlzePBB6N0bYmNde0IBUDCiLMAmTpzIwIED2bx5M5s2bWLr1q00aNCAmJgY/vOf/5CcnAy4RNK4cWO2b9/OokWLADhw4ADJycnUr1+fpUuXkpqaytatW1m4cGHY1wokg+rVq3Pw4EEmTpwIwCmnnEKdOnWYPNlN83zkyBESExMBGDRoEC95ybNZs2b+nQhjCqujR2HWLLjzTqhXzyWAJ5+ESpXguefg119hxQq3rF27ApMcoCiVIKJk3LhxPPjgg+mW9evXj9WrV1OvXj1atmxJyZIlGTx4MEOHDmXChAncddddHD58mLJlyzJ79mzOPfdcGjRoQLNmzWjatCmtW7cO+1qVK1dm8ODB/PWvf6VmzZrpSikffvght956KyNGjKBkyZJ89tlnnHHGGdSoUYOmTZtyxRVX+HoejCk0fvsN5s+HBQvc/eLF7pLTsmXhoovg8cfh0kv9u7IoDxWaOaljY2M1dMKg1atX07Rp0yhFVDAkJibSokULlixZQqVKlaIdTp6xz4aJyNGj8NNPaclg/nzYssWtK1UK2rSBDh2ga1e44AIoVy6q4Z4MEVmsqrHh1lkJogibPXs2t9xyC8OGDStSycGYDG3bdmLp4Ig3DFzduq5D2r33uvtWrfJ1A3NusARRhF1wwQVs3pzhbIPGFG7JyfDzzzBvHvzwg0sIW73xRUuXdqWDoUNdMujQAWpne8aBAs8ShDGmaDh40JUM5s1ztwULXF8EcI3L55zjkkHHjq6PU6lS0Y03H7AEYYwpnLZvh++/T0sIS5e6fgjFikFMDNx0E3TqBOeeC3XqRDvafMkShDGm4FN1YxcFksG8ebBhg1tXtqyrInroIZcQOnSACDqhGksQxpiCaO9eWLQIfvzR3ebPh1273Lq//MUlgqFDXemgVSuw6WVPSsHpsVEAdevWjZkzZ6Zb9tJLL3H77bdnuE/Xrl0JXK57ySWXsHfv3hO2GTlyJM8991ymrz158mRWrUqb3XXEiBHMnj07O+Fn6t5776V27drpengb44tjx9zVRG+8ATfe6MYrqlLF9Tl45BFYvx569YJ33nHDYP/+O3z+OQwb5jqmWXI4aVaC8NGAAQMYP348F1988fFl48eP55lnnolo/2nTpp30a0+ePJnLLrvseO/oUaNGnfSxQqWmpjJp0iTq1q3Ld999R7du3XLt2MGSk5NtbKiiRhU2bUorGfz4o+uHEBgypkYNaN8ebrjBffm3bet6LBtfWAnCR1deeSVTp07l6NGjAGzatInffvuNzp07c/vttxMbG0vz5s159NFHw+5fv359du7cCcCTTz5Jo0aN6NSpE2vWrDm+zVtvvUXbtm2JiYmhX79+JCYm8sMPPzBlyhQeeOABzj77bNavX8+gQYOOD73x9ddf06pVK1q0aMHNN9/MEe867/r16/Poo4/SunVrWrRowS+//BI2rm+//ZbmzZtz++23M27cuOPL//jjD/r06UNMTAwxMTH88MMPAHzwwQe0bNmSmJgYBg4cCJAuHoAKFSocP3bnzp3p1avX8eR2xRVX0KZNG5o3b86YMWOO7zNjxgxat25NTEwM3bt3JzU1lYYNGxKYnzw1NZWzzjqLcPOVm3wiORm++871Pr78cpcAzjgDBgyAN990Dcp33AETJrjEsX27G+PooYdcxzRLDr7y9eeZiPQAXsZNGDRWVZ8KWV8PeB+o7G0zXFWneev+iZuSNAW4W1XT19VkUzRG+65atSrt2rVj+vTp9O7dm/Hjx3P11VcjIjz55JNUrVqVlJQUunfvzrJly2jZsmXY4yxevJjx48ezdOlSkpOTad26NW3atAGgb9++DB48GIB//etfvP3229x111306tWLyy67jCuvvDLdsZKSkhg0aBBff/01jRo14oYbbmD06NHce++9gBvHacmSJbzxxhs899xzjB079oR4xo0bx4ABA+jduzcPPfQQx44do2TJktx999106dKFSZMmkZKSwsGDB1m5ciVPPPEEP/zwA9WrV2f37t1ZntclS5awYsUKGjRoAMA777xD1apVOXz4MG3btqVfv36kpqYyePBg5s6dS4MGDdi9ezfFihXj+uuv5+OPP+bee+9l9uzZxMTEkNF85SZKEhPd2EWTJ8OXX7q2AxFXdXTppa6E0L49/PWvVj0UZb6VIESkOPA60BNoBgwQkdDR4P4FfKqqrXBTkr7h7dvMe94cN2f1G97xCpxANRO46qUBAwYA8Omnn9K6dWtatWrFypUr07UXhIqLi6NPnz6UK1eOihUr0qtXr+PrVqxYQefOnWnRogUff/wxK1euzDSeNWvW0KBBAxo1agTAjTfeyNy5c4+v79u3LwBt2rRh06ZNJ+x/9OhRpk2bxhVXXEHFihVp37798XaWOXPmHG9fKV68OJUqVWLOnDlcddVVVK9eHXBJMyvt2rU7nhwAXnnlFWJiYujQoQNbt27l119/ZcGCBZx33nnHtwsc9+abb+aDDz4AXGK56aabsnw9kwd274YPPoC+faF6dTfU9eTJbi6EiRNhzx5YtQrefRduu80alvMJP0sQ7YB1qroBQETGA72B4G9CBQLXm1UCfvMe9wbGq+oRYKOIrPOON/9kg4nSaN/07t2bYcOGsWTJEhITE2nTpg0bN27kueeeY9GiRVSpUoVBgwZlOSx3RgYNGsTkyZOJiYnhvffe49tvv81RvKW9oQOKFy9+fKTZYDNnzmTv3r20aNECcGM5lS1blssuuyxbrxM8hHlqaurxajiA8uXLH3/87bffMnv2bObPn0+5cuXo2rVrpueqbt261KhRgzlz5rBw4UI+/vjjbMVlctGWLa46aNIkmDvX9UGoXRtuvtkliC5dLAnkc362QdQGtgY9T/CWBRsJXC8iCcA04K5s7IuIDBGReBGJz6/1zBUqVKBbt27cfPPNx0sP+/fvp3z58lSqVIk//viD6dOnZ3qM8847j8mTJ3P48GEOHDjAl19+eXzdgQMHqFWrFseOHUv3ZXjKKadw4MCBE47VuHFjNm3axLp16wA3ymuXLl0ifj/jxo1j7NixbNq0iU2bNrFx40ZmzZpFYmIi3bt3Z/To0YCbH3vfvn2cf/75fPbZZ+zyLkEMVDHVr1+fxYsXAzBlyhSOHTsW9vX27dtHlSpVKFeuHL/88gsLFiwAoEOHDsydO5eNGzemOy7A3/72N66//nquuuoqm6M7L6m6Ya2feMINU3H66XD33e6qogcfhIULXdJ47TXXfmDJId+LdiP1AOA9Va0DXAJ8KCIRx6SqY1Q1VlVj83M984ABA/j555+PJ4iYmBhatWpFkyZNuPbaazn33HMz3b9169Zcc801xMTE0LNnz3TDeD/++OO0b9+ec889lyZNmhxf3r9/f5599llatWrF+vXrjy8vU6YM7777LldddRUtWrSgWLFi3HbbbRG9j8TERGbMmMGll156fFn58uXp1KkTX375JS+//DLffPMNLVq0oE2bNqxatYrmzZvz8MMP06VLF2JiYvj73/8OwODBg/nuu++IiYlh/vz56UoNwXr06EFycjJNmzZl+PDhdOjQAYBTTz2VMWPG0LdvX2JiYrjmmmuO79OrVy8OHjxo1Ut5ISXFdUp74AFo2BBatHCXnpYuDU8/7TqvrVrl5kJo27ZAzYVgfBzuW0Q6AiNV9WLv+T8BVPX/grZZCfRQ1a3e8w1AB1zj9PFtRWSmd6wMq5hsuG8TEB8fz7Bhw4iLi8twG/ts5MChQ66R+Ysv4KuvYOdOVxro3t1VHfXq5abVNAVCtIb7XgQ0FJEGwDZco/O1IdtsAboD74lIU6AMsAOYAnwiIi8ApwENgfDTqBkT5KmnnmL06NHW9pDbtm93yeCLL2D2bDcEduXKcMklLiH06GGXnBZCviUIVU0WkaHATNwlrO+o6koRGQXEq+oU4D7gLREZhmuwHqSuSLNSRD7FNWgnA3eqaopfsZrCY/jw4QwfPjzaYRR8qrByJUyZ4m4//uiW16/vrjLq1Qs6d7Z2hELO134QXp+GaSHLRgQ9XgWErYBX1SeBJ3MhBkQkp4cxhUhhmUUx1yUnQ1xcWlIIDHbXrp1reO7Vy/VNsP+nIqNQj2NQpkwZdu3aRbVq1SxJGMAlh127dlGmTJloh5I/7Nnj2hOmTIFp09zz0qXdVUYPPgiXXQannRbtKE2UFOoEUadOHRISEmyoBZNOmTJlqFNUx/9XdUMKTJsG06e7UVBTU6FaNVdC6N0bLrwQvKFPTNFWqBNEyZIl0/XINaZI2rvXlRKmTYMZM1y/BHB9FR56yPVmbt8erM+ICVGoE4QxRZKqm2t5+nSXFObPd/0VAkNk9+zprjqqUSPakZp8zhKEMYXBvn2ulDB9urtt3+6Wt24Nw4enlRJs+HSTDfZpMaag+v13+O9/4bPP3NVHKSmuL8LFF6eVEmrWjHaUpgCzBGFMQRKcFL77zlUnNW0K//iH67TWoYOVEkyusU+SMfldRknhkUfgqqugeXPrm2B8YQnCmPzIkoLJByxBGJNfWFIw+YwlCGOiaf16d9XR559bUjD5jiUIY/JSYqJLBIHLUb2JmywpmPzIEoQxflKFtWtdD+bp011ySEqCsmWhWzc341rPnnDWWdGO1JgTWIIwJrcdOgTffJNWSvCmRaVxYzdUds+ebqjssmWjG6cxWbAEYUxOqcLq1WmlhLlz4ehRKF/ezbL2wAOu05qNC2YKGEsQxpys/fvh3Xfh9dfh11/dsubNXbVRjx7QqZMbOtuYAsrXBCEiPYCXcTPKjVXVp0LWvwh0856WA/6iqpW9dc8AlwLFgFnAPWozvZj8YMMGePVVePttOHAAzjkH7r/fJYV69aIdnTG5xrcEISLFgdeBC4EEYJGITPFmkQNAVYcFbX8X0Mp7fA5uprmW3up5QBfgW7/iNSZTqq6B+aWX3OQ6xYvDNdfAPfdA27bRjs4YX/hZgmgHrFPVDQAiMh7ojZtnOpwBwKPeYwXKAKUAAUoCf/gYqzHhJSXBuHEuMSxbBtWruzkU7rjDZlozhZ6fCaI2sDXoeQLQPtyGInI60ACYA6Cq80XkG2A7LkG8pqqrw+w3BBgCUM+K9iY3/f47jB7tbjt2uLmYx46Fa6+1q49MkZFfGqn7AxNVNQVARM4CmgKBeSFniUhnVY0L3klVxwBjAGJjY619wuTckiWutDB+PCQnuzmZ77kHzj/fOq+ZIsfPBLENqBv0vI63LJz+wJ1Bz/sAC1T1IICITAc6AnFh9jUmZ5KT4Ysv4OWX3bwKFSq4/gp33QUNG0Y7OmOippiPx14ENBSRBiJSCpcEpoRuJCJNgCrA/KDFW4AuIlJCREriGqhPqGIyJke2boXHHoMzz4Qrr3TPX3gBEhLglVcsOZgiz7cShKomi8hQYCbuMtd3VHWliIwC4lU1kCz6A+NDLmGdCJwPLMc1WM9Q1S/9itUUIceOwdSp8NZbrmNbaipceCG8+CL07u2uTjLGACCFpWtBbGysxsfHRzsMk19t2OAamd991zVA16oFN98Mt9xiPZxNkSYii1U1Nty6/NJIbUzuO3IEJk92pYWvv4Zixdy0nIMHu3ubmtOYTNl/iCl8Vq92pYUPPoCdO+H002HUKLjpJqhTJ+v9jTGAJQhTWCQmwsSJrrQwb54rHfTu7UoLF1xgbQvGnARLEKZg++MPeOop17awb5+bV+Hpp+HGG6FGjWhHZ0yBZgnCFEz79sFzz7mrj5KS4OqrYcgQ6NLFOrQZk0ssQZiCJSkJ3ngD/v1v2LXLDZj3+OPWZ8EYH/jZUc6Y3JOc7KqRGjWC++6DNm0gPt4NiWHJwRhfWIIw+Zuqu1S1ZUvXb6FWLXfJ6syZLkkYY3xjCcLkX9995ybj6dPH9Xj+/HNYsMANnGeM8Z0lCJP/LF0KPXtC165ufKSxY2HFCujb1xqgjclDliBM/rF+vZtvoVUr+PFHePZZN9fzLbdYr2djosD+60z0/f67uxJpzBgoVcrN2PbAA1C5crQjM6ZIswRhomf/fldKeOEFOHrU9Xp+5BHXEG2MiTpLECbvHTnipvJ84gnXl6F/f1eCOOusaEdmjAlibRAm76SkwIcfQuPGMGwYtG7t+jKMG2fJwZh8yNcEISI9RGSNiKwTkeFh1r8oIku921oR2Ru0rp6I/E9EVovIKhGp72esxkeqMG2aSwg33ADVq8OsWfC//1lfBmPyMd+qmESkOPA6cCGQACwSkSmquiqwjaoOC9r+LqBV0CE+AJ5U1VkiUgFI9StW46Mff4QHH3R9Gs480/V8vuoqNzeDMSZf8/O/tB2wTlU3qOpRYDzQO5PtBwDjAESkGVBCVWcBqOpBVU30MVaT29asgX79oEMHNz/D66/DqlVu7CRLDsYUCH7+p9YGtgY9T/CWnUBETgcaAHO8RY2AvSLyXxH5SUSe9UokofsNEZF4EYnfsWNHLodvTspvv8Gtt0Lz5q4KadQo17/hjjvcJazGmAIjv/yU6w9MVNUU73kJoDNwP9AWOAMYFLqTqo5R1VhVjT311FPzKlYTzt69rv/CWWe5QfWGDnXzQD/yCFSoEO3ojDEnwc8EsQ2oG/S8jrcsnP541UueBGCpVz2VDEwGWvsSpcmZxEQ3L8MZZ7iJe/r2ddVLL70ElrSNKdD8TBCLgIYi0kBESuGSwJTQjUSkCVAFmB+yb2URCXzDnA+sCt3XRNHhw/Dyyy4xPPAAtG8PS5bARx9BgwbRjs4Ykwt8u4pJVZNFZCgwEygOvHajROQAACAASURBVKOqK0VkFBCvqoFk0R8Yr6oatG+KiNwPfC0iAiwG3vIrVpMNR464eZ//7/9ce8P557u5oDt1inZkxphcJkHfywVabGysxsfHRzuMwuvoUXjnHXjySUhIgPPOg8cecyOuGmMKLBFZrKqx4dbll0Zqk18dO+aG227YEG6/HerVg9mz4dtvLTkYU8hZgjDhBab4bNzYDaJXq5abxW3ePOje3eZlMKYIsARh0ktOduMlNW3qpvisVg2mToX58+GiiywxGFOEZJkgRORyEbFEUtilpMAnn7gObjfc4PouTJkCCxfCJZdYYjCmCIrki/8a4FcReca7JNUUJqpurucWLeC666BMGfjvf90lq5dfbonBmCIsywShqtfjBtFbD7wnIvO9IS5O8T06468DB2DgQLjySpcIPvsMfvoJ+vSxxGCMiawNQlX3AxNxA+7VAvoAS7wRWE1BtHixG3573Dg3XtKyZS5R2EB6xhhPJG0QvURkEvAtUBJop6o9gRjgPn/DM7lOFV58ETp2hKQkd7nqI49A8RPGQjTGFHGR9KTuB7yoqnODF6pqoojc4k9Yxhc7dsBNN7mrknr3dh3fqlaNdlRRoeo6gsfHu1q1Q4dcjixRwt0Cj8MtC7e+aVPXvm81c6YwiSRBjAS2B56ISFmghqpuUtWv/QrM5LJvvnGN0Lt3w2uvueG3i9C32W+/uVq1+Pi0+z/+cOuKFXNt88nJ7mKulJTMj5WR2rXdlcAXXwwXXOCuEDamIIskQXwGnBP0PMVb1taXiEzuSk52bQxPPAGNGsH06RATE+2ofLV9e/pksHixWwYuGTRp4r7E27SB2Fh3OsqXT9tfNS1RJCenJY7Qx4H7I0fca82cCZMmuf6FItC2rXudiy92YxmW8GHks717Yd06qFzZTdhXhHK+yQNZjsUkIktV9eyQZT+rar76lrGxmMLYssWVGubNc1VLr76a/puwkIiLcwWkQEL47Te3XMQlg9jYtGRw9tn+noLkZFi0yCWLmTNdN5LUVKhUyXVADySM00+P/JhJSS4JrF2bdvv1V3f/559p21WpAu3auVvbtu6+Ro3cf4+mcMlsLKZIEsQs4NXA6Ksi0hu4W1W753qkOWAJIsTkya4n9LFj8OabLlEUMitWwP33uy9iETcqSCARtGkDrVpFf66iPXvc0FWBhJGQ4JY3bpyWLLp2hdKlYfPm9EkgcNuyxZVqAmrWdIXBwO2ss2DXLpeMFi6E5ctdUgI3dFYgabRr585LtM8JuPg2bHAXzy1b5kp2gffTsCGckocX0e/f7340FNXrNHKaIM4EPgZOAwQ3jegNqroutwPNCUsQnqQk9635+uvu22D8ePcNUoj8/juMGAFvvw0VK7qLsP72N/c4P1N103MHksV337k/V2Am1qNH07atWNElkYYN0yeDhg2zfp+HDrmG90DCWLTIfRmD+yJu1ix9SaNFCyhZ0p/3DO4LePly+Plnlwx+/tk9P3QoLSbV9EmwVq209xv8/s84wyXTSKWmus/L5s3utmXLiff79rnquQ8/dBf3FTU5ShBBB6kAoKoHczG2XGMJAvjlF7jmGvdfeN998O9/F6p5oA8fhhdecBPXJSXBnXe65FBQG4MPH3bVY7Nnp/8F3aiRm4wvN9sTdu50iSKQNBYudMvANdDXrw/Vq7tzGXwfblmVKuG7ywSXCoKTwcaNadtUruzafGJioGVLd9+smTve+vVppaY1a9IeB083X6yYizX4XDVq5M5V8Bd/4PHWra4QHaxKFVeyOv10d1+rFowZ47b917/czc+Emd/kOEGIyKVAc6BMYJmqjsq1CHNBkU4QqvDee24e6HLl4P333fhJhURqKnz8sZvyOiEBrrgCnn7afTGYk6MKmzalJYstW1w11c6daffBJZpgxYq5L9lA0qhWzX2Jh5YKGjVKSwKBhFCnTvYT3549aW0uwe0va9fCwZCfq8WKwWmnpX35h97Xqxe+BLZvH9x1lytFtG/v7hs2zF6cBVVOq5jeBMoB3YCxwJXAQlXNsg+EiPQAXsbNKDdWVZ8KWf+id1y81/iLqlYOWl8RN9XoZFUdmtlrFdkEceSIq1/56CPo1s3dn3ZatKPKNXPnwt//7hqf27SB55+HLl2iHVXhp+q+fEOTRvDj4GVVqpxYKihXzv8Yf//dlTZEXBKoXTtnv/4nTIDbbnOljpdegltuKfxXhuU0QSxT1ZZB9xWA6araOYv9igNrgQuBBNw80wNUNezc0t6wHa1U9eagZS8DpwK7LUGEkZTkhseYOtVdyvrQQxG3tKWmwoIF7rLMr792ueWxx/JHAya4X4n/+Idra69Tx81weu21NhKI8V9CAtx4I8yZ4/qTvvWWq/Lzw969rrSyYYNLssG3qlXTP/ertjizBIGqZnrDlRYAFuAaqksD6yLYryMwM+j5P4F/ZrL9D8CFQc/b4MZ+GgS8ltXrtWnTRouUxETVHj1c295//hPRLkeOqM6YoXrrrao1a7pdS5ZUbd/ePa5XT/Wrr3yOOwu7dqnec49qiRKqFSqoPvGE6qFD0Y3JFD0pKarPP69aqpRqjRqq06bl7vGXLXP/h+XKuf+98uUDzfQZ38qVU61dW7VFC9XzzlPt3Vt10CDVYcNU33zz5GMB4jWD79VIuu58KSKVgWeBJYACb0WwX23cFU8BCUD7cBuKyOlAA2CO97wY8DxwPXBBRi8gIkOAIQD16tWLIKRCIjHR/bT5+mt3Kc/NN2e46cGDrm/cpEmuoBG4pO+SS1xd/qWXumv0v/8ehgyByy6Dq6+Gl192l1PmlSNH3IVXjz/uYvzb31yJJi9jMCagWDFXtXnBBe4K8UsucRdFPPPMyVedHTvmSsSvveaqTsuUcce+8053SXZysitR7N7t2l2Cb+GWbdzoql737HH9e269NXfPAZB5CQI3mN85Qc9LA5Uy2ydo2ytx7Q6B5wPJoCQAPIjraxF4PhT4h/d4UEb7Bd+KTAni4EHVbt1URVTffz/sJn/+qfr226qXXaZaurT79VG9uurNN6t++aXq4cPhD33kiOrjj7t9KlVyBZOUFB/fi6oeO6Y6frzqmWe6OC++WHX5cn9f05jsOHzY/UoH1SZNVBcvzt7+27erPvaY6mmnuWM0aKD67LOutJxbkpNPfl8yKUFE8kX/U1bbZLBfxFVMwE8hiehjYAuwCdgJ7Aeeyuz1ikSC2L/flS2LFVP96KN0qzZtUn3xxbTVoHr66ar33qv63XfuizhSa9aodu3qjtGpk+qqVbn7NlRVt21THTnSFZlBtXlzV/1lTH41a5b7ki9RQvX//i/zL+XUVNXvv1cdMMBV44KrEf7qq5x9mfshpwniOdyIrpLVtiH7lQA24KqOSgE/A83DbNfESwRhj28lCM++farnnqtavLj7ya3uQzhpkmq7dnq8nrJFC9VHHlFdssStP1mpqarvvKNatar7gI8YkXHJIzvHnD1btV8/9zYCJYYvvsh//zTGhLNrl+qVV7rPbufOqhs3pl9/6JDq2LGqrVq5bSpVcj/S1q6NSrgRyWmCOACkAke9X/IHgP1Z7eftewnuSqb1wMPeslFAr6BtRmZWOrAEoap796p26OB+ukycqKmpqpMnq559tvsLNmzoiqy//pr7L/3HH6rXXedep3Fj1W+/zf4xdu92pZtGjdxxqlVTfeAB1XXrcj9eY/yWmupqd085RbViRdUPP1Rdv171/vtVq1RJ+6H2n/+4GuH8LkcJoqDcCm2C2L1btW1b1ZIlNfW/k/SLL9J+nZx1luoHH2Sv+uhkzZjh6k5B9ZZbIqs/XbhQ9aabVMuWdft17OjizWlJxJj8YMMGV6gPlN6LF1e96ipXpZuT0nteyyxBRNIP4rxwyzVkAqFoK5T9IHbtgosuQpev4KvhcYyc2o4lS9y4MY884q6A8GMI6YwkJrori55/3vWefekl6N8/fUeixEQ3/NPo0W501fLlXZy33+6utDCmMElJcZ/1vXvdgMm1a0c7ouzLaUe5L4OelgHaAYtV9fzcCzHnCl2C2LkT7X4BU1c1YGT991i8rhJnnOESw/XX521iCLV0qbskdtEi6NED3njDDcvw5ptuxI+9e11P2ttvh4ED3WW0xpj8KbMEkeXXjKpeHnKwusBLuRSbCUP/+JNp7R9j5Ja3idc2NEhxs4Nef33+GETs7LNh/nzXb+Hhh92cC0ePutj69nWT1XXuXPiHKDCmsDuZ36EJQNPcDsS4mszpn+xm5N/+YFHS6zSoeZi3n3S/wvNDYghWvDjcfTf06eNGVz3tNNe5zSaoMabwyDJBiMiruN7T4DrOnY3rUW1yiaqbH2DkQ0f58aeq1Jf9jL3/F274d5N8lxhC1a3rShLGmMInkhJEcMV+MjBOVb/3KZ4iZ9UqGDwYfvgBTi/xJ2NKP82N066h1Pmdoh2aMaaIiyRBTASSVDUF3CitIlJOVRP9Da1wS0lxVwE9/DBUKJfCf6qPYFDSm5Sa+SWcc060wzPGGCIZPPlroGzQ87LAbH/CKRo2boTzz3czg17c7SgrT+nIkGOvU2r2NEsOxph8I5IEUUaDphn1Hvs8FUjhpOrGlm/Z0s0Z/O47yuRifanx+88wY4abysoYY/KJSBLEIRFpHXgiIm2Aw/6FVDht3+6G0h4yxE0Uv3w5DNr9AjJtKjz3HHToEO0QjTEmnUjaIO4FPhOR3wABagLX+BpVITNhgusbkJgIr7zixn8vtnABDB/uOg4MzXSyPGOMiYpIOsotEpEmQGNv0RpVPeZvWIXDrl0uGUyYAO3awQcfQOPGuBk++vd3c2m+/bb1KDPG5EtZVjGJyJ1AeVVdoaorgAoicof/oRVs06bBX/8Kn38OTzzhZmxr3BjXEHHTTbBtm8sclStHO1RjjAkrkjaIwaq6N/BEVfcAg/0LqWA7cMD1a7j0UqheHRYudJeyHh876dVX4Ysv4OmnXbHCGGPyqUgSRHGRtDoQESmOmwDIhPjuO3eF0ttvwz/+4UYzbdUqaIP4eHdt6+WXw7BhUYvTGGMiEUmCmAFMEJHuItIdGAdMj+TgItJDRNaIyDoRGR5m/YsistS7rRWRvd7ys0VkvoisFJFlIpKvG8WTkuC++6BbNzdGUVycKyCULh200b59cM01ULOmG/LU2h2MMflcJFcxPQgMAW7zni/DXcmUKa+k8TpwIW6Av0UiMkVVVwW2UdVhQdvfBQR+bycCN6jqryJyGrBYRGYGV3XlFzt2uE5vK1a44a2feQYqVAjZSNWNZLd5M8ydC1WrRiVWY4zJjkiuYkoVkR+BM4GrgerA5xEcux2wTlU3AIjIeKA3sCqD7QcAj3qvuTbo9X8TkT+BU4F8lSCOHHGjma5b5xqle/bMYMPRo2HiRFessJ7SxpgCIsMEISKNcF/aA4CdwAQAVe0W4bFrA1uDnicAYbsKi8jpQANgTph17XBtHuvDrBuCK91Qr169CMPKHYFCwfffu4uRMkwOP/3k2ht69nTtD8YYU0Bk1gbxC3A+cJmqdlLVV4EUn+LoD0wMDAgYICK1gA+Bm1Q1NXQnVR2jqrGqGnvqqaf6FFp4Tz4JH30Ejz8OV1+dwUYHDriV1avD++9DsUiafIwxJn/I7BurL7Ad+EZE3vIaqLPTsroNqBv0vI63LJz+uMbv40SkIjAVeFhVF2TjdX336adpU38+/HAGG6nCrbfChg0wbhzkcQIzxpicyjBBqOpkVe0PNAG+wQ258RcRGS0iF0Vw7EVAQxFpICKlcElgSuhGXi/tKsD8oGWlgEnAB6o6MTtvyG8//gg33gjnngtjx2ZyMdLYsS4xjBoF552XpzEaY0xuyLLOQ1UPqeon3tzUdYCfcFc2ZbVfMjAUmAmsBj5V1ZUiMkpEegVt2h8Yr6oatOxq4DxgUNBlsGdH/rb8sXkz9OoFtWrBpEkhl7EGW7bMzcd54YXwz3/maYzGGJNbJP33csEVGxur8fHxWW94kvbvh06dXJKYPx+aNctgw4MHITbW9XtYutQmaTbG5GsislhVY8Oti6QfRJGXnAwDBrjpQadPzyQ5gBudb+1a+PprSw7GmALNEkQE7r/f9XMYPdrVGmXovffckK2PPuq6VRtjTAFm111mYfRoePlluOceuO22TDZctcqVHrp1c5c4GWNMAWcJIhMzZ8Jdd7mRWZ9/PpMNExNdf4cKFeDjj92ATMYYU8BZFVMGVq1y3/nNmrmrVTP9zr/7brfDzJnuEidjjCkErAQRxo4dbv7osmXhq6/glFMy2XjLFje+9333ZdFAYYwxBYuVIEIkJcEVV8D27W5+hyyHeJo7191fd53vsRljTF6yBBEkMADfDz+kzSOdpXnzoGJFaNHC9/iMMSYvWRVTkCeecG3MmQ7AFyouzg3hbQ3TxphCxhKEZ8IEGDECBg7MZAC+ULt2ucbpzp19jc0YY6LBEgSwYEHaAHxvvZWN2UDnzXP3liCMMYVQkU8QW7ZA795Qu3YWA/CFM28elCoFbdv6Fp8xxkRLkU8Q1aq5yd6++uokpmyIi3PJoUwZX2IzxphoKvIJonx5N4RS06bZ3PHQIVi82KqXjDGFVpFPECdt4UI3zKslCGNMIWUJ4mTFxbnW7HPOiXYkxhjjC18ThIj0EJE1IrJORIaHWf9i0Ixxa0Vkb9C6G0XkV+92o59xnpS4ONc5rnLlaEdijDG+8K0ntYgUB14HLgQSgEUiMkVVVwW2UdVhQdvfBbTyHlcFHgViAQUWe/vu8SvebElOdtPKDRoU7UiMMcY3fpYg2gHrVHWDqh4FxgO9M9l+ADDOe3wxMEtVd3tJYRbQw8dYs2fpUtdI3alTtCMxxhjf+JkgagNbg54neMtOICKnAw2AOdnZV0SGiEi8iMTv2LEjV4KOSFycu7cGamNMIZZfGqn7AxNVNSU7O6nqGFWNVdXYU7PdiSEH4uKgQQPXu84YYwopPxPENqBu0PM63rJw+pNWvZTdffOWqutBbdVLxphCzs8EsQhoKCINRKQULglMCd1IRJoAVYD5QYtnAheJSBURqQJc5C2LvrVr3YxCVr1kjCnkfLuKSVWTRWQo7ou9OPCOqq4UkVFAvKoGkkV/YLyqatC+u0XkcVySARilqrv9ijVbrP3BGFNESND3coEWGxur8fHx/r/QoEEwdSr8+Wc2hn01xpj8SUQWq2psuHX5pZG64IiLc+0PlhyMMYWcJYjs+O032LDBqpeMMUWCJYjssAmCjDFFiCWI7IiLg3Ll4Oyzox2JMcb4zhJEdsTFQceOULJktCMxxhjfWYKI1N69sGyZVS8ZY4oMSxCRmj/f9aK2HtTGmCLCEkSk4uKgRAno0CHakRhjTJ6wBBGpuDho3dpNYm2MMUWAJYhIJCW5OaiteskYU4RYgohEfDwcPWoN1MaYIsUSRCQCA/RZCcIYU4RYgojEvHnQtClUrx7tSIwxJs9YgshKSgp8/72VHowxRY4liKysWAH79ln7gzGmyPE1QYhIDxFZIyLrRGR4BttcLSKrRGSliHwStPwZb9lqEXlFJErja9sAfcaYIsq3GeVEpDjwOnAhkAAsEpEpqroqaJuGwD+Bc1V1j4j8xVt+DnAu0NLbdB7QBfjWr3gzFBcHtWvD6afn+UsbY0w0+VmCaAesU9UNqnoUGA/0DtlmMPC6qu4BUNU/veUKlAFKAaWBksAfPsYanqpLEJ072wRBxpgix88EURvYGvQ8wVsWrBHQSES+F5EFItIDQFXnA98A273bTFVdHfoCIjJEROJFJH7Hjh25/w42bnSTBFn1kjGmCIp2I3UJoCHQFRgAvCUilUXkLKApUAeXVM4XkRO+pVV1jKrGqmrsqaeemvvRBdof7AomY0wR5GeC2AbUDXpex1sWLAGYoqrHVHUjsBaXMPoAC1T1oKoeBKYDHX2MNby4OKhcGf761zx/aWOMiTY/E8QioKGINBCRUkB/YErINpNxpQdEpDquymkDsAXoIiIlRKQkroH6hCom38XFwbnnQrFoF7SMMSbv+fbNp6rJwFBgJu7L/VNVXSkio0Skl7fZTGCXiKzCtTk8oKq7gInAemA58DPws6p+6VesYe3YAWvWWPuDMabI8u0yVwBVnQZMC1k2IuixAn/3bsHbpAC3+hlblqz9wRhTxFndSUbi4qB0aYiNjXYkxhgTFZYgMjJvHrRv75KEMcYUQZYgwjl4EJYsseolY0yRZgkinAUL3Ciu1kBtjCnCLEGEExfnLm0955xoR2KMMVFjCSKcefOgZUuoWDHakRhjTNRYggh17JirYrLqJWNMEWcJItSSJZCYaAnCGFPkWYIIZR3kjDEGsARxorg4OPNMqFUr2pEYY0xUWYIIlprqShBWvWSMMZYg0lmzBnbtsgRhjDFYgkgvLs7dW/uDMcZYgkgnLg7+8hdo2DDakRhjTNRZgggWF+eql0SiHYkxxkSdJYiArVth82arXjLGGI+vCUJEeojIGhFZJyLDM9jmahFZJSIrReSToOX1ROR/IrLaW1/fz1iP93+wBmpjjAF8nFFORIoDrwMXAgnAIhGZoqqrgrZpCPwTOFdV94jIX4IO8QHwpKrOEpEKQKpfsQKueqlCBYiJ8fVljDGmoPCzBNEOWKeqG1T1KDAe6B2yzWDgdVXdA6CqfwKISDOghKrO8pYfVNVEH2N1JYhzzoESvs7CaowxBYafCaI2sDXoeYK3LFgjoJGIfC8iC0SkR9DyvSLyXxH5SUSe9Uok6YjIEBGJF5H4HTt2nHyke/bAihXW/mCMMUGi3UhdAmgIdAUGAG+JSGVveWfgfqAtcAYwKHRnVR2jqrGqGnvqqaeefBTffw+q1v5gjDFB/EwQ24C6Qc/reMuCJQBTVPWYqm4E1uISRgKw1KueSgYmA619i3TePChZEtq18+0ljDGmoPEzQSwCGopIAxEpBfQHpoRsMxlXekBEquOqljZ4+1YWkUCx4HxgFX6Ji4M2baBcOd9ewhhjChrfEoT3y38oMBNYDXyqqitFZJSI9PI2mwnsEpFVwDfAA6q6S1VTcNVLX4vIckCAt3wJ9PBhWLTIqpeMMSaEr5fsqOo0YFrIshFBjxX4u3cL3XcW0NLP+ADYtw+uvBIuvtj3lzLGmILErumsWRM++STr7YwxpoiJ9lVMxhhj8ilLEMYYY8KyBGGMMSYsSxDGGGPCsgRhjDEmLEsQxhhjwrIEYYwxJixLEMYYY8IS15m54BORHcDmHByiOrAzl8Lxg8WXMxZfzlh8OZOf4ztdVcMOh11oEkROiUi8qsZGO46MWHw5Y/HljMWXM/k9voxYFZMxxpiwLEEYY4wJyxJEmjHRDiALFl/OWHw5Y/HlTH6PLyxrgzDGGBOWlSCMMcaEZQnCGGNMWEUqQYhIDxFZIyLrRGR4mPWlRWSCt/5HEamfh7HVFZFvRGSViKwUkXvCbNNVRPaJyFLvNiLcsXyOc5OILPdePz7MehGRV7xzuExEWudhbI2Dzs1SEdkvIveGbJOn51BE3hGRP0VkRdCyqiIyS0R+9e6rZLDvjd42v4rIjXkY37Mi8ov395skIpUz2DfTz4KP8Y0UkW1Bf8NLMtg30/93H+ObEBTbJhFZmsG+vp+/HFPVInEDigPrgTOAUsDPQLOQbe4A3vQe9wcm5GF8tYDW3uNTgLVh4usKfBXl87gJqJ7J+kuA6bh5xDsAP0bx7/07rhNQ1M4hcB7QGlgRtOwZYLj3eDjwdJj9qgIbvPsq3uMqeRTfRUAJ7/HT4eKL5LPgY3wjgfsj+Ptn+v/uV3wh658HRkTr/OX0VpRKEO2Adaq6QVWPAuOB3iHb9Abe9x5PBLqLiORFcKq6XVWXeI8PAKuB2nnx2rmsN/CBOguAyiJSKwpxdAfWq2pOetfnmKrOBXaHLA7+nL0PXBFm14uBWaq6W1X3ALOAHnkRn6r+T1WTvacLgDq5/bqRyuD8RSKS//ccyyw+77vjamBcbr9uXilKCaI2sDXoeQInfgEf38b7B9kHVMuT6IJ4VVutgB/DrO4oIj+LyHQRaZ6ngTkK/E9EFovIkDDrIznPeaE/Gf9jRvsc1lDV7d7j34EaYbbJL+fxZlyJMJysPgt+GupVgb2TQRVdfjh/nYE/VPXXDNZH8/xFpCgliAJBRCoAnwP3qur+kNVLcFUmMcCrwOS8jg/opKqtgZ7AnSJyXhRiyJSIlAJ6AZ+FWZ0fzuFx6uoa8uW15iLyMJAMfJzBJtH6LIwGzgTOBrbjqnHyowFkXnrI9/9LRSlBbAPqBj2v4y0Lu42IlAAqAbvyJDr3miVxyeFjVf1v6HpV3a+qB73H04CSIlI9r+LzXnebd/8nMAlXlA8WyXn2W09giar+EboiP5xD4I9AtZt3/2eYbaJ6HkVkEHAZcJ2XxE4QwWfBF6r6h6qmqGoq8FYGrxvt81cC6AtMyGibaJ2/7ChKCWIR0FBEGni/MPsDU0K2mQIErha5EpiT0T9HbvPqK98GVqvqCxlsUzPQJiIi7XB/v7xMYOVF5JTAY1xj5oqQzaYAN3hXM3UA9gVVp+SVDH+5RfsceoI/ZzcCX4TZZiZwkYhU8apQLvKW+U5EegD/AHqpamIG20TyWfArvuA2rT4ZvG4k/+9+ugD4RVUTwq2M5vnLlmi3kuflDXeFzVrc1Q0Pe8tG4f4RAMrgqiXWAQuBM/Iwtk64qoZlwFLvdglwG3Cbt81QYCXuiowFwDl5fP7O8F77Zy+OwDkMjlGA171zvByIzeMYy+O+8CsFLYvaOcQlqu3AMVw9+C24dq2vgV+B2UBVb9tYYGzQvjd7n8V1wE15GN86XP194HMYuLLvNGBaZp+FPIrvQ++ztQz3pV8rND7v+Qn/73kRn7f8vcBnLmjbPD9/Ob3ZUBvGGGPCKkpVTMYYY7LBEoQxxpiwLEEYY4wJyxKEMcaYsCxBGGOMCcsST/kAcQAAAdVJREFUhDFZEJGUkFFic21kUBGpHzwSqDH5SYloB2BMAXBYVc+OdhDG5DUrQRhzkrzx/J/xxvRfKCJnecvri8gcbzC5r0Wknre8hje/ws/e7RzvUMVF5C1x84D8T0TKetvfLW5+kGUiMj5Kb9MUYZYgjMla2ZAqpmuC1u1T1RbAa8BL3rJXgfdVtSVuoLtXvOWvAN+pGyiwNa4HLUBD4HVVbQ7sBfp5y4cDrbzj3ObXmzMmI9aT2pgsiMhBVa0QZvkm4HxV3eANtPi7qlYTkZ244R+Oecu3q2p1EdkB1FHVI0HHqI+b96Gh9/xBoKSqPiEiM4CDuBFnJ6s3yKAxecVKEMbkjGbwODuOBD1OIa1t8FLcuFatgUXeCKHG5BlLEMbkzDVB9/O9xz/gRg8FuA6I8x5/DdwOICLFRaRSRgcVkWJAXVX9BngQN/T8CaUYY/xkv0iMyVrZkInnZ6hq4FLXKiKyDFcKGOAtuwt4V0QeAHYAN3nL7wHGiMgtuJLC7biRQMMpDnzkJREBXlHVvbn2joyJgLVBGHOSvDaIWFXdGe1YjPGDVTEZY4wJy0oQxhhjwrIShDHGmLAsQRhjjAnLEoQxxpiwLEEYY4wJyxKEMcaYsP4fyg4pFNtGn3UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "acc = history.history['accuracy']\n",
    "val_acc = history.history['val_accuracy']\n",
    "\n",
    "#------------------------------------------------\n",
    "# Plot training and validation accuracy per epoch\n",
    "#------------------------------------------------\n",
    "plt.plot(epochs, acc, 'r')\n",
    "plt.plot(epochs, val_acc, 'b')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.legend([\"Accuracy\", \"Validation Accuracy\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more rigorous way of setting the passing threshold of this assignment is to use the slope of your `val_loss` curve.\n",
    "\n",
    "**To pass this assignment the slope of your `val_loss` curve should be 0.0005 at maximum.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slope of your validation loss curve is -0.00041\n"
     ]
    }
   ],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "# Test the slope of your val_loss curve\n",
    "slope, *_ = linregress(epochs, val_loss)\n",
    "print(f\"The slope of your validation loss curve is {slope:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**If your model generated a validation loss curve that meets the criteria above, run the following cell and then submit your assignment for grading. Otherwise, try with a different architecture.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "deletable": false,
    "editable": false,
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# grader-required-cell\n",
    "\n",
    "with open('history.pkl', 'wb') as f:\n",
    "    pickle.dump(history.history, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations on finishing this week's assignment!**\n",
    "\n",
    "You have successfully implemented a neural network capable of classifying sentiment in text data while doing a fairly good job of not overfitting! Nice job!\n",
    "\n",
    "**Keep it up!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "  <summary><font size=\"2\" color=\"darkgreen\"><b>Please click here if you want to experiment with any of the non-graded code.</b></font></summary>\n",
    "    <p><i><b>Important Note: Please only do this when you've already passed the assignment to avoid problems with the autograder.</b></i>\n",
    "    <ol>\n",
    "        <li> On the notebook’s menu, click “View” > “Cell Toolbar” > “Edit Metadata”</li>\n",
    "        <li> Hit the “Edit Metadata” button next to the code cell which you want to lock/unlock</li>\n",
    "        <li> Set the attribute value for “editable” to:\n",
    "            <ul>\n",
    "                <li> “true” if you want to unlock it </li>\n",
    "                <li> “false” if you want to lock it </li>\n",
    "            </ul>\n",
    "        </li>\n",
    "        <li> On the notebook’s menu, click “View” > “Cell Toolbar” > “None” </li>\n",
    "    </ol>\n",
    "    <p> Here's a short demo of how to do the steps above: \n",
    "        <br>\n",
    "        <img src=\"https://drive.google.com/uc?export=view&id=14Xy_Mb17CZVgzVAgq7NCjMVBvSae3xO1\" align=\"center\">\n",
    "</details>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}