lectures/cap09_4_sentiment_analysis_dnn.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cap09_4_sentiment_analysis_dnn.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOmNPXdqoTqULgrCxGuNgvp",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/viniciusrpb/cic0269_natural_language_processing/blob/main/lectures/cap09_4_sentiment_analysis_dnn.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Capítulo 9 - Redes Neurais Artificiais\n",
        "\n",
        "## 9.4. Estudo de caso: Análise de Sentimentos utilizando uma Rede Neural Profunda\n",
        "\n",
        "### Tarefa: classificação de polaridade em tweets\n",
        "\n",
        "Classificar um *tweet* em alguma das três polaridades: positiva (*positive*), negativa (*negative*) e positiva (*neutral*).\n",
        "\n",
        "### O *corpus*\n",
        "\n",
        "Para isso, vamos precisar utilizar as bibliotecas keras, tensorflow, numpy, sklearn, matplotlib.\n",
        "\n",
        "Fonte: [SemEval-2013 Task 2: Sentiment Analysis in Twitter](https://aclanthology.org/S13-2052/)\n",
        "\n",
        "O *corpus* de tweets empregado nesse estudo de caso podem ser acessados na pasta \"corpus_tweets\" deste repositório. Segue a especificação de cada arquivo:\n",
        "\n",
        "*   ```twitter-2013train-A.txt```: conjunto de tweets para treinamento\n",
        "*   ```twitter-2013dev-A.txt```: conjunto de tweets para validação\n",
        "*   ```twitter-2013test-A.txt```: conjunto de tweets para testes"
      ],
      "metadata": {
        "id": "IipByK9A-0GD"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AuV9oScUNg6G"
      },
      "source": [
        "!pip install -U keras\n",
        "!pip install -U tensorflow"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dense,Activation\n",
        "from tensorflow.keras.optimizers import SGD\n",
        "from sklearn.metrics import classification_report\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "metadata": {
        "id": "rX3Wa1BUB0bx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Importando as classes necessárias a partir das bibliotecas"
      ],
      "metadata": {
        "id": "WWVshYwOMSFz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.model_selection import train_test_split"
      ],
      "metadata": {
        "id": "GETnhj__U_Iy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Carregamento do corpus para Pandas DataFrames\n",
        "\n",
        "\n",
        "\n",
        "**PS.:** Mais informações sobre a divisão de um conjunto de dados em treinamento, validação e teste, você encontra nesse [link](https://machinelearningmastery.com/difference-test-validation-datasets/).\n"
      ],
      "metadata": {
        "id": "fp0s4KlqiQUq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "path_train = 'twitter-2013train-A.txt'\n",
        "path_valid = 'twitter-2013dev-A.txt'\n",
        "path_test = 'twitter-2013test-A.txt'"
      ],
      "metadata": {
        "id": "MY8M1KY0TpiW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_train = pd.read_csv(path_train, names=[\"id\",\"polarity\",\"text\"],encoding='utf8', sep='\\t')\n",
        "df_valid = pd.read_csv(path_valid, names=[\"id\",\"polarity\",\"text\"],encoding='utf8', sep='\\t')\n",
        "df_test = pd.read_csv(path_test, names=[\"id\",\"polarity\",\"text\"],encoding='utf8', sep='\\t')"
      ],
      "metadata": {
        "id": "2MJ0FVjpkEHG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_train"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "uq2wnaK6fRVh",
        "outputId": "aa53618b-636e-4292-c29f-8396d6c379cc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                      id  polarity  \\\n",
              "0     264183816548130816  positive   \n",
              "1     263405084770172928  negative   \n",
              "2     262163168678248449  negative   \n",
              "3     264249301910310912  negative   \n",
              "4     262682041215234048   neutral   \n",
              "...                  ...       ...   \n",
              "9679  103158179306807296  positive   \n",
              "9680  103157324096618497  positive   \n",
              "9681  100259220338905089   neutral   \n",
              "9682  104230318525001729  positive   \n",
              "9683  100461938533863424  negative   \n",
              "\n",
              "                                                   text  \n",
              "0     Gas by my house hit $3.39!!!! I\\u2019m going t...  \n",
              "1     Theo Walcott is still shit\\u002c watch Rafa an...  \n",
              "2     its not that I\\u2019m a GSP fan\\u002c i just h...  \n",
              "3     Iranian general says Israel\\u2019s Iron Dome c...  \n",
              "4     Tehran\\u002c Mon Amour: Obama Tried to Establi...  \n",
              "...                                                 ...  \n",
              "9679  RT @MNFootNg It's monday and Monday Night Foot...  \n",
              "9680  All I know is the road for that Lomardi start ...  \n",
              "9681  All Blue and White fam, we r meeting at Golden...  \n",
              "9682  @DariusButler28   Have a great game agaist Tam...  \n",
              "9683  I'm pisseeedddd that I missed Kid Cudi's show ...  \n",
              "\n",
              "[9684 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6cd6afbd-65e8-4eab-bfd1-63abc91b4b88\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>polarity</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>264183816548130816</td>\n",
              "      <td>positive</td>\n",
              "      <td>Gas by my house hit $3.39!!!! I\\u2019m going t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>263405084770172928</td>\n",
              "      <td>negative</td>\n",
              "      <td>Theo Walcott is still shit\\u002c watch Rafa an...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>262163168678248449</td>\n",
              "      <td>negative</td>\n",
              "      <td>its not that I\\u2019m a GSP fan\\u002c i just h...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>264249301910310912</td>\n",
              "      <td>negative</td>\n",
              "      <td>Iranian general says Israel\\u2019s Iron Dome c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>262682041215234048</td>\n",
              "      <td>neutral</td>\n",
              "      <td>Tehran\\u002c Mon Amour: Obama Tried to Establi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9679</th>\n",
              "      <td>103158179306807296</td>\n",
              "      <td>positive</td>\n",
              "      <td>RT @MNFootNg It's monday and Monday Night Foot...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9680</th>\n",
              "      <td>103157324096618497</td>\n",
              "      <td>positive</td>\n",
              "      <td>All I know is the road for that Lomardi start ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9681</th>\n",
              "      <td>100259220338905089</td>\n",
              "      <td>neutral</td>\n",
              "      <td>All Blue and White fam, we r meeting at Golden...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9682</th>\n",
              "      <td>104230318525001729</td>\n",
              "      <td>positive</td>\n",
              "      <td>@DariusButler28   Have a great game agaist Tam...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9683</th>\n",
              "      <td>100461938533863424</td>\n",
              "      <td>negative</td>\n",
              "      <td>I'm pisseeedddd that I missed Kid Cudi's show ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>9684 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6cd6afbd-65e8-4eab-bfd1-63abc91b4b88')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6cd6afbd-65e8-4eab-bfd1-63abc91b4b88 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6cd6afbd-65e8-4eab-bfd1-63abc91b4b88');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Informações sobre cada parte do *corpus* de tweets:"
      ],
      "metadata": {
        "id": "oXUO88MRwdcY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"O conjunto de treinamento possui {df_train.shape} tweets\")\n",
        "print(f\"O conjunto de validação possui {df_valid.shape} tweets\")\n",
        "print(f\"O conjunto de testes possui {df_test.shape} tweets\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Le_z-K3-wivo",
        "outputId": "05808757-bcc0-48b9-825b-2a2e02050906"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "O conjunto de treinamento possui (9684, 3) tweets\n",
            "O conjunto de validação possui (1654, 3) tweets\n",
            "O conjunto de testes possui (3547, 3) tweets\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Pré-processamento dos Tweets\n",
        "\n",
        "Remove o atributo **id**, pois não será necessário o uso desse atributo no processo de classificação.\n",
        "\n"
      ],
      "metadata": {
        "id": "cycRNw7tfZLZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_train.drop(labels=['id'],axis=1)\n",
        "df_valid.drop(labels=['id'],axis=1)\n",
        "df_test.drop(labels=['id'],axis=1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "W4ZbUgNheyGK",
        "outputId": "2b85b2e8-feff-4973-bd74-c50a774917cc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      polarity                                               text\n",
              "0     positive  @jjuueellzz down in the Atlantic city, ventnor...\n",
              "1     positive  Musical awareness: Great Big Beautiful Tomorro...\n",
              "2      neutral  On Radio786 100.4fm 7:10 Fri Oct 19 Labour ana...\n",
              "3     negative  Kapan sih lo ngebuktiin,jan ngomong doang Susa...\n",
              "4      neutral  Excuse the connectivity of this live stream, f...\n",
              "...        ...                                                ...\n",
              "3542  negative  Khaleda Zia's present India visit may have a b...\n",
              "3543   neutral  FYI, golf fans: @jameslepp will join Moj on We...\n",
              "3544  negative  @__Aniko you think mr.Calle let practice with ...\n",
              "3545  positive  Don't hide under your desk! It's just a salsa ...\n",
              "3546   neutral  Saturday flashmob on the music of The Runaways...\n",
              "\n",
              "[3547 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-743b6459-64bf-4f5e-9312-bb8e20b2e571\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>polarity</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>positive</td>\n",
              "      <td>@jjuueellzz down in the Atlantic city, ventnor...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>positive</td>\n",
              "      <td>Musical awareness: Great Big Beautiful Tomorro...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>neutral</td>\n",
              "      <td>On Radio786 100.4fm 7:10 Fri Oct 19 Labour ana...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>negative</td>\n",
              "      <td>Kapan sih lo ngebuktiin,jan ngomong doang Susa...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>neutral</td>\n",
              "      <td>Excuse the connectivity of this live stream, f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3542</th>\n",
              "      <td>negative</td>\n",
              "      <td>Khaleda Zia's present India visit may have a b...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3543</th>\n",
              "      <td>neutral</td>\n",
              "      <td>FYI, golf fans: @jameslepp will join Moj on We...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3544</th>\n",
              "      <td>negative</td>\n",
              "      <td>@__Aniko you think mr.Calle let practice with ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3545</th>\n",
              "      <td>positive</td>\n",
              "      <td>Don't hide under your desk! It's just a salsa ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3546</th>\n",
              "      <td>neutral</td>\n",
              "      <td>Saturday flashmob on the music of The Runaways...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3547 rows × 2 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-743b6459-64bf-4f5e-9312-bb8e20b2e571')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-743b6459-64bf-4f5e-9312-bb8e20b2e571 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-743b6459-64bf-4f5e-9312-bb8e20b2e571');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Além disso, por questões do uso dos métodos da classe Sequential no Keras-Tensorflow, convertemos o atributo de classe para o tipo categórico e para cada instância, mapeamos seus valores para números inteiros. Lembre-se que os **modelos de aprendizado de máquina processam apenas números inteiros ou reais**."
      ],
      "metadata": {
        "id": "Plz1Pm15ztNO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_train['polarity'] = pd.Categorical(df_train['polarity'])\n",
        "df_valid['polarity'] = pd.Categorical(df_valid['polarity'])\n",
        "df_test['polarity'] = pd.Categorical(df_test['polarity'])\n",
        "\n",
        "y_train = df_train['polarity'].cat.codes\n",
        "y_valid = df_valid['polarity'].cat.codes\n",
        "y_test = df_test['polarity'].cat.codes"
      ],
      "metadata": {
        "id": "4MbE7PZqztrV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vamos ver como e repare que o mapeamento do atributo categórico \"polarity\" para valores categóricos inteiros ficou como:\n",
        "\n",
        "*   polaridade \"negativa\" *(negative)*: valor $0$\n",
        "*   polaridade \"neutra\" *(neutral)*: valor $1$\n",
        "*   polaridade \"positiva\" *(positive)*: valor $2$"
      ],
      "metadata": {
        "id": "soW04sEVto_b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for i in range(0,5):\n",
        "    v = df_train['polarity'][i]\n",
        "    print(f'Valor original da classe {v} Transformação inteira: {y_train[i]}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4LxkFq1ttpRZ",
        "outputId": "04c0a741-1ae3-4973-c932-8de641786d2c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Valor original da classe positive Transformação inteira: 2\n",
            "Valor original da classe negative Transformação inteira: 0\n",
            "Valor original da classe negative Transformação inteira: 0\n",
            "Valor original da classe negative Transformação inteira: 0\n",
            "Valor original da classe neutral Transformação inteira: 1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A seguir, transformamos os valores da classe \"polaridade\", nossa classe de interesse, para one-hot-encoding. Nessa representação, como temos três classes, utilizaremos três bits, em que para cada instância, apenas um deles estará ativado (com valor $1$) com base na presença do valor classe.\n",
        "\n",
        "Veja como fica a representação one-hot encoding para cada classe:\n",
        "\n",
        "*   polaridade \"negativa\" *(negative)*: valor $0$, one hot encoding: $[1,0,0]$\n",
        "*   polaridade \"neutra\" *(neutral)*: valor $1$, one hot encoding: $[0,1,0]$\n",
        "*   polaridade \"positiva\" *(positive)*: valor $2$, one hot encoding: $[0,0,1]$\n",
        "\n",
        "Utilizaremos a função *to_categorical* do Keras para esse propósito (se bem que poderíamos facilmente implementar uma por nossa conta):"
      ],
      "metadata": {
        "id": "qRN-0SW50IG0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.utils.np_utils import to_categorical\n",
        "\n",
        "num_classes = 3\n",
        "\n",
        "y_train_enc = to_categorical(y_train,3)\n",
        "y_valid_enc = to_categorical(y_valid,3)\n",
        "y_test_enc = to_categorical(y_test,3)"
      ],
      "metadata": {
        "id": "bZqJdMXW0INV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Apenas verificando como ficou o pré-processamento do atributo de classe:"
      ],
      "metadata": {
        "id": "ONGb3W5ky3Sv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for i in range(0,5):\n",
        "    v = df_train['polarity'][i]\n",
        "    print(f\"Tweet {i}: valor original da classe '{v}'; mapeamento para inteiro: {y_train[i]}; representação one-hot-encoding [{int(y_train_enc[i][0])},{int(y_train_enc[i][1])},{int(y_train_enc[i][2])}]\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4qWlGYzJxb-s",
        "outputId": "b2f98398-37fa-49f2-c0f8-84ff3e41bee1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tweet 0: valor original da classe 'positive'; mapeamento para inteiro: 2; representação one-hot-encoding [0,0,1]\n",
            "Tweet 1: valor original da classe 'negative'; mapeamento para inteiro: 0; representação one-hot-encoding [1,0,0]\n",
            "Tweet 2: valor original da classe 'negative'; mapeamento para inteiro: 0; representação one-hot-encoding [1,0,0]\n",
            "Tweet 3: valor original da classe 'negative'; mapeamento para inteiro: 0; representação one-hot-encoding [1,0,0]\n",
            "Tweet 4: valor original da classe 'neutral'; mapeamento para inteiro: 1; representação one-hot-encoding [0,1,0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Extração de características dos tweets\n",
        "\n",
        "Método Term Frequeny-Inverse Document Frequency (TF-IDF) implementado pela classe *TfidfVectorizer*.\n",
        "\n",
        "Aqui deve-se tomar o cuidado de calcular o vocabulário apenas no conjunto de treinamento:"
      ],
      "metadata": {
        "id": "Q_mzac_gfstB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tfidfvectorizer = TfidfVectorizer(norm=None, stop_words='english', max_df=0.95, min_df=2)\n",
        "\n",
        "tfidfvectorizer.fit(df_train['text'])\n",
        "\n",
        "tfidf_train = tfidfvectorizer.transform(df_train['text']).toarray()\n",
        "tfidf_valid = tfidfvectorizer.transform(df_valid['text']).toarray()\n",
        "tfidf_test = tfidfvectorizer.transform(df_test['text']).toarray()\n",
        "\n",
        "N = tfidf_train.shape[0]\n",
        "vocab_size = tfidf_train.shape[1]\n",
        "\n",
        "print(f'O conjunto de treinamento possui {N} instâncias e um vocabulário de {vocab_size} palavras (termos).')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 236
        },
        "id": "7FbtfK40T4Sc",
        "outputId": "611e65b3-cbf4-430f-aa3b-d4ddf3d49e64"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-b5a52cb3607a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0msoma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdf_train\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Tweet text'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m   \u001b[0msoma\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mmedia\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msoma\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf_train\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Tweet text'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'df_train' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Colocamos as instâncias (juntamente com seus atributos TF-IDF) nas respectivas matrizes $X$ (para cada tipo de dados), em que cada linha de $X$ está associada a um *tweet*. Repare que as classes (labels, rótulos etc) já foram separadas anteriormente e estão armazenadas nas variáveis ```y```. Por exemplo, a instância de treinamento $\\mathbf{x_i}$, denotada por ```X_train[i]``` é da classe ```y_train[i]``` $\\in \\{0,1,2\\}$."
      ],
      "metadata": {
        "id": "GGw9jEA2gCTn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_train = tfidf_train\n",
        "X_valid = tfidf_valid\n",
        "X_test = tfidf_test"
      ],
      "metadata": {
        "id": "kXNL8-WsT6wl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Modelo de Classificação: Rede Neural Profunda\n",
        "\n",
        "Vamos criar uma rede neural profunda do tipo feed forward multilayer perceptron. O especialista em questão (eu, você etc) decidiu criar uma rede com duas camadas intermediárias: a primeira contendo $20$ neurônios e a segunda contendo $10$ neurônios.\n",
        "\n"
      ],
      "metadata": {
        "id": "TSV_ATefiiUB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "number_of_neurons_layer1 = 20\n",
        "number_of_neurons_layer2 = 10"
      ],
      "metadata": {
        "id": "oH1D3HT-jSN4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Em seguida, criamos o modelo baseado na rede neural. Observe que agora temos um problema de classificação multi-classe e, por isso, a camada de saída contém três neurônios, em que cada um está associado com uma das possíveis classe (\"positive\", \"negative\", \"neutral\"). Isso significa que o $k$-ésimo neurônio da camada de saída produz na saída a probabilidade de classificação de uma instância para a classe $k$."
      ],
      "metadata": {
        "id": "897HPB35zsCd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = Sequential()\n",
        "model.add(Dense(number_of_neurons_layer1,input_dim=vocab_size, activation=\"tanh\"))\n",
        "model.add(Dense(number_of_neurons_layer2,activation=\"tanh\"))\n",
        "model.add(Dense(3,activation=\"sigmoid\"))"
      ],
      "metadata": {
        "id": "LvWB8hO2jGCf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "O ajuste do modelo para o treinamento consiste em utilizarmos um otimizador da classe SGD (*Stochastic Gradient Descent*), em que colocamos aleatoriamente um valor da taxa de aprendizado ```learning_rate=0.001```.\n",
        "\n",
        "Outra alteração se trata da função *loss* (que calcula o erro da modelo). Como temos um problema de classificação multiclasse, temos que empregar a função ```categorical_crossentropy```."
      ],
      "metadata": {
        "id": "peU5AQYB0Rc6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sgd = SGD(learning_rate=0.001)\n",
        "model.compile(loss=\"categorical_crossentropy\",optimizer=sgd,metrics=['accuracy'])\n",
        "history_fine = model.fit(X_train,y_train_enc,validation_data=(X_valid,y_valid_enc),epochs=30)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T95n0SaLzpj1",
        "outputId": "acb272a8-2f08-479f-d4ac-124dba388567"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/30\n",
            "303/303 [==============================] - 3s 9ms/step - loss: 1.0758 - accuracy: 0.4208 - val_loss: 1.0637 - val_accuracy: 0.4444\n",
            "Epoch 2/30\n",
            "303/303 [==============================] - 2s 8ms/step - loss: 1.0080 - accuracy: 0.4975 - val_loss: 1.0367 - val_accuracy: 0.4837\n",
            "Epoch 3/30\n",
            "303/303 [==============================] - 2s 7ms/step - loss: 0.9687 - accuracy: 0.5382 - val_loss: 1.0190 - val_accuracy: 0.4940\n",
            "Epoch 4/30\n",
            "303/303 [==============================] - 2s 8ms/step - loss: 0.9399 - accuracy: 0.5625 - val_loss: 1.0049 - val_accuracy: 0.5097\n",
            "Epoch 5/30\n",
            "303/303 [==============================] - 3s 9ms/step - loss: 0.9158 - accuracy: 0.5829 - val_loss: 0.9921 - val_accuracy: 0.5242\n",
            "Epoch 6/30\n",
            "303/303 [==============================] - 2s 7ms/step - loss: 0.8943 - accuracy: 0.6002 - val_loss: 0.9802 - val_accuracy: 0.5357\n",
            "Epoch 7/30\n",
            "303/303 [==============================] - 2s 8ms/step - loss: 0.8742 - accuracy: 0.6140 - val_loss: 0.9690 - val_accuracy: 0.5435\n",
            "Epoch 8/30\n",
            "303/303 [==============================] - 2s 8ms/step - loss: 0.8551 - accuracy: 0.6305 - val_loss: 0.9583 - val_accuracy: 0.5514\n",
            "Epoch 9/30\n",
            "303/303 [==============================] - 2s 7ms/step - loss: 0.8366 - accuracy: 0.6421 - val_loss: 0.9481 - val_accuracy: 0.5556\n",
            "Epoch 10/30\n",
            "303/303 [==============================] - 2s 7ms/step - loss: 0.8186 - accuracy: 0.6541 - val_loss: 0.9386 - val_accuracy: 0.5611\n",
            "Epoch 11/30\n",
            "303/303 [==============================] - 2s 8ms/step - loss: 0.8010 - accuracy: 0.6649 - val_loss: 0.9296 - val_accuracy: 0.5719\n",
            "Epoch 12/30\n",
            "303/303 [==============================] - 2s 7ms/step - loss: 0.7839 - accuracy: 0.6753 - val_loss: 0.9211 - val_accuracy: 0.5840\n",
            "Epoch 13/30\n",
            "303/303 [==============================] - 2s 7ms/step - loss: 0.7671 - accuracy: 0.6848 - val_loss: 0.9131 - val_accuracy: 0.5883\n",
            "Epoch 14/30\n",
            "303/303 [==============================] - 2s 7ms/step - loss: 0.7506 - accuracy: 0.6942 - val_loss: 0.9058 - val_accuracy: 0.5919\n",
            "Epoch 15/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.7344 - accuracy: 0.7024 - val_loss: 0.8993 - val_accuracy: 0.5973\n",
            "Epoch 16/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.7186 - accuracy: 0.7096 - val_loss: 0.8934 - val_accuracy: 0.5998\n",
            "Epoch 17/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.7032 - accuracy: 0.7204 - val_loss: 0.8881 - val_accuracy: 0.6058\n",
            "Epoch 18/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.6880 - accuracy: 0.7280 - val_loss: 0.8832 - val_accuracy: 0.6125\n",
            "Epoch 19/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.6732 - accuracy: 0.7363 - val_loss: 0.8789 - val_accuracy: 0.6149\n",
            "Epoch 20/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.6587 - accuracy: 0.7451 - val_loss: 0.8755 - val_accuracy: 0.6149\n",
            "Epoch 21/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.6445 - accuracy: 0.7505 - val_loss: 0.8723 - val_accuracy: 0.6149\n",
            "Epoch 22/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.6306 - accuracy: 0.7585 - val_loss: 0.8700 - val_accuracy: 0.6215\n",
            "Epoch 23/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.6170 - accuracy: 0.7656 - val_loss: 0.8676 - val_accuracy: 0.6233\n",
            "Epoch 24/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.6037 - accuracy: 0.7720 - val_loss: 0.8655 - val_accuracy: 0.6233\n",
            "Epoch 25/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.5907 - accuracy: 0.7778 - val_loss: 0.8640 - val_accuracy: 0.6239\n",
            "Epoch 26/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.5779 - accuracy: 0.7850 - val_loss: 0.8631 - val_accuracy: 0.6264\n",
            "Epoch 27/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.5654 - accuracy: 0.7897 - val_loss: 0.8625 - val_accuracy: 0.6258\n",
            "Epoch 28/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.5531 - accuracy: 0.7947 - val_loss: 0.8627 - val_accuracy: 0.6270\n",
            "Epoch 29/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.5412 - accuracy: 0.8003 - val_loss: 0.8630 - val_accuracy: 0.6252\n",
            "Epoch 30/30\n",
            "303/303 [==============================] - 1s 4ms/step - loss: 0.5294 - accuracy: 0.8073 - val_loss: 0.8637 - val_accuracy: 0.6258\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vamos analisar o comportamento da função *loss* no decorrer das épocas:"
      ],
      "metadata": {
        "id": "SVYG8Cut07Zi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "loss = history_fine.history['loss']\n",
        "val_loss = history_fine.history['val_loss']\n",
        "\n",
        "\n",
        "plt.plot(history_fine.history['loss'])\n",
        "plt.plot(history_fine.history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['training set','validation set'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "sUDT-dn41VLH",
        "outputId": "e8968920-180f-445d-9b98-25388ea27a26"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEWCAYAAACEz/viAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xUZdr/8c+VTnoh1FR6CSUQujQRROyoIIorNmxrWV121Z+7lmf3WZ9dVFZXQRC74gIKdkEUBKSD9F4SUgiQBFIgfe7fH2fAUBISSDKZmev9es0rM6fNdTIw35z7Puc+YoxBKaWUe/NwdAFKKaUcT8NAKaWUhoFSSikNA6WUUmgYKKWUQsNAKaUUGgZKVZuIvCcif6vmsskicsWlbkep+qJhoJRSSsNAKaWUhoFyMfbmmUkisllETojITBFpKiLfiUi+iCwSkbAKy18nIttE5LiILBGRjhXmJYrIBvt6/wX8znqva0Rko33dFSLS9SJrvk9E9opIjoh8KSIt7NNFRF4VkSMikiciW0QkwT5vlIhst9eWLiJ/vKhfmFJ2GgbKFd0EDAfaAdcC3wHPAJFY/+YfBRCRdsAs4HH7vG+Br0TER0R8gPnAh0A4MMe+XezrJgLvAPcDEcBbwJci4luTQkXkcuAfwBigOZACfGqfPQIYZN+PEPsy2fZ5M4H7jTFBQALwU03eV6mzaRgoV/S6MeawMSYdWAasNsb8aowpAuYBifblxgLfGGN+MMaUApOBRkB/oC/gDUwxxpQaY+YCayu8x0TgLWPMamNMuTHmfaDYvl5N3A68Y4zZYIwpBp4G+olIHFAKBAEdADHG7DDGHLKvVwp0EpFgY8wxY8yGGr6vUmfQMFCu6HCF54XneR1of94C6y9xAIwxNiAVaGmfl27OHMkxpcLzWOBJexPRcRE5DkTb16uJs2sowPrrv6Ux5ifgP8AbwBERmS4iwfZFbwJGASki8rOI9Kvh+yp1Bg0D5c4ysL7UAauNHusLPR04BLS0TzslpsLzVODvxpjQCg9/Y8ysS6whAKvZKR3AGPOaMaYn0AmruWiSffpaY8z1QBOs5qzZNXxfpc6gYaDc2WzgahEZJiLewJNYTT0rgJVAGfCoiHiLyGigd4V1ZwAPiEgfe0dvgIhcLSJBNaxhFnCXiHS39zf8L1azVrKI9LJv3xs4ARQBNnufxu0iEmJv3soDbJfwe1BKw0C5L2PMLmA88DqQhdXZfK0xpsQYUwKMBiYAOVj9C59XWHcdcB9WM84xYK992ZrWsAj4C/AZ1tFIa+BW++xgrNA5htWUlA38yz7vDiBZRPKAB7D6HpS6aKI3t1FKKaVHBkoppTQMlFJK1WEYiMg79isnt1Yyv4OIrBSRYr16UimlHKsujwzeA0ZWMT8H60rQyXVYg1JKqWrwqqsNG2OW2q+irGz+EawLaa6uyXYbN25s4uIq3axSSqnzWL9+fZYxJrKy+XUWBrVJRCZiXf5PTEwM69atc3BFSinlXEQkpar5TtGBbIyZboxJMsYkRUZWGmxKKaUuklOEgVJKqbqlYaCUUqru+gxEZBYwBGgsImnAc1hDAmOMmSYizYB1WJfc20TkcaCTMSavpu9VWlpKWloaRUVFtVa/qh1+fn5ERUXh7e3t6FKUUlWoy7OJxl1gfiYQVRvvlZaWRlBQEHFxcZw5yKRyJGMM2dnZpKWlER8f7+hylFJVcIlmoqKiIiIiIjQIGhgRISIiQo/YlHICLhEGgAZBA6Wfi1LOwWXC4EKKSsvJOF6ITUdpVUqpc7hNGJSU2cgqKCavsLTWt338+HHefPPNi1p31KhRHD9+vMpl/vrXv7Jo0aKL2v6lmD9/Ptu3b6/391VK1T+3CYMgPy98vTzIKiip9W1XFQZlZWVVrvvtt98SGhpa5TIvvvgiV1xxxUXXd7E0DJRyH24TBiJCRIAvJ0vKOFlS9Rd0TT311FPs27eP7t27M2nSJJYsWcLAgQO57rrr6NSpEwA33HADPXv2pHPnzkyfPv30unFxcWRlZZGcnEzHjh2577776Ny5MyNGjKCwsBCACRMmMHfu3NPLP/fcc/To0YMuXbqwc+dOAI4ePcrw4cPp3Lkz9957L7GxsWRlZZ1RZ3l5ORMmTCAhIYEuXbrw6quvArBv3z5GjhxJz549GThwIDt37mTFihV8+eWXTJo0ie7du7Nv375a/Z0ppRoWpxibqCZe+Gob2zMquVTBlHOi1ODl4YGvV/VzsFOLYJ67tnOl81966SW2bt3Kxo0bAViyZAkbNmxg69atp0+pfOeddwgPD6ewsJBevXpx0003ERERccZ29uzZw6xZs5gxYwZjxozhs88+Y/z48ee8X+PGjdmwYQNvvvkmkydP5u233+aFF17g8ssv5+mnn+b7779n5syZ56y3ceNG0tPT2brVGlX8VPPUxIkTmTZtGm3btmX16tU89NBD/PTTT1x33XVcc8013HzzzdX+XSmlnJPLhUGlbKVQVoyv+FBsAx88qMvzXHr37n3GufWvvfYa8+bNAyA1NZU9e/acEwbx8fF0794dgJ49e5KcnHzebY8ePfr0Mp9/bt2Wd/ny5ae3P3LkSMLCws5Zr1WrVuzfv59HHnmEq6++mhEjRlBQUMCKFSu45ZZbTi9XXFx8kXutlHJWLhcGlf4Fb7PBsQNQnEemCUOCmtE02K/O6ggICDj9fMmSJSxatIiVK1fi7+/PkCFDznvuva+v7+nnnp6ep5uJKlvO09Pzgn0SFYWFhbFp0yYWLFjAtGnTmD17NlOmTCE0NPT0UY1Syj25TZ8BHh4QHg+Nwmgmx/AqyKi100yDgoLIz8+vdH5ubi5hYWH4+/uzc+dOVq1aVSvvW9GAAQOYPXs2AAsXLuTYsWPnLJOVlYXNZuOmm27ib3/7Gxs2bCA4OJj4+HjmzJkDWFcNb9q0qVr7pZRyHe4TBgDiAaGxlPiGE0EupdkpUAuBEBERwYABA0hISGDSpEnnzB85ciRlZWV07NiRp556ir59+17ye57tueeeY+HChSQkJDBnzhyaNWtGUFDQGcukp6czZMgQunfvzvjx4/nHP/4BwMcff8zMmTPp1q0bnTt35osvvgDg1ltv5V//+heJiYnagayUixPjZBdhJSUlmbNvbrNjxw46duxY7W0YY8jOTKGxOQZ+oRAWawWFEysuLsbT0xMvLy9WrlzJgw8+2GCafmr6+Silap+IrDfGJFU23+X6DKpDRDCBzcnIE1oU5UBOOYTFg4eno0u7aAcPHmTMmDHYbDZ8fHyYMWOGo0tSSjkRtwwDgLAAb3bmheLn7UN4cSZk74OIVuDhnL+Stm3b8uuvvzq6DKWUk3LutpFL4OXhQZi/N+kl/pSHxELpScjaC+W1P1yFUko1dG4bBgARgb5W/0G5P4S3gvJiyNoDZXqevVLKvbh1GPh5exLo60X2iRJsvkEQ0QZsZVYglJ7/HH+llHJFbh0GAI0DfSktt1mjmfoEQOO2gIGjuyAvA2zlji5RKaXqnNuHQZCfFz5eHmSfGs3UuxFEdoBGYVBwGI7uhKLcWn/fwMBAADIyMiod+2fIkCGcfRrt2aZMmcLJkydPv67OkNi1LTk5mU8++aRe31MpVbvcPgxOjWZ6oqSMwlOjmXp6W9ceRLSxrj/I2W+dbVQHfQktWrQ4PSLpxTg7DKozJHZt0zBQyvm5fRiAdZqph8i59zrwDYLI9hDcAkoK4MhOyM8EYztjsaeeeoo33njj9Ovnn3+eyZMnU1BQwLBhw04PN33qyt6KkpOTSUhIAKCwsJBbb72Vjh07cuONN54xNtGDDz5IUlISnTt35rnnngOswe8yMjIYOnQoQ4cOBX4bEhvglVdeISEhgYSEBKZMmXL6/SobKruiOXPmkJCQQLdu3Rg0aBBgDYE9adIkevXqRdeuXXnrrbdO7/+yZcvo3r376WGxlVLOxTlPqq/Kd09B5pYareIFtC0rp9RmMD6eyNnjmTbrAsNfhLw0yD8EJ3MgJAr8ggEYO3Ysjz/+OA8//DAAs2fPZsGCBfj5+TFv3jyCg4PJysqib9++XHfddZXeF3jq1Kn4+/uzY8cONm/eTI8ePU7P+/vf/054eDjl5eUMGzaMzZs38+ijj/LKK6+wePFiGjdufMa21q9fz7vvvsvq1asxxtCnTx8GDx5MWFhYtYbKfvHFF1mwYAEtW7Y83ew0c+ZMQkJCWLt2LcXFxQwYMIARI0bw0ksvMXnyZL7++usa/d6VUg2HHhnYeXt6gIHS8kqG5/DysU4/DW8NGMjZBzkHoLyExMREjhw5QkZGBps2bSIsLIzo6GiMMTzzzDN07dqVK664gvT0dA4fPlxpDUuXLj39pdy1a1e6du16et7s2bPp0aMHiYmJbNu27YJ3IFu+fDk33ngjAQEBBAYGMnr0aJYtWwZUb6jsAQMGMGHCBGbMmEF5udWJvnDhQj744AO6d+9Onz59yM7OZs+ePVXWoZRyDq53ZHDVSxe1mgdw5GgBxWU22jcLwqOSv97xCwafjnDiMOQfhuI8CGjMLTeNZu7cuWRmZjJ27FjAGgDu6NGjrF+/Hm9vb+Li4s47dPWFHDhwgMmTJ7N27VrCwsKYMGHCRW3nlOoMlT1t2jRWr17NN998Q8+ePVm/fj3GGF5//XWuvPLKM5ZdsmTJRdeilGoY9MiggjNOM62KhwcENYcmHcE3GAqOMHZYDz79+APmzp1z+kYxubm5NGnSBG9vbxYvXkxKSkqVmx00aNDpjtitW7eyefNmAPLy8ggICCAkJITDhw/z3XffnV6nsmGmBw4cyPz58zl58iQnTpxg3rx5DBw4sNq/i3379tGnTx9efPFFIiMjSU1N5corr2Tq1KmUllq/n927d3PixAkd6lopF+B6RwaXIMjPCx9P6zTTUH+fC6/g5WvdI6G0iM6NwsnPy6Vl41CaNyqDshJuv/12rr32Wrp06UJSUhIdOnSocnMPPvggd911Fx07dqRjx4707NkTgG7dupGYmEiHDh2Ijo5mwIABp9eZOHEiI0eOpEWLFixevPj09B49ejBhwgR69+4NwL333ktiYmKld08726RJk9izZw/GGIYNG0a3bt3o2rUrycnJ9OjRA2MMkZGRzJ8/n65du+Lp6Um3bt2YMGECf/jDH6r1HkqphsMth7CuytH8Yg7lFtK2SSCNfGqYlWXF1rUJJ3Os1/7hENjUCg03pkNYK+V4FxrCWpuJzhLmb51meiS/mBoHpZcvhMZAk07gH2GFwpHtcCwFSi++jV8ppepanYWBiLwjIkdEZGsl80VEXhORvSKyWUR6nG+5+ubl6UGTIF9yC0s5drLkwiucdyM+EBoNTTtBQCQUHoejO6yzj0pO1G7BSilVC+ryyOA9YGQV868C2tofE4Gpl/JmtdncFRnkS6CvFxnHiygqvYSxiTx9rOsRmnaCwCZQnA9Zu61xjwqP1cotNxs6Z2uGVMpd1VkYGGOWAjlVLHI98IGxrAJCRaT5xbyXn58f2dnZtfbFIyJEh/vjIcLBnJPYbJe4XU9vCG4JTTtDcJQ1+N2xZKsJqeCwNVKqCzLGkJ2djZ+fn6NLUUpdgCPPJmoJpFZ4nWafdujsBUVkItbRAzExMedsKCoqirS0NI4ePVqrBRaXlpNeUEJWmidh1Tm7qLoMUGagOAvK0qzxj3wCrOEvnPROa5Xx8/MjKirK0WUopS7AKb55jDHTgelgnU109nxvb2/i4+Pr5L1f+m4n037ex+vjErm2W4vaf4OMjbBqKmz9zDpCaD8K+j0EsQOgsgvflFKqljnybKJ0ILrC6yj7tAblyRHt6BETytOfbyEluw46f1t0h9FvweNbYOATcHAFvHc1TO0Pa2ZAUV7tv6dSSp3FkWHwJfA7+1lFfYFcY8w5TUSO5u3pwWvjEvEQeGTWr5SU2S680sUIbg7D/gp/2A7XvW51Pn/7R3i5A3z5KBzaVDfvq5RS1O2ppbOAlUB7EUkTkXtE5AERecC+yLfAfmAvMAN4qK5quVRRYf788+aubE7L5f++31m3b+bjDz1+B/f/DPcthoQbYfNseGsQzLgcfv1Yb8mplKp1LnEFcn356xdb+WBlCjPvTGJYx6b198aFx2HTp7DuHcjaBX4h0P12SLrbfptOpZSq2oWuQNYwqIGi0nJGv7mCjNxCvntsIM1DGtVvAcZAyi+wdibs+ApspRA3EBLvgE7XWbfsVEqp89DhKGqRn7cn/7ktkZIyG4/N2khZeR31H1RGBOIug1vehSe2W30MuWkwbyJMbg9fPwEZv7rFxWxKqdqlYVBDrSID+dsNCaxJzuG1Hx14Y5fAJjDwSXhkA9z5NbQfCRs/hulDYNpAWP3WbwPmKaXUBWgYXITRPaK4qUcUry/ey4p9WY4txsMD4gfC6Onw5C64+mXw8ITv/mSdiTT3bti3GGz1fBSjlHIq2mdwkU4Ul3Htf5ZTUFTG5w/1JyrM39ElnSlzC2z4EDb/F4qOQ0gMdLvVekS0dnR1Sql6ph3IdWhnZh63TFtJSCNvZt3Xl+jwBhYIYA2dvfNr+PUjOPAzGBtE9bZCIWE0NApzdIVKqXqgYVDHNqcdZ/zbqwny8+bTiQ00EE7Jy4Atc2DjLGtIbU8faH8VdBsHba6wBtRTSrkkDYN6sCUtl/EzVxPo69XwAwGss40ObbKuXdgyB05mgX9j6HKzdcTQvLuOi6SUi9EwqCdb03O5/W0rEGbd15eYiAYeCKeUl8LeH2HTLNj1LZSXQOP20OUW6HIThLdydIVKqVqgYVCPtqZbRwj+3p58OrGf8wTCKYXHYNs82DLXurgNoGWSFQydb4SgerzqWilVqzQM6tm2DOsIwd/bk1kT+xIbEeDoki5Obhps/dxqRsrcbN1zIX6QFQwdr7WGxFBKOQ0NAwfYnpHH7W+vws/bk1n39SWusZMGwilHd1lHC1vmwLED4OkL7UZAws3QdoQ1uJ5SqkHTMHCQU4Hg6+XJpxNdIBDA6nhO3wBb51o34yk4DN7+ViB0vsEeDC6wn0q5IA0DB9pxKI/b316Nt6fw6cR+xLtCIJxiK4fk5bD9C2vQvBNHwKsRtB0Ona6HdiPBN9DRVSql7DQMHGxnZh63zbAC4ZP7+tI60gW/IG3lcHAlbJsPO760jhi8/KxrFzrdAO2uBL9gR1eplFvTMGgAdmXmc9uMVZQbw5u396B/68aOLqnu2MohdfVvwZB/yOpjaDUEOoyCdlfpWUlKOYCGQQORkn2Ce95fR3LWCV64vjO394l1dEl1z2aDtDVWU9LOr+H4QUAgKgnaj7Ieke31Ajel6oGGQQOSV1TKY7N+ZfGuo9zZL5a/XNMJL083GTjWGDi8zbqwbec3cGijNT28lRUKHa6G6D7WiKtKqVqnYdDAlNsML323gxnLDnBZm8a8cVsPQvzdcEyg3HTY/R3s/BYOLLXu2tYo3Op4bj8SWg3VfgalapGGQQM1e10q/2/eFqLC/JnxuyTaNHHBjuXqKsqDfT9awbBnARTlgoc3xPa3wqHdlTrstlKXSMOgAVuXnMP9H66npNzGf27rweB2kY4uyfHKy6x+ht3fw+4FcHSnNT2iDbS90gqGmH7g5ePYOpVyMhoGDVzasZPc98F6dmXm8ezVnbhrQByiHaq/OZYMuxda4ZC8zBpIzzcYWg+1LnJrNRRCWjq6SqUaPA0DJ3CiuIwnZm9kwbbD3NormhevT8DHy006lmuiuMC6Qc/u762AKMi0pjduD60vtx5xA/QqaKXOQ8PASdhshlcX7eb1n/bSOy6c/9yWSJNgP0eX1XAZA0e2w76frEfKCigrsvoaYvpaRw6tL4dm3az7RCvl5jQMnMwXG9P582ebCfT14tWx3RnYVvsRqqW00LoKet9i63F4izXdPwLiB1sjrsYPsk5l1WY45YY0DJzQnsP5PPzJBvYcKeChIa35wxXt3Od6hNqSfxj2L7GOGvYv+a1JKagFxF0G8QMhbiCExWk4KLegYeCkCkvKeeGrbXy6NpWk2DBeG5dIi9BGji7LORkD2Xut6xmSl1kD7J04as0LibZC4VQ4hEY7tlal6ohDw0BERgL/BjyBt40xL501PxZ4B4gEcoDxxpi0qrbpLmFwyhcb03nm8y14e3kw+eZuXNFJx/W5ZMZYp6weWAbJSyH5FyjMseaFRENUL+tq6Ohe0KwreLrhRYHK5TgsDETEE9gNDAfSgLXAOGPM9grLzAG+Nsa8LyKXA3cZY+6oarvuFgYAB7JO8PDHG9h+KI97LovnzyM76NlGtclmszqjk5fBwVWQthby0q15Xo2gRaIVDNF9IKo3BGo/jnI+jgyDfsDzxpgr7a+fBjDG/KPCMtuAkcaYVLFOrs81xlQ5BoE7hgFAUWk5//h2B++vTKFbVAivj+vhfPdYdia5aZC6xgqG1NVwaLM1ZAZY/QxRvaFlDysomnXR01lVg+fIMLgZ64v+XvvrO4A+xpjfV1jmE2C1MebfIjIa+AxobIzJPmtbE4GJADExMT1TUlLqpGZn8P3WQ0yauxkM/N/NXRnVpbmjS3IPpYVwaJMVDKlrIG3db53S4gGRHaxgOPVomgDeemqwajgaehi0AP4DxANLgZuABGPM8cq2665HBhWl5pzk97N+ZVPqccb1juHZqzsS4Ovl6LLcT94ha/TVjF+tR/oGOJllzfPwgiYd7UcOXa2jhyaddPA95TANupnorOUDgZ3GmKiqtqthYCkps/Hywl1MX7afmHB/XhnTnZ6xYY4uy70ZY/U1nAqHU4/CY78tExprHTU0S4Cmna3nYfF6YZyqc44MAy+sDuRhQDpWB/JtxphtFZZpDOQYY2wi8neg3Bjz16q2q2FwplX7s3ly9iYO5Rby0JA2PDqsrXYuNySnAuLwNsjcAoe3Ws+z94KxWct4B0DTTlY4NOlkHVFEdtSOalWrHH1q6ShgCtappe8YY/4uIi8C64wxX9qbkv4BGKxmooeNMcVVbVPD4Fz5RaW8+NV25qxPo3OLYKaM7U7bpkGOLktVpeSkdXrrqXDI3Go9L6rQQurf2AqGU4/IjtCkAzTSI0BVc3rRmRtZsC2Tpz/fQkFxGX+6sj13D4jHw0OvrnUaxkDBYes01yM7rZ9Hd8KRHVBS8NtyQc2hcTtrWO/Tj9ZWE5Sn9h2p89MwcDNH84t5+vPNLNpxhH6tIpg8phst9cpl52YM5KaeGRBZeyB7j3UjoFM8vKzTXisGRHhr66rq4Jbg5euwXVCOp2Hghowx/HdtKv/z9XY8RHj+us6M7tFS75PgaoyBkzlW/8MZj32Qs88axbWiwKYQEmV/RFd4bn/tH6HjNLkwDQM3djD7JE/M3si6lGNcldCM/7khgcaB+tehW7DZrI7rnP3Wz9w06+giN816HE+FssIz1/H0heDm1lFEcAv7o6XVLHVqWmAT8PB0zD6pS6Jh4ObKbYbpS/fzyg+7CPLz5vnrOnNt1+Z6lODujLFOeT0VEMdTrdDIPwR5GdbzvAzrznIViad1hBEQYXVw+0dAQGPreUCE9dq/sX1ahNXZreHRIGgYKAB2H85n0pxNbErLZUSnpvztxgSaBOkVsqoKxsDJbHs4VAiI/Ezr4roTWdb8k9lQnHf+bYiHFQgVA+KMAKkwzT8cGoWDjw6zUhc0DNRpZeU2Zi4/wMs/7KaRtyfPXduJGxO1L0HVgrLi34LhVEicDouss6ZlWX0dVPLd4+VnBUijcHtAhFZ4Hm5dxe0TCL5Bv/30DQSfIOu5l6/2fZyHhoE6x76jBfxp7mbWpxzj8g5N+N8bu9AsRI8SVD2ylUPh8QpBkWU1W53MsX4W5sDJYxWe51g/bWUX3raHlxUSPoHW+FCevlZAePlV+Olz5mtPH2uo8jN+nue5h5dVu7FZD1s5mPKznpsKy1ScZ7PmmfOsX15qNcmVFf/2/IyHfVrXsdD7vov6lWsYqPMqtxneW5HMvxbsxNvDg2ev6ciYpGg9SlANlzHW9RZFedbP4gKreer083woya/w/ASUF1tfsGVF9p8Vn1f4eerL9tTItPVBPOwPTyuUzgies8Lo1PzOo6FHlaP8V/52GgaqKslZJ/jTZ5tZcyCHgW0b89JNXfW6BOW+jLGOPs7+i/zUc1vZb1/gHp5Wc9Tp5x5nPbc/Ks47Pa3+h4zRMFAXZLMZPlqdwkvf7USAp0Z15PbeMXr1slIu5EJhoCOaKTw8hN/1i2PB44NIjAnjL/O3Muatlew9ku/o0pRS9UTDQJ0WHe7Ph/f05l83d2Xv0QJG/Xs5Uxbtpris3NGlKaXqmIaBOoOIcEtSNIueGMzIhGZMWbSHq19bzvqUHEeXppSqQxoG6rwaB/ry2rhE3p3Qi8KScm6etpK/zN9KflE9nm2hlKo3GgaqSkM7NGHhHwZxV/94PlqdwvBXlrJwW6ajy1JK1TINA3VBAb5e/PXaTsx7aACh/t5M/HA9D360niN5RRdeWSnlFDQMVLV1jw7lq0cuY9KV7flx5xGGvfIzH65KwWZzrtOTlVLn0jBQNeLt6cHDQ9vw/WMD6dIyhL/M38roqSvYlpF74ZWVUg2WhoG6KK0iA/n43j68OrYbqTknufb15fzP19spKK7G2DFKqQZHw0BdNBHhxsQofnpyCLf2jmHm8gMMf+Vnvt+aibNd2a6Uu9MwUJcsxN+b/72xC5892J+QRt488NF67vtgHWnHTjq6NKVUNWkYqFrTMzaMrx65jGdGdeCXvdkMf2Up037eR2m5zdGlKaUuQMNA1SpvTw8mDmrNoicHM6BNY176bifXvLacdcl6BbNSDVm1wkBEHhORYLHMFJENIjKirotTzqtlaCPevjOJ6Xf0JL+olJunreSPczaRXVDs6NKUUudR3SODu40xecAIIAy4A3ipzqpSLmNE52b88MRg7h/civm/pnP5yz/z0aoUyvXaBKUalOqGwamB7UcBHxpjtlWYplSVAny9ePqqjnz32EA6NAvi2flbufHNX9icdtzRpSml7KobButFZCFWGCwQkSBAewVVjbRtGsSnE/syZWx3Mo4Xcf0bv/Ds/C3kntTB75RytOqGwT3AU0AvY8xJwBu460IriRmSvSAAABcqSURBVMhIEdklIntF5KnzzI8RkcUi8quIbBaRUTWqXjkdEeGGxJb89MfB3Nkvjk9WH+Tyl5cwd32aXpuglANVNwz6AbuMMcdFZDzwLFDl+AMi4gm8AVwFdALGiUinsxZ7FphtjEkEbgXerEnxynkF+3nz/HWd+fL3lxET4c8f52xizFsr2ZmZ5+jSlHJL1Q2DqcBJEekGPAnsAz64wDq9gb3GmP3GmBLgU+D6s5YxQLD9eQiQUc16lItIaBnCZw/05/9u6sLeIwVc/dpyXvhqG7mF2nSkVH2qbhiUGesY/nrgP8aYN4CgC6zTEkit8DrNPq2i54HxIpIGfAs8cr4NichEEVknIuuOHj1azZKVs/DwEMb2iuGnJ4cwtlc0761IZtjLS5i9LlVHRFWqnlQ3DPJF5GmsU0q/EREPrH6DSzUOeM8YE4X9TCX7ts9gjJlujEkyxiRFRkbWwtuqhigswIf/vbELXz58GTHh/vxp7mZGT13BplQ960ipulbdMBgLFGNdb5AJRAH/usA66UB0hddR9mkV3QPMBjDGrAT8gMbVrEm5qC5RIcx9oD8v39KNtGOF3PDmLzz12Wa9YE2pOlStMLAHwMdAiIhcAxQZYy7UZ7AWaCsi8SLig9VB/OVZyxwEhgGISEesMNB2IIWHh3BTzygW/3Ew9wyIZ+76NIZOXsL7K5Ip07GOlKp11R2OYgywBrgFGAOsFpGbq1rHGFMG/B5YAOzAOmtom4i8KCLX2Rd7ErhPRDYBs4AJRs8vVBUE+Xnz7DWd+P7xgXSNCuW5L7dxzevLWb0/29GlKeVSpDrfvfYv6+HGmCP215HAImNMtzqu7xxJSUlm3bp19f22qgEwxvD91kz+9s0O0o8Xcm23Fjx9VQdahDZydGlKNXgist4Yk1TZ/Or2GXicCgK77Bqsq1StEBGu6tKcRU8M5tFhbVm4LZPLX17Cvxftoai03NHlKeXUqvuF/r2ILBCRCSIyAfgG61RQpepdIx9Pnhjejh+fHMywDk15ddFuhr38M99uOaRXMSt1karVTAQgIjcBA+wvlxlj5tVZVVXQZiJ1tlX7s3nhq+3sOJRH31bh/PWaznRqEXzhFZVyIxdqJqp2GDQUGgbqfMpthllrDvLywl3kFpYyrncMT45oT3iAj6NLU6pBuKQwEJF8rCEjzpkFGGNMvf/5pWGgqpJ7spRXF+3mw1UpBPh48ofh7RjfNxZvT+3iUu5NjwyUW9p9OJ8Xv9rO8r1ZtG0SyLPXdGJwO716Xbmv2jqbSCmn0q5pEB/e05vpd/SkpNzGne+sYcK7a9h7JN/RpSnVIGkYKJclIozo3IyFfxjE/xvVkfUpx7hyyjKe+2Irx06UOLo8pRoUDQPl8ny9PLlvUCuW/HEI43pH8+GqFAb/azFvL9tPSZkObaEUaBgoNxIR6MvfbujCd48Nolt0KH/7ZgdXTlnKD9sP6/UJyu1pGCi3075ZEB/c3Zt3J/RCBO77YB23v72aHYf0LmvKfWkYKLckIgzt0IQFjw/i+Ws7sf1QHle/toynPtvMkfwiR5enVL3TMFBuzdvTgwkD4lnyxyHc2T+OuevTGPKvJbz24x5OlpQ5ujyl6o2GgVJAqL8Pz13bmR+eGMzgdpG88sNuhk5ewpx1qZTrrTeVG9AwUKqC+MYBTB3fkzkP9KNZSCMmzd3MNa8v55e9WY4uTak6pWGg1Hn0igtn/kP9eW1cInmFpdz+9mrufm8tew7rRWvKNWkYKFUJEeG6bi348cnBPH1VB9Ym53DllKU8M28LR/P1fszKtejYREpVU86JEl77cQ8frUrB18uD+we35t6B8fj7eDm6NKUuSAeqU6qW7T9awD+/38X32zKJDPLlD1e0Y0xSFF46MqpqwHSgOqVqWavIQKbd0ZPPHuxHbLg/z8zbwpVTlrJwW6ZeyaycloaBUhepZ2w4cx7ox/Q7emKAiR+uZ8xbK9lw8JijS1OqxjQMlLoEp0dGfXwQf78xgeTsk4x+cwUPfrSe/UcLHF2eUtWmfQZK1aITxWW8vewA05fuo6jMxrje0Tw2rB2RQb6OLk25Oe1AVsoBjuYX8/pPe/hk9UF8vDy497J47h3UimA/b0eXptyUhoFSDnQg6wQvL9zF15sPEebvzcND2zC+byx+3p6OLk25GQ0DpRqALWm5/HPBTpbtyaJFiB+PD2/HTT2i8PQQR5em3ISeWqpUA9AlKoQP7+nDJ/f2ITLIlz/N3czIKUtZoKejqgaiTsNAREaKyC4R2SsiT51n/qsistH+2C0ix+uyHqUcrX+bxsx/eADTxveg3Bju/3A9o6euYNX+bEeXptxcnTUTiYgnsBsYDqQBa4FxxpjtlSz/CJBojLm7qu1qM5FyFWXlNj7bkMarP+whM6+Iwe0imXRlexJahji6NOWCHNlM1BvYa4zZb4wpAT4Frq9i+XHArDqsR6kGxcvTg7G9YlgyaQjPjOrAprTjXPP6ch7+eAN7j+g1Cqp+1WUYtARSK7xOs087h4jEAvHAT5XMnygi60Rk3dGjR2u9UKUcyc/bk4mDWrP0T0N5dFhbluw6wohXf2bSnE2kHTvp6PKUm2goHci3AnONMeXnm2mMmW6MSTLGJEVGRtZzaUrVj2A/b54Y3o6lfxrK3QPi+WJTBkMnL+G5L7bqfZlVnavLMEgHoiu8jrJPO59b0SYipQCICPTl2Ws68fOkIdzcM5qPVh9k8D+X8H/f7yT3ZKmjy1Muqi47kL2wOpCHYYXAWuA2Y8y2s5brAHwPxJtqFKMdyMrdHMg6wZRFu/lyUwaBvl7cP6gVdw2IJ8BX76Ogqs9hHcjGmDLg98ACYAcw2xizTUReFJHrKix6K/BpdYJAKXcU3ziAf9+ayLePDqRPfDiTF+5m4D8XM33pPgpLztuyqlSN6RXISjmZDQeP8crC3Szfm0VkkC8PDWnNuN4xOsSFqpIOR6GUi1q9P5tXftjN6gM5NAv24+HL2zA2KRofr4ZyXohqSDQMlHJhxhhW7Mvm5YW72HDwOC1DG/HosDaM7hGFt96GU1WgYaCUGzDG8PPuo7z6w242peUSG+HPo5e35YbEljoYngJ0oDql3IKIMKR9E+Y/PIC3f5dEgI8XT87ZxPBXf+aLjemU25zrjz5V/zQMlHIhIsIVnZry9SOXMfX2Hvh4evDYpxs1FNQFaTORUi7MZjMs2JbJv3/cw87MfFpFBvDYsLZc07WFNh+5Ge0zUEppKCgNA6XUbzQU3JeGgVLqHOcLhUcvb8s1XZvjpaekuiQNA6VUpc4OhbgIfx4a2oYbE1vqdQouRsNAKXVBNpth4fbDvP7THrZl5BEV1oiHhrThpp4t8fXSYS5cgYaBUqrajDEs3nWEf/+4l02px2ke4scDg1sztle0jn3k5DQMlFI1Zoxh2Z4sXv9pD2uTj9EkyJeJg1pxe59YGvloKDgjDQOl1EUzxrBqfw6v/biHlfuziQjw4d6BrRjfN4YgP29Hl6dqQMNAKVUr1iXn8NpPe1m6+yjBfl5MGBDPXf3jCAvwcXRpqho0DJRStWpT6nHeWLyXhdsP4+/jyfi+sdx7WTxNgv0cXZqqgoaBUqpO7MrM580le/lqUwZenh6MTYpm4qBWRIf7O7o0dR4aBkqpOpWcdYK3lu5j7vo0jIHru7fkwSGtadMk0NGlqQo0DJRS9eJQbiHTl+5n1pqDFJfZuCqhGQ8NaUNCyxBHl6bQMFBK1bOsgmLeWX6AD1emkF9cxqB2kTw4uDV9W4UjouMfOYqGgVLKIXILS/l4dQrvLD9AVkEJ3aNDeXBIa4Z3bIqHDopX7zQMlFIOVVRazpz1aUxfuo/UnEJaRwbwwODWXN+9JT5eOv5RfdEwUEo1CGXlNr7ZcoipS/axMzOf5iF+3DuwFbf2iibA18vR5bk8DQOlVINijGHJ7qNMXbKPNQdyCPX35s5+cfyuXywRgb6OLs9laRgopRqs9Sk5TF2yn0U7DuPn7cEtPaO5d2A8sREBji7N5WgYKKUavD2H85mxbD/zf82gzGbjqoTmTBzUim7RoY4uzWVoGCilnMaRvCLeXZHMR6tSyC8qo098OPcPbsWQdk30DKRLdKEwqNOufBEZKSK7RGSviDxVyTJjRGS7iGwTkU/qsh6lVMPWJNiPP4/swMqnh/Hs1R05mHOSu99bx8h/L2XOulRKymyOLtFl1dmRgYh4AruB4UAasBYYZ4zZXmGZtsBs4HJjzDERaWKMOVLVdvXIQCn3UVpu4+vNGbz18352ZubTNNiXuwbEM653DCGNdAjtmnBYM5GI9AOeN8ZcaX/9NIAx5h8VlvknsNsY83Z1t6thoJT7McawdE8W05fu45e92QT4eDKmVzR3D4jXgfGq6UJhUJcn97YEUiu8TgP6nLVMOwAR+QXwxAqP78/ekIhMBCYCxMTE1EmxSqmGS0QY3C6Swe0i2ZaRy8xl1nAX769I5qqE5tw7MJ7EmDBHl+nUHH35nxfQFhgCjANmiMg5pw8YY6YbY5KMMUmRkZH1XKJSqiHp3CKEV8Z2Z9mfh3LfoFYs3XOUG99cwc1TV/D91kzKbc51UkxDUZdhkA5EV3gdZZ9WURrwpTGm1BhzAKuPoW0d1qSUchHNQxrx9FUdWfX0MJ67thOH84t44KP1XP7yEj5YmczJkjJHl+hU6rLPwAvry30YVgisBW4zxmyrsMxIrE7lO0WkMfAr0N0Yk13ZdrXPQCl1PmXlNhZuP8z0pfvZmHqckEbe3NYnht/1i6V5SCNHl+dwDuszMMaUicjvgQVY/QHvGGO2iciLwDpjzJf2eSNEZDtQDkyqKgiUUqoyXp4ejOrSnKsSmrE+5Rgzlx/grZ/3MWPpfkZ1ac49l8XrRWxV0IvOlFIuKzXnJO+tSOa/a1MpKC6jZ2wY91wWz4hOTfHydHSXaf3SK5CVUm4vv6iUOevSeHfFAVJzCmkZ2ogJ/eMY2zuaYD/3uF5Bw0AppezKbYZFOw4zc/kB1hzIIcDHk1uSopnQP464xq49OJ6GgVJKncfW9FzeWX6ArzZnUGYzXN6+CRMGxHFZm8YueXtODQOllKrCkbwiPlp9kE9Wp5BVUEKbJoFM6B/H6B4t8fdxnZvuaBgopVQ1FJeV8/WmQ7y74gBb0/MI9vPi1t4x3NE31iWGvNAwUEqpGjDGsD7lGO/+ksz32zIxxjC8U1Mm9I+nb6twp21CcuTYREop5XREhKS4cJLiwsk4XsiHq1KYteYgC7YdpkOzIO7sH8f13Vu4VBMS6JGBUkpdUFFpOfN/Tee9FcnszMwn2M+LMUnR3NEv1mlu0anNREopVUuMMaxNPsb7K5NZsDWTcmMY0i6S3/WPY3DbyAZ9NzZtJlJKqVoiIvSOD6d3fDiZuUV8suYgn6w+yF3vriU2wp87+sZyS89oQvyd70I2PTJQSqlLUFJm47uth/hwZQrrUo7RyNuTGxJb8rt+sXRsHuzo8k7TZiKllKonW9Nz+XBlCvM3plNcZiMpNozxfWO5qkszfL08HVqbhoFSStWz4ydLmLs+jY9WpZCcfZKIAB/G9Irmtt4xDrtmQcNAKaUcxGYz/LIviw9XprBox2EMMLR9E+7oG8ugdpF41mOHs4aBUko1ABnHC5m15iCz1qSSVVBMdHgjbusdy5ikKCICfev8/TUMlFKqASkps7FweyYfrkxh9YEcfDw9GJnQjNv7xNA7vu6ucNYwUEqpBmr34Xw+XpXC57+mk19URpsmgdzWO4abekTV+umpGgZKKdXAFZaU89XmDD5efZBNqcfx9fLgmq4tuK1PDD1iQmvlaEHDQCmlnMjW9Fw+WXOQL35N50RJOR2aBXF7nxiuT2x5SXdl0zBQSiknVFBcxpcbM/h4dQrbMvJo5O3JkyPace/AVhe1PR2OQimlnFCgrxe39YlhXO9oNqXl8snqFFqENqqz99MwUEqpBkxE6B4dSvfo0Dp9H4863bpSSimnoGGglFJKw0AppZSGgVJKKTQMlFJKoWGglFIKDQOllFJoGCillMIJh6MQkaNAykWu3hjIqsVyGgJX2ydX2x9wvX1ytf0B19un8+1PrDEmsrIVnC4MLoWIrKtqbA5n5Gr75Gr7A663T662P+B6+3Qx+6PNREoppTQMlFJKuV8YTHd0AXXA1fbJ1fYHXG+fXG1/wPX2qcb741Z9Bkoppc7P3Y4MlFJKnYeGgVJKKfcJAxEZKSK7RGSviDzl6Hpqg4gki8gWEdkoIk53L1AReUdEjojI1grTwkXkBxHZY/8Z5sgaa6qSfXpeRNLtn9NGERnlyBprQkSiRWSxiGwXkW0i8ph9ulN+TlXsjzN/Rn4iskZENtn36QX79HgRWW3/zvuviPhUuR136DMQEU9gNzAcSAPWAuOMMdsdWtglEpFkIMkY45QXy4jIIKAA+MAYk2Cf9k8gxxjzkj20w4wxf3ZknTVRyT49DxQYYyY7sraLISLNgebGmA0iEgSsB24AJuCEn1MV+zMG5/2MBAgwxhSIiDewHHgMeAL43BjzqYhMAzYZY6ZWth13OTLoDew1xuw3xpQAnwLXO7gmt2eMWQrknDX5euB9+/P3sf6jOo1K9slpGWMOGWM22J/nAzuAljjp51TF/jgtYymwv/S2PwxwOTDXPv2Cn5G7hEFLILXC6zSc/B+AnQEWish6EZno6GJqSVNjzCH780ygqSOLqUW/F5HN9mYkp2hSOZuIxAGJwGpc4HM6a3/AiT8jEfEUkY3AEeAHYB9w3BhTZl/kgt957hIGruoyY0wP4CrgYXsThcswVhumK7RjTgVaA92BQ8DLji2n5kQkEPgMeNwYk1dxnjN+TufZH6f+jIwx5caY7kAUVktIh5puw13CIB2IrvA6yj7NqRlj0u0/jwDzsP4ROLvD9nbdU+27RxxczyUzxhy2/2e1ATNwss/J3g79GfCxMeZz+2Sn/ZzOtz/O/hmdYow5DiwG+gGhIuJln3XB7zx3CYO1QFt777oPcCvwpYNruiQiEmDvAENEAoARwNaq13IKXwJ32p/fCXzhwFpqxakvTbsbcaLPyd45ORPYYYx5pcIsp/ycKtsfJ/+MIkUk1P68EdaJMjuwQuFm+2IX/Izc4mwiAPupYlMAT+AdY8zfHVzSJRGRVlhHAwBewCfOtk8iMgsYgjXc7mHgOWA+MBuIwRqqfIwxxmk6ZCvZpyFYzQ8GSAbur9De3qCJyGXAMmALYLNPfgarnd3pPqcq9mcczvsZdcXqIPbE+gN/tjHmRft3xKdAOPArMN4YU1zpdtwlDJRSSlXOXZqJlFJKVUHDQCmllIaBUkopDQOllFJoGCillELDQKl6JSJDRORrR9eh1Nk0DJRSSmkYKHU+IjLePkb8RhF5yz4QWIGIvGofM/5HEYm0L9tdRFbZBzmbd2qQMxFpIyKL7OPMbxCR1vbNB4rIXBHZKSIf26+KVcqhNAyUOouIdATGAgPsg3+VA7cDAcA6Y0xn4Gesq4sBPgD+bIzpinVl66npHwNvGGO6Af2xBkADa6TMx4FOQCtgQJ3vlFIX4HXhRZRyO8OAnsBa+x/tjbAGYrMB/7Uv8xHwuYiEAKHGmJ/t098H5tjHjWppjJkHYIwpArBvb40xJs3+eiMQh3VDEqUcRsNAqXMJ8L4x5ukzJor85azlLnYsl4rjw5Sj/w9VA6DNREqd60fgZhFpAqfv9xuL9f/l1CiQtwHLjTG5wDERGWiffgfws/0uWmkicoN9G74i4l+ve6FUDehfJEqdxRizXUSexbqLnAdQCjwMnAB62+cdwepXAGt44Gn2L/v9wF326XcAb4nIi/Zt3FKPu6FUjeiopUpVk4gUGGMCHV2HUnVBm4mUUkrpkYFSSik9MlBKKYWGgVJKKTQMlFJKoWGglFIKDQOllFLA/wdZufyr+ic++AAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "O comportamento acima das curvas de loss mostra que o modelo aprende bem os padrões dos dados de treinamento, ganhando mais confiança nas suas classificações. Já a curva para os dados de validação (desconhecidos para o treinamento do modelo) mostram que a *loss* se estabiliza após algumas épocas.\n",
        "\n",
        "**O que fazer para melhorar?**\n",
        "\n",
        "Dá para fazer muita coisa...\n",
        "\n",
        "*   Otimização dos hiperparâmetros: altere a quantidade de camadas intermediárias, a quantidade de neurônios em cada camada intermediária, a quantidade de épocas de treinamento, o algoritmo de otimização (veja [aqui](https://keras.io/api/optimizers/) outros otimizadores);\n",
        "*   Etapas de pré-processamento dos *tweets*: remoção de URLs, @'s, etc...\n",
        "*   A caracterização dos *tweets* (por exemplo, o [BERT](https://keras.io/examples/nlp/text_extraction_with_bert/)!)...\n",
        "\n",
        "\n",
        "### Avaliação de performance utilizando os dados de teste\n",
        "\n",
        "Agora fazemos a classificação das instâncias do conjunto de teste utilizando o método *predict*. Observe que uma matriz é gerada na saída, em que cada linha representa um *tweet* e as colunas representam:\n",
        "\n",
        "\n",
        "*   Coluna $0$: probabilidade estimada do modelo em classificar o $i$-ésimo *tweet* como sendo da classe \"0\" (polaridade negativa)\n",
        "*   Coluna $1$: probabilidade estimada do modelo em classificar o $i$-ésimo *tweet* como sendo da classe \"1\" (polaridade neutra)\n",
        "*   Coluna $2$: probabilidade estimada do modelo em classificar o $i$-ésimo *tweet* como sendo da classe \"2\" (polaridade positiva)\n",
        "\n"
      ],
      "metadata": {
        "id": "uWuQ0prhiiY4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y = model.predict(tfidf_test)\n",
        "y"
      ],
      "metadata": {
        "id": "V1mox-in1dt-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7b02b702-f6d7-4a43-ad43-1d7c27e6cecc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.28323925, 0.36455244, 0.56240934],\n",
              "       [0.2587081 , 0.27768308, 0.7726947 ],\n",
              "       [0.37565696, 0.74888754, 0.45494932],\n",
              "       ...,\n",
              "       [0.40348127, 0.65766186, 0.5104213 ],\n",
              "       [0.11433679, 0.3026824 , 0.7258324 ],\n",
              "       [0.36048415, 0.5728992 , 0.53748167]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "De maneira genérica, podemos calcular argmax das probabilidades, tomando-se as colunas como referência. Suponha que todas as probabilidades estejam na variável ```y_prob```, em que cada linha se trata de uma instância e teremos três colunas:\n",
        "\n",
        "*   ```y_prob[i][0]```: estimativa da probabilidade de classificar a $i$-ésima instância  de ```X_test``` como sendo da classe \"0\"\n",
        "*   ```y_prob[i][1]```: estimativa da probabilidade de classificar a $i$-ésima instância de ```X_test``` como sendo da classe \"1\"\n",
        "*   ```y_prob[i][2]```: estimativa da probabilidade de classificar a $i$-ésima instância de ```X_test``` como sendo da classe \"2\"\n",
        "\n",
        "Assim, com o uso de $argmax$, para cada instância $i$ podemos obter o índice $j$ da coluna associada à classe com a maior probabilidade:"
      ],
      "metadata": {
        "id": "Vlq2q3Fuk185"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_prob = model.predict(tfidf_test)\n",
        "y_pred = np.argmax(y_prob,axis=1)"
      ],
      "metadata": {
        "id": "ScVRiIVWk2px"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vamos imprimir os resultados da classificação de *tweets* para os dados de teste:"
      ],
      "metadata": {
        "id": "_QjL-SHZrl7h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(y_test,y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lSqjqALvrmDw",
        "outputId": "6f36d76d-06d6-4b3a-966c-cb8e561da934"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.61      0.16      0.25       559\n",
            "           1       0.58      0.81      0.68      1513\n",
            "           2       0.70      0.61      0.65      1475\n",
            "\n",
            "    accuracy                           0.63      3547\n",
            "   macro avg       0.63      0.53      0.53      3547\n",
            "weighted avg       0.64      0.63      0.60      3547\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Outra possibilidade é o uso do método *evaluate*:\n",
        "\n"
      ],
      "metadata": {
        "id": "PvAl8nUD2CQ9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(model.evaluate(tfidf_test,y_test_enc))"
      ],
      "metadata": {
        "id": "6RMsY7GK2E1r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bf4d506a-729c-44d7-a3f3-6d21a9d06ba5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "111/111 [==============================] - 0s 2ms/step - loss: 0.8355 - accuracy: 0.6264\n",
            "[0.8354759216308594, 0.6264448761940002]\n"
          ]
        }
      ]
    }
  ]
}