lectures/cap04_text_preprocessing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cap04_text_preprocessing.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOuAb7BmG1JeI+NPwbz/vUT",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/viniciusrpb/cic0269_natural_language_processing/blob/main/cap04_text_preprocessing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Capítulo 4 - Pré-Processamento de Textos\n",
        "\n",
        "Neste capítulo, estudaremos abordagens para pré-processar os textos visando analisar o texto e efetuar alterações de modo a manter palavras representativas à semântica e ao discurso.\n",
        "\n",
        "Assim, palavras como artigos e preposições podem ser removidas\n",
        "\n",
        "## 4.1. Dicas sobre o Google Colab\n",
        "\n",
        "Primeiramente, vamos instalar a biblioteca nltk para o uso de stemmers e lematizadores:"
      ],
      "metadata": {
        "id": "4tTX7YJtZcYH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install nltk"
      ],
      "metadata": {
        "id": "_rjQlS-8aRN0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "df4b5478-1dd7-4a91-a13f-1e380dfbb850"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (3.7)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from nltk) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from nltk) (1.1.0)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.7/dist-packages (from nltk) (2022.6.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from nltk) (4.64.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Agora vamos importar as bibliotecas necessárias para utilizarmos as funções e métodos para pré-processamento de textos utilizando NLTK:"
      ],
      "metadata": {
        "id": "H3XHXGWIcJED"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import re\n",
        "import nltk\n",
        "from nltk.stem.porter import *\n",
        "from nltk.stem import WordNetLemmatizer"
      ],
      "metadata": {
        "id": "BTin7dPbcIBU"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Faz o download da lista de stopwords"
      ],
      "metadata": {
        "id": "4r-a56pocbeC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('stopwords')\n",
        "nltk.download('omw-1.4')\n",
        "nltk.download('wordnet')"
      ],
      "metadata": {
        "id": "YYtzneP1cbwp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f11dcf5a-010e-44cf-d560-4a4d6089b228"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/omw-1.4.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Podemos carregar arquivos diretamente do Google Drive de acordo com os comandos abaixo:"
      ],
      "metadata": {
        "id": "m_E6YAHLcu3w"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "agELqmI0BzKj",
        "outputId": "a853dfec-0362-49ea-bf93-1e3ed170048e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Copiando o arquivo "
      ],
      "metadata": {
        "id": "lnzIkspn2s5-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cp -r '/content/drive/MyDrive/sentiment_analysis/polarity_classification_2013/twitter-2013train-A.txt' 'twitter-2013train-A.txt'"
      ],
      "metadata": {
        "id": "k-9vJdVu2tCq"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Importando arquivo csv que estão no Drive para o ambiente Colab"
      ],
      "metadata": {
        "id": "xZvP-HA5DdJA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv('twitter-2013train-A.txt',sep='\\t',encoding='utf-8',names=['id','polarity','text'])"
      ],
      "metadata": {
        "id": "xEHl86HHFIo9"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "G1vhbfW_FZZV",
        "outputId": "eb7efcd3-0b20-48a3-b12e-cfe15b0291da"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                      id  polarity  \\\n",
              "0     264183816548130816  positive   \n",
              "1     263405084770172928  negative   \n",
              "2     262163168678248449  negative   \n",
              "3     264249301910310912  negative   \n",
              "4     262682041215234048   neutral   \n",
              "...                  ...       ...   \n",
              "9679  103158179306807296  positive   \n",
              "9680  103157324096618497  positive   \n",
              "9681  100259220338905089   neutral   \n",
              "9682  104230318525001729  positive   \n",
              "9683  100461938533863424  negative   \n",
              "\n",
              "                                                   text  \n",
              "0     Gas by my house hit $3.39!!!! I\\u2019m going t...  \n",
              "1     Theo Walcott is still shit\\u002c watch Rafa an...  \n",
              "2     its not that I\\u2019m a GSP fan\\u002c i just h...  \n",
              "3     Iranian general says Israel\\u2019s Iron Dome c...  \n",
              "4     Tehran\\u002c Mon Amour: Obama Tried to Establi...  \n",
              "...                                                 ...  \n",
              "9679  RT @MNFootNg It's monday and Monday Night Foot...  \n",
              "9680  All I know is the road for that Lomardi start ...  \n",
              "9681  All Blue and White fam, we r meeting at Golden...  \n",
              "9682  @DariusButler28   Have a great game agaist Tam...  \n",
              "9683  I'm pisseeedddd that I missed Kid Cudi's show ...  \n",
              "\n",
              "[9684 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-815fff57-9d58-4d39-8c1f-4a52015a44d2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>polarity</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>264183816548130816</td>\n",
              "      <td>positive</td>\n",
              "      <td>Gas by my house hit $3.39!!!! I\\u2019m going t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>263405084770172928</td>\n",
              "      <td>negative</td>\n",
              "      <td>Theo Walcott is still shit\\u002c watch Rafa an...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>262163168678248449</td>\n",
              "      <td>negative</td>\n",
              "      <td>its not that I\\u2019m a GSP fan\\u002c i just h...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>264249301910310912</td>\n",
              "      <td>negative</td>\n",
              "      <td>Iranian general says Israel\\u2019s Iron Dome c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>262682041215234048</td>\n",
              "      <td>neutral</td>\n",
              "      <td>Tehran\\u002c Mon Amour: Obama Tried to Establi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9679</th>\n",
              "      <td>103158179306807296</td>\n",
              "      <td>positive</td>\n",
              "      <td>RT @MNFootNg It's monday and Monday Night Foot...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9680</th>\n",
              "      <td>103157324096618497</td>\n",
              "      <td>positive</td>\n",
              "      <td>All I know is the road for that Lomardi start ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9681</th>\n",
              "      <td>100259220338905089</td>\n",
              "      <td>neutral</td>\n",
              "      <td>All Blue and White fam, we r meeting at Golden...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9682</th>\n",
              "      <td>104230318525001729</td>\n",
              "      <td>positive</td>\n",
              "      <td>@DariusButler28   Have a great game agaist Tam...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9683</th>\n",
              "      <td>100461938533863424</td>\n",
              "      <td>negative</td>\n",
              "      <td>I'm pisseeedddd that I missed Kid Cudi's show ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>9684 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-815fff57-9d58-4d39-8c1f-4a52015a44d2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-815fff57-9d58-4d39-8c1f-4a52015a44d2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-815fff57-9d58-4d39-8c1f-4a52015a44d2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Outra abordagem se trata de ler diretamente os arquivos diretamente dos repositórios do Github. Lembre-se que colocar a URL do RAW file do dataset:"
      ],
      "metadata": {
        "id": "FnUymVVrUXam"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv('https://raw.githubusercontent.com/viniciusrpb/cic0269_natural_language_processing/main/corpus_tweets/twitter-2013train-A.txt',sep='\\t',encoding='utf-8',names=['id','polarity','text'])"
      ],
      "metadata": {
        "id": "JAo8bCrZGGSd"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "TBVepwWCGT3a",
        "outputId": "d0041d66-b8ad-45c4-e112-d71e895d85bb"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                      id  polarity  \\\n",
              "0     264183816548130816  positive   \n",
              "1     263405084770172928  negative   \n",
              "2     262163168678248449  negative   \n",
              "3     264249301910310912  negative   \n",
              "4     262682041215234048   neutral   \n",
              "...                  ...       ...   \n",
              "9679  103158179306807296  positive   \n",
              "9680  103157324096618497  positive   \n",
              "9681  100259220338905089   neutral   \n",
              "9682  104230318525001729  positive   \n",
              "9683  100461938533863424  negative   \n",
              "\n",
              "                                                   text  \n",
              "0     Gas by my house hit $3.39!!!! I\\u2019m going t...  \n",
              "1     Theo Walcott is still shit\\u002c watch Rafa an...  \n",
              "2     its not that I\\u2019m a GSP fan\\u002c i just h...  \n",
              "3     Iranian general says Israel\\u2019s Iron Dome c...  \n",
              "4     Tehran\\u002c Mon Amour: Obama Tried to Establi...  \n",
              "...                                                 ...  \n",
              "9679  RT @MNFootNg It's monday and Monday Night Foot...  \n",
              "9680  All I know is the road for that Lomardi start ...  \n",
              "9681  All Blue and White fam, we r meeting at Golden...  \n",
              "9682  @DariusButler28   Have a great game agaist Tam...  \n",
              "9683  I'm pisseeedddd that I missed Kid Cudi's show ...  \n",
              "\n",
              "[9684 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d5a986af-b02f-4a83-927b-54d8b38823b0\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>polarity</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>264183816548130816</td>\n",
              "      <td>positive</td>\n",
              "      <td>Gas by my house hit $3.39!!!! I\\u2019m going t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>263405084770172928</td>\n",
              "      <td>negative</td>\n",
              "      <td>Theo Walcott is still shit\\u002c watch Rafa an...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>262163168678248449</td>\n",
              "      <td>negative</td>\n",
              "      <td>its not that I\\u2019m a GSP fan\\u002c i just h...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>264249301910310912</td>\n",
              "      <td>negative</td>\n",
              "      <td>Iranian general says Israel\\u2019s Iron Dome c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>262682041215234048</td>\n",
              "      <td>neutral</td>\n",
              "      <td>Tehran\\u002c Mon Amour: Obama Tried to Establi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9679</th>\n",
              "      <td>103158179306807296</td>\n",
              "      <td>positive</td>\n",
              "      <td>RT @MNFootNg It's monday and Monday Night Foot...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9680</th>\n",
              "      <td>103157324096618497</td>\n",
              "      <td>positive</td>\n",
              "      <td>All I know is the road for that Lomardi start ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9681</th>\n",
              "      <td>100259220338905089</td>\n",
              "      <td>neutral</td>\n",
              "      <td>All Blue and White fam, we r meeting at Golden...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9682</th>\n",
              "      <td>104230318525001729</td>\n",
              "      <td>positive</td>\n",
              "      <td>@DariusButler28   Have a great game agaist Tam...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9683</th>\n",
              "      <td>100461938533863424</td>\n",
              "      <td>negative</td>\n",
              "      <td>I'm pisseeedddd that I missed Kid Cudi's show ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>9684 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d5a986af-b02f-4a83-927b-54d8b38823b0')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d5a986af-b02f-4a83-927b-54d8b38823b0 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d5a986af-b02f-4a83-927b-54d8b38823b0');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Como estamos interessados em analisar apenas os textos, vamos selecionar apenas a coluna 'text':"
      ],
      "metadata": {
        "id": "v4j_LHMgUpYF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_df = df['text'] "
      ],
      "metadata": {
        "id": "sPa2v3C6GifE"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ftmFICd0GqOM",
        "outputId": "dbd8b3ad-8881-4466-fbba-0b8ff6710f89"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0       Gas by my house hit $3.39!!!! I\\u2019m going t...\n",
              "1       Theo Walcott is still shit\\u002c watch Rafa an...\n",
              "2       its not that I\\u2019m a GSP fan\\u002c i just h...\n",
              "3       Iranian general says Israel\\u2019s Iron Dome c...\n",
              "4       Tehran\\u002c Mon Amour: Obama Tried to Establi...\n",
              "                              ...                        \n",
              "9679    RT @MNFootNg It's monday and Monday Night Foot...\n",
              "9680    All I know is the road for that Lomardi start ...\n",
              "9681    All Blue and White fam, we r meeting at Golden...\n",
              "9682    @DariusButler28   Have a great game agaist Tam...\n",
              "9683    I'm pisseeedddd that I missed Kid Cudi's show ...\n",
              "Name: text, Length: 9684, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Para pegar instâncias específicas ou colunas específicas do DataFrame, pode-se utilizar o iloc https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.iloc.html"
      ],
      "metadata": {
        "id": "CZgTmzJ-HkZ-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = text_df[0]\n",
        "sentence = sentence+\" \"+text_df[1]"
      ],
      "metadata": {
        "id": "JS6p16ZSHNOZ"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "OqIPftEQHVea",
        "outputId": "6d3e5aa3-4ab5-428d-fe4e-e1a575a371e2"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Gas by my house hit $3.39!!!! I\\\\u2019m going to Chapel Hill on Sat. :) Theo Walcott is still shit\\\\u002c watch Rafa and Johnny deal with him on Saturday.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Tokenização: quebrar uma sentença em palavras"
      ],
      "metadata": {
        "id": "AaQhbflHHzgi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = sentence.lower()"
      ],
      "metadata": {
        "id": "jzTaod6VH3KB"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "z6eMO9N7H7t-",
        "outputId": "980a61cf-d795-4e65-92b3-8d395d148261"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'gas by my house hit $3.39!!!! i\\\\u2019m going to chapel hill on sat. :) theo walcott is still shit\\\\u002c watch rafa and johnny deal with him on saturday.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Podemos também utilizar expressões regulares em Python para quebrar a sentença conforme alguns padrões estejam presentes.\n",
        "\n",
        "Na regra abaixo, o regex indica que na ocorrência de símbolos de pontuação constantemente presentes em tweets, como hashtags, @s, pontos de exclamação, vírgulas e outros, sejam considerados na quebra da sentença em palavras, mas que não sejam considerados como palavras individuais:"
      ],
      "metadata": {
        "id": "5DltTWUgU1RU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokens = re.split(r'[\\s!$@#,.;]+',sentence)"
      ],
      "metadata": {
        "id": "K30NgXTPJPdY"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokens"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KrqAhNzfI55G",
        "outputId": "659fd605-92fa-408c-942e-a9c852c24deb"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['gas',\n",
              " 'by',\n",
              " 'my',\n",
              " 'house',\n",
              " 'hit',\n",
              " '3',\n",
              " '39',\n",
              " 'i\\\\u2019m',\n",
              " 'going',\n",
              " 'to',\n",
              " 'chapel',\n",
              " 'hill',\n",
              " 'on',\n",
              " 'sat',\n",
              " ':)',\n",
              " 'theo',\n",
              " 'walcott',\n",
              " 'is',\n",
              " 'still',\n",
              " 'shit\\\\u002c',\n",
              " 'watch',\n",
              " 'rafa',\n",
              " 'and',\n",
              " 'johnny',\n",
              " 'deal',\n",
              " 'with',\n",
              " 'him',\n",
              " 'on',\n",
              " 'saturday',\n",
              " '']"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Remoção de stopwords"
      ],
      "metadata": {
        "id": "56_lFbRwKd7t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "stop_list = ['and','or','the','is','on','in']"
      ],
      "metadata": {
        "id": "a_tMTdDpKh6P"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "novel_tokens = []\n",
        "for t in tokens:\n",
        "    if t not in stop_list:\n",
        "        novel_tokens.append(t)"
      ],
      "metadata": {
        "id": "BdGxX-7KKvTM"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "novel_tokens"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ez7pi4CJLIgf",
        "outputId": "76985d9d-fbb6-4dbf-9ed5-a4d1e5790635"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['gas',\n",
              " 'by',\n",
              " 'my',\n",
              " 'house',\n",
              " 'hit',\n",
              " '3',\n",
              " '39',\n",
              " 'i\\\\u2019m',\n",
              " 'going',\n",
              " 'to',\n",
              " 'chapel',\n",
              " 'hill',\n",
              " 'sat',\n",
              " ':)',\n",
              " 'theo',\n",
              " 'walcott',\n",
              " 'still',\n",
              " 'shit\\\\u002c',\n",
              " 'watch',\n",
              " 'rafa',\n",
              " 'johnny',\n",
              " 'deal',\n",
              " 'with',\n",
              " 'him',\n",
              " 'saturday',\n",
              " '']"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "stop_list_nltk = nltk.corpus.stopwords.words('english')"
      ],
      "metadata": {
        "id": "Vt681aeKLq4D"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "stop_list_nltk"
      ],
      "metadata": {
        "id": "pFl8nYcnMTEF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f42fa110-59e6-4ee1-ccf6-cec8ad669cfe"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['i',\n",
              " 'me',\n",
              " 'my',\n",
              " 'myself',\n",
              " 'we',\n",
              " 'our',\n",
              " 'ours',\n",
              " 'ourselves',\n",
              " 'you',\n",
              " \"you're\",\n",
              " \"you've\",\n",
              " \"you'll\",\n",
              " \"you'd\",\n",
              " 'your',\n",
              " 'yours',\n",
              " 'yourself',\n",
              " 'yourselves',\n",
              " 'he',\n",
              " 'him',\n",
              " 'his',\n",
              " 'himself',\n",
              " 'she',\n",
              " \"she's\",\n",
              " 'her',\n",
              " 'hers',\n",
              " 'herself',\n",
              " 'it',\n",
              " \"it's\",\n",
              " 'its',\n",
              " 'itself',\n",
              " 'they',\n",
              " 'them',\n",
              " 'their',\n",
              " 'theirs',\n",
              " 'themselves',\n",
              " 'what',\n",
              " 'which',\n",
              " 'who',\n",
              " 'whom',\n",
              " 'this',\n",
              " 'that',\n",
              " \"that'll\",\n",
              " 'these',\n",
              " 'those',\n",
              " 'am',\n",
              " 'is',\n",
              " 'are',\n",
              " 'was',\n",
              " 'were',\n",
              " 'be',\n",
              " 'been',\n",
              " 'being',\n",
              " 'have',\n",
              " 'has',\n",
              " 'had',\n",
              " 'having',\n",
              " 'do',\n",
              " 'does',\n",
              " 'did',\n",
              " 'doing',\n",
              " 'a',\n",
              " 'an',\n",
              " 'the',\n",
              " 'and',\n",
              " 'but',\n",
              " 'if',\n",
              " 'or',\n",
              " 'because',\n",
              " 'as',\n",
              " 'until',\n",
              " 'while',\n",
              " 'of',\n",
              " 'at',\n",
              " 'by',\n",
              " 'for',\n",
              " 'with',\n",
              " 'about',\n",
              " 'against',\n",
              " 'between',\n",
              " 'into',\n",
              " 'through',\n",
              " 'during',\n",
              " 'before',\n",
              " 'after',\n",
              " 'above',\n",
              " 'below',\n",
              " 'to',\n",
              " 'from',\n",
              " 'up',\n",
              " 'down',\n",
              " 'in',\n",
              " 'out',\n",
              " 'on',\n",
              " 'off',\n",
              " 'over',\n",
              " 'under',\n",
              " 'again',\n",
              " 'further',\n",
              " 'then',\n",
              " 'once',\n",
              " 'here',\n",
              " 'there',\n",
              " 'when',\n",
              " 'where',\n",
              " 'why',\n",
              " 'how',\n",
              " 'all',\n",
              " 'any',\n",
              " 'both',\n",
              " 'each',\n",
              " 'few',\n",
              " 'more',\n",
              " 'most',\n",
              " 'other',\n",
              " 'some',\n",
              " 'such',\n",
              " 'no',\n",
              " 'nor',\n",
              " 'not',\n",
              " 'only',\n",
              " 'own',\n",
              " 'same',\n",
              " 'so',\n",
              " 'than',\n",
              " 'too',\n",
              " 'very',\n",
              " 's',\n",
              " 't',\n",
              " 'can',\n",
              " 'will',\n",
              " 'just',\n",
              " 'don',\n",
              " \"don't\",\n",
              " 'should',\n",
              " \"should've\",\n",
              " 'now',\n",
              " 'd',\n",
              " 'll',\n",
              " 'm',\n",
              " 'o',\n",
              " 're',\n",
              " 've',\n",
              " 'y',\n",
              " 'ain',\n",
              " 'aren',\n",
              " \"aren't\",\n",
              " 'couldn',\n",
              " \"couldn't\",\n",
              " 'didn',\n",
              " \"didn't\",\n",
              " 'doesn',\n",
              " \"doesn't\",\n",
              " 'hadn',\n",
              " \"hadn't\",\n",
              " 'hasn',\n",
              " \"hasn't\",\n",
              " 'haven',\n",
              " \"haven't\",\n",
              " 'isn',\n",
              " \"isn't\",\n",
              " 'ma',\n",
              " 'mightn',\n",
              " \"mightn't\",\n",
              " 'mustn',\n",
              " \"mustn't\",\n",
              " 'needn',\n",
              " \"needn't\",\n",
              " 'shan',\n",
              " \"shan't\",\n",
              " 'shouldn',\n",
              " \"shouldn't\",\n",
              " 'wasn',\n",
              " \"wasn't\",\n",
              " 'weren',\n",
              " \"weren't\",\n",
              " 'won',\n",
              " \"won't\",\n",
              " 'wouldn',\n",
              " \"wouldn't\"]"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Stemming"
      ],
      "metadata": {
        "id": "H7ZMTjMr19Mk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "stemmer = PorterStemmer()"
      ],
      "metadata": {
        "id": "9OZnm1qRMez8"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokens_final = [stemmer.stem(t) for t in novel_tokens]"
      ],
      "metadata": {
        "id": "8U_43tNbONdj"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokens_final"
      ],
      "metadata": {
        "id": "PtMgwt2LOrAc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dc126061-652b-4f9f-c37a-5a761510ca4a"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['ga',\n",
              " 'by',\n",
              " 'my',\n",
              " 'hous',\n",
              " 'hit',\n",
              " '3',\n",
              " '39',\n",
              " 'i\\\\u2019m',\n",
              " 'go',\n",
              " 'to',\n",
              " 'chapel',\n",
              " 'hill',\n",
              " 'sat',\n",
              " ':)',\n",
              " 'theo',\n",
              " 'walcott',\n",
              " 'still',\n",
              " 'shit\\\\u002c',\n",
              " 'watch',\n",
              " 'rafa',\n",
              " 'johnni',\n",
              " 'deal',\n",
              " 'with',\n",
              " 'him',\n",
              " 'saturday',\n",
              " '']"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "novel_tokens"
      ],
      "metadata": {
        "id": "E9xSFAwyOzeV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ef1e97ff-7d95-4c60-8870-9059f8021426"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['gas',\n",
              " 'by',\n",
              " 'my',\n",
              " 'house',\n",
              " 'hit',\n",
              " '3',\n",
              " '39',\n",
              " 'i\\\\u2019m',\n",
              " 'going',\n",
              " 'to',\n",
              " 'chapel',\n",
              " 'hill',\n",
              " 'sat',\n",
              " ':)',\n",
              " 'theo',\n",
              " 'walcott',\n",
              " 'still',\n",
              " 'shit\\\\u002c',\n",
              " 'watch',\n",
              " 'rafa',\n",
              " 'johnny',\n",
              " 'deal',\n",
              " 'with',\n",
              " 'him',\n",
              " 'saturday',\n",
              " '']"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "striing = \"Ola<p> eu sou </p>\"\n",
        "\n",
        "token = re.sub(r'<p>','',striing)\n",
        "print(token)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mo1WC7sdStv9",
        "outputId": "1032f970-5c4e-4527-8af4-5f87e866f668"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Ola eu sou </p>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "mapping = {}\n",
        "mapping[':)'] = 'happy'"
      ],
      "metadata": {
        "id": "Zyu0F6-9S7DP"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Lemmatization\n",
        "\n"
      ],
      "metadata": {
        "id": "2bxD-eK80JQ5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "print(lemmatizer.lemmatize(\"cheers\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FEF7NB2n12oR",
        "outputId": "57fee984-66b6-4f1e-f315-757ff9fbda12"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "cheer\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokens_final = [lemmatizer.lemmatize(w) for w in novel_tokens]\n",
        "print(tokens_final)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8--rLziw4Hkg",
        "outputId": "f5adbe5b-1439-41fe-b47e-22458113a0dc"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['gas', 'by', 'my', 'house', 'hit', '3', '39', 'i\\\\u2019m', 'going', 'to', 'chapel', 'hill', 'sat', ':)', 'theo', 'walcott', 'still', 'shit\\\\u002c', 'watch', 'rafa', 'johnny', 'deal', 'with', 'him', 'saturday', '']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tweet_final = ' '.join([lemmatizer.lemmatize(w) for w in novel_tokens])\n",
        "print(tweet_final)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KushNul74TKI",
        "outputId": "a6a911ed-dd39-43fb-bba5-2ba75fb22af2"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gas by my house hit 3 39 i\\u2019m going to chapel hill sat :) theo walcott still shit\\u002c watch rafa johnny deal with him saturday \n"
          ]
        }
      ]
    }
  ]
}