Sentiment_Analysis_CNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Sentiment_Analysis_CNN.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "6MJ0omqaHyhC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Sentiment Analysis using a Convolutional Neural Network (CNN)\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "CJR7t6vdOAr1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Sentiment Analysis, or opinion mining, is the process of determining whether language reflects positive, negative, or neutral sentiment. Using sentiment algorithms, developers and brand managers can gain insights into customer opinions about a topic. Sentiment analysis is probably one the most common applications in Natural Language processing (NLP).\n",
        "\n",
        "Traditional approaches frequently use the Bag Of Words (BOW) model, where a document is mapped to a feature vector, and then classified by machine learning techniques. Although the BOW approach is simple and quite efficient, a great deal of the information from the original natural language is lost (Xia & Zong, 2010), e.g., word order is disrupted and syntactic structures are broken. Therefore, various types of features have been exploited, such as higher order n-grams (Pak & Paroubek, 2010). Another kind of feature that can be used is Part Of Speech (POS) tagging, which is commonly used during a syntactic analysis process, as described in Gimpel et al. (2011). \n",
        "\n",
        "In this notebook, I will use a different technique to detect sentiment analysis. Using a deep learning model, I will train a classifier movie reviews on IMDB data set, using Convolutional Neural Networks (CNN)."
      ]
    },
    {
      "metadata": {
        "id": "BISE5LmCUQht",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Installing and Importing the required libraries"
      ]
    },
    {
      "metadata": {
        "id": "50xDE-iVyTVF",
        "colab_type": "code",
        "outputId": "0a798eaf-dcdd-45fc-ff49-1391a8aab6a8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190
        }
      },
      "cell_type": "code",
      "source": [
        "!python -m spacy download en\n",
        "!pip install torch\n",
        "import spacy\n",
        "spacy.load('en')\n",
        "import torch\n",
        "from torchtext import data\n",
        "from torchtext import datasets\n",
        "import random"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0 in /usr/local/lib/python3.6/dist-packages (2.0.0)\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/lib/python3.6/dist-packages/en_core_web_sm -->\n",
            "    /usr/local/lib/python3.6/dist-packages/spacy/data/en\n",
            "\n",
            "    You can now load the model via spacy.load('en')\n",
            "\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.0.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "q5P7SiYnVMOc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " we'll prepare the data:"
      ]
    },
    {
      "metadata": {
        "id": "dGyX2GerVFfL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "SEED = 1234\n",
        "\n",
        "torch.manual_seed(SEED)\n",
        "torch.cuda.manual_seed(SEED)\n",
        "torch.backends.cudnn.deterministic = True\n",
        "\n",
        "TEXT = data.Field(tokenize='spacy')\n",
        "LABEL = data.LabelField(dtype=torch.float)\n",
        "\n",
        "train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)\n",
        "\n",
        "train_data, valid_data = train_data.split(random_state=random.seed(SEED))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3xAeQqp2U4aI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Build the vocab and load the pre-trained word embeddings:\n"
      ]
    },
    {
      "metadata": {
        "id": "R4zo6LuAzpkC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "TEXT.build_vocab(train_data, max_size=25000, vectors=\"glove.6B.100d\")\n",
        "LABEL.build_vocab(train_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ts1ykqShVW-G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We create the iterators:"
      ]
    },
    {
      "metadata": {
        "id": "HKbzong3ztD1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "BATCH_SIZE = 64\n",
        "\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
        "    (train_data, valid_data, test_data), \n",
        "    batch_size=BATCH_SIZE, \n",
        "    device=device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kAojK6nPVw6-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "# Build the Model\n",
        "\n",
        "Now to build our model:"
      ]
    },
    {
      "metadata": {
        "id": "uKB3rQvlzzRV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "class CNN(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, dropout):\n",
        "        super().__init__()\n",
        "        \n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
        "        self.conv_0 = nn.Conv2d(in_channels=1, out_channels=n_filters, kernel_size=(filter_sizes[0],embedding_dim))\n",
        "        self.conv_1 = nn.Conv2d(in_channels=1, out_channels=n_filters, kernel_size=(filter_sizes[1],embedding_dim))\n",
        "        self.conv_2 = nn.Conv2d(in_channels=1, out_channels=n_filters, kernel_size=(filter_sizes[2],embedding_dim))\n",
        "        self.fc = nn.Linear(len(filter_sizes)*n_filters, output_dim)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        \n",
        "        #x = [sent len, batch size]\n",
        "        \n",
        "        x = x.permute(1, 0)\n",
        "                \n",
        "        #x = [batch size, sent len]\n",
        "        \n",
        "        embedded = self.embedding(x)\n",
        "                \n",
        "        #embedded = [batch size, sent len, emb dim]\n",
        "        \n",
        "        embedded = embedded.unsqueeze(1)\n",
        "        \n",
        "        #embedded = [batch size, 1, sent len, emb dim]\n",
        "        \n",
        "        conved_0 = F.relu(self.conv_0(embedded).squeeze(3))\n",
        "        conved_1 = F.relu(self.conv_1(embedded).squeeze(3))\n",
        "        conved_2 = F.relu(self.conv_2(embedded).squeeze(3))\n",
        "            \n",
        "        #conv_n = [batch size, n_filters, sent len - filter_sizes[n]]\n",
        "        \n",
        "        pooled_0 = F.max_pool1d(conved_0, conved_0.shape[2]).squeeze(2)\n",
        "        pooled_1 = F.max_pool1d(conved_1, conved_1.shape[2]).squeeze(2)\n",
        "        pooled_2 = F.max_pool1d(conved_2, conved_2.shape[2]).squeeze(2)\n",
        "        \n",
        "        #pooled_n = [batch size, n_filters]\n",
        "        \n",
        "        cat = self.dropout(torch.cat((pooled_0, pooled_1, pooled_2), dim=1))\n",
        "\n",
        "        #cat = [batch size, n_filters * len(filter_sizes)]\n",
        "            \n",
        "        return self.fc(cat)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RZ89V917z4KC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class CNN(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, dropout):\n",
        "        super().__init__()\n",
        "        \n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
        "        self.convs = nn.ModuleList([nn.Conv2d(in_channels=1, out_channels=n_filters, kernel_size=(fs,embedding_dim)) for fs in filter_sizes])\n",
        "        self.fc = nn.Linear(len(filter_sizes)*n_filters, output_dim)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        \n",
        "        #x = [sent len, batch size]\n",
        "        \n",
        "        x = x.permute(1, 0)\n",
        "                \n",
        "        #x = [batch size, sent len]\n",
        "        \n",
        "        embedded = self.embedding(x)\n",
        "                \n",
        "        #embedded = [batch size, sent len, emb dim]\n",
        "        \n",
        "        embedded = embedded.unsqueeze(1)\n",
        "        \n",
        "        #embedded = [batch size, 1, sent len, emb dim]\n",
        "        \n",
        "        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]\n",
        "            \n",
        "        #conv_n = [batch size, n_filters, sent len - filter_sizes[n]]\n",
        "        \n",
        "        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]\n",
        "        \n",
        "        #pooled_n = [batch size, n_filters]\n",
        "        \n",
        "        cat = self.dropout(torch.cat(pooled, dim=1))\n",
        "\n",
        "        #cat = [batch size, n_filters * len(filter_sizes)]\n",
        "            \n",
        "        return self.fc(cat)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Rpnx-zGrV3FA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We create an instance of our CNN class:"
      ]
    },
    {
      "metadata": {
        "id": "wwV0Wezf0EJ4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = 100\n",
        "N_FILTERS = 100\n",
        "FILTER_SIZES = [3,4,5]\n",
        "OUTPUT_DIM = 1\n",
        "DROPOUT = 0.5\n",
        "\n",
        "model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5wLh-Lu3V-zK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "And load the pre-trained embeddings:\n"
      ]
    },
    {
      "metadata": {
        "id": "sXqJV5o10H7R",
        "colab_type": "code",
        "outputId": "ddd16798-cbe3-43aa-88e4-7da2c3cae4ea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "cell_type": "code",
      "source": [
        "pretrained_embeddings = TEXT.vocab.vectors\n",
        "\n",
        "model.embedding.weight.data.copy_(pretrained_embeddings)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
              "        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
              "        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],\n",
              "        ...,\n",
              "        [-0.2932, -0.5023,  0.6352,  ..., -0.3866,  0.6527,  0.2257],\n",
              "        [-0.3827,  0.0350,  0.0140,  ..., -0.1223, -0.5914,  0.0914],\n",
              "        [ 0.0540, -0.2875,  0.0737,  ..., -0.2098,  0.2665,  0.2918]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "metadata": {
        "id": "a_zrysnhWEy9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Train the Model\n",
        "\n",
        "We initialize the optimizer, loss function (criterion) and place the model and criterion on the GPU (if available):\n"
      ]
    },
    {
      "metadata": {
        "id": "NXdi1Kxi0Kpq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch.optim as optim\n",
        "\n",
        "optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "criterion = nn.BCEWithLogitsLoss()\n",
        "\n",
        "model = model.to(device)\n",
        "criterion = criterion.to(device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "uTR3m6-wWNfg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We implement the function to calculate accuracy:\n"
      ]
    },
    {
      "metadata": {
        "id": "aJ_vc96A0Oja",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def binary_accuracy(preds, y):\n",
        "    \"\"\"\n",
        "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
        "    \"\"\"\n",
        "\n",
        "    #round predictions to the closest integer\n",
        "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
        "    correct = (rounded_preds == y).float() #convert into float for division \n",
        "    acc = correct.sum()/len(correct)\n",
        "    return acc"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FhofJfxpWTkZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We define a function for training our model. **Note: **as we are using dropout, we must remember to use model.train() to ensure the dropout is \"turned on\" while training.\n"
      ]
    },
    {
      "metadata": {
        "id": "hrq3l6A70RXG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def train(model, iterator, optimizer, criterion):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.train()\n",
        "    \n",
        "    for batch in iterator:\n",
        "        \n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        predictions = model(batch.text).squeeze(1)\n",
        "        \n",
        "        loss = criterion(predictions, batch.label)\n",
        "        \n",
        "        acc = binary_accuracy(predictions, batch.label)\n",
        "        \n",
        "        loss.backward()\n",
        "        \n",
        "        optimizer.step()\n",
        "        \n",
        "        epoch_loss += loss.item()\n",
        "        epoch_acc += acc.item()\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9r-KkF7UWdFG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We define a function for testing our model. **Note: **, as we are now using dropout, we must remember to use model.eval() to ensure the dropout is \"turned off\" while evaluating.\n"
      ]
    },
    {
      "metadata": {
        "id": "P2GEnV9V0VSj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def evaluate(model, iterator, criterion):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.eval()\n",
        "    \n",
        "    with torch.no_grad():\n",
        "    \n",
        "        for batch in iterator:\n",
        "\n",
        "            predictions = model(batch.text).squeeze(1)\n",
        "            \n",
        "            loss = criterion(predictions, batch.label)\n",
        "            \n",
        "            acc = binary_accuracy(predictions, batch.label)\n",
        "\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_acc += acc.item()\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PJ1f0ZunWlIW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally, we train our model:\n"
      ]
    },
    {
      "metadata": {
        "id": "bLvABLF40Y-Z",
        "colab_type": "code",
        "outputId": "d4af8571-cd66-4228-e68c-73008ce88d61",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "N_EPOCHS = 5\n",
        "\n",
        "for epoch in range(N_EPOCHS):\n",
        "\n",
        "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
        "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
        "    \n",
        "    print(f'| Epoch: {epoch+1:02} | Train Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}% | Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:.2f}% |')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "| Epoch: 01 | Train Loss: 0.495 | Train Acc: 74.73% | Val. Loss: 0.340 | Val. Acc: 85.51% |\n",
            "| Epoch: 02 | Train Loss: 0.305 | Train Acc: 86.89% | Val. Loss: 0.287 | Val. Acc: 88.24% |\n",
            "| Epoch: 03 | Train Loss: 0.223 | Train Acc: 91.11% | Val. Loss: 0.279 | Val. Acc: 88.53% |\n",
            "| Epoch: 04 | Train Loss: 0.151 | Train Acc: 94.40% | Val. Loss: 0.277 | Val. Acc: 89.07% |\n",
            "| Epoch: 05 | Train Loss: 0.090 | Train Acc: 97.21% | Val. Loss: 0.296 | Val. Acc: 88.99% |\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "fFVFUDs4WtYN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We got a good accuracy: 88.99%"
      ]
    },
    {
      "metadata": {
        "id": "8BETrbio0dKz",
        "colab_type": "code",
        "outputId": "77346bf1-bf42-427b-f1ea-4ee2e80023ff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
        "\n",
        "print(f'| Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}% |')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "| Test Loss: 0.311 | Test Acc: 88.16% |\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EHKKvL9JW2c8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# User Input\n",
        "\n",
        "As a sanity check we can check some input sentences\n",
        "\n",
        "Note: As mentioned in the implementation details, the input sentence has to be at least as long as the largest filter height used. We modify our predict_sentiment function to also accept a minimum length argument. If the tokenized input sentence is less than min_len tokens, we append padding tokens (<pad>) to make it min_len tokens.\n"
      ]
    },
    {
      "metadata": {
        "id": "CKXXszH70dOP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import spacy\n",
        "nlp = spacy.load('en')\n",
        "\n",
        "def predict_sentiment(sentence, min_len=5):\n",
        "    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]\n",
        "    if len(tokenized) < min_len:\n",
        "        tokenized += ['<pad>'] * (min_len - len(tokenized))\n",
        "    indexed = [TEXT.vocab.stoi[t] for t in tokenized]\n",
        "    tensor = torch.LongTensor(indexed).to(device)\n",
        "    tensor = tensor.unsqueeze(1)\n",
        "    prediction = torch.sigmoid(model(tensor))\n",
        "    return prediction.item()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ABgWUGTtW-fp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "An example negative review:"
      ]
    },
    {
      "metadata": {
        "id": "Qi1_jKZy0dSU",
        "colab_type": "code",
        "outputId": "4a55f2c0-ed7b-4b56-bd80-6b0e5b80da9c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "predict_sentiment(\"This film is boring\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.050386134535074234"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "metadata": {
        "id": "MfiwqRbEXCOc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "An example positive review:"
      ]
    },
    {
      "metadata": {
        "id": "rcIyPYGJ0nM-",
        "colab_type": "code",
        "outputId": "00563949-a30e-478d-ab39-cc2150c92e5b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "predict_sentiment(\"This film is great\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9362199902534485"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    }
  ]
}