Sentiment_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment analysis of YELP reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "tqdm.pandas(desc=\"progress-bar\")"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "This data was obtained from the kaggle competition: https://www.kaggle.com/c/sentclf\n",
    "Since it was private competition, we are using the train data only and then train-test split it\n",
    "All rights are own by YELP reviews dataset.\n",
    "Please refer acknowledgement of the kaggle page to more details on restrictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DATA PREPROCESSING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Stemming "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('/Users/ishansharma/Downloads/yelp_review_full_csv/train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>5</th>\n",
       "      <th>dr. goldberg offers everything i look for in a general practitioner.  he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first.  really, what more do you need?  i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Unfortunately, the frustration of being Dr. Go...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Been going to Dr. Goldberg for over 10 years. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>Got a letter in the mail last week that said D...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>I don't know what Dr. Goldberg was like before...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Top notch doctor in a top notch practice. Can'...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   5  \\\n",
       "0  2   \n",
       "1  4   \n",
       "2  4   \n",
       "3  1   \n",
       "4  5   \n",
       "\n",
       "  dr. goldberg offers everything i look for in a general practitioner.  he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first.  really, what more do you need?  i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.  \n",
       "0  Unfortunately, the frustration of being Dr. Go...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "1  Been going to Dr. Goldberg for over 10 years. ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "2  Got a letter in the mail last week that said D...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "3  I don't know what Dr. Goldberg was like before...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "4  Top notch doctor in a top notch practice. Can'...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 649999 entries, 0 to 649998\n",
      "Data columns (total 2 columns):\n",
      " #   Column                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Non-Null Count   Dtype \n",
      "---  ------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  --------------   ----- \n",
      " 0   5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       649999 non-null  int64 \n",
      " 1   dr. goldberg offers everything i look for in a general practitioner.  he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first.  really, what more do you need?  i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.  649999 non-null  object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 9.9+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "import string\n",
    "from nltk.corpus import stopwords\n",
    "from unidecode import unidecode\n",
    "from langdetect import detect\n",
    "from nltk.stem import SnowballStemmer\n",
    "from googletrans import Translator"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "To download stopwords:\n",
    "Go to command prompt or terminal and run the below commands\n",
    "->import nltk \n",
    "->nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading collection 'all'\n",
      "[nltk_data]    | \n",
      "[nltk_data]    | Downloading package abc to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package abc is already up-to-date!\n",
      "[nltk_data]    | Downloading package alpino to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package alpino is already up-to-date!\n",
      "[nltk_data]    | Downloading package biocreative_ppi to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package biocreative_ppi is already up-to-date!\n",
      "[nltk_data]    | Downloading package brown to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package brown is already up-to-date!\n",
      "[nltk_data]    | Downloading package brown_tei to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package brown_tei is already up-to-date!\n",
      "[nltk_data]    | Downloading package cess_cat to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package cess_cat is already up-to-date!\n",
      "[nltk_data]    | Downloading package cess_esp to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package cess_esp is already up-to-date!\n",
      "[nltk_data]    | Downloading package chat80 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package chat80 is already up-to-date!\n",
      "[nltk_data]    | Downloading package city_database to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package city_database is already up-to-date!\n",
      "[nltk_data]    | Downloading package cmudict to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package cmudict is already up-to-date!\n",
      "[nltk_data]    | Downloading package comparative_sentences to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package comparative_sentences is already up-to-\n",
      "[nltk_data]    |       date!\n",
      "[nltk_data]    | Downloading package comtrans to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package comtrans is already up-to-date!\n",
      "[nltk_data]    | Downloading package conll2000 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package conll2000 is already up-to-date!\n",
      "[nltk_data]    | Downloading package conll2002 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package conll2002 is already up-to-date!\n",
      "[nltk_data]    | Downloading package conll2007 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package conll2007 is already up-to-date!\n",
      "[nltk_data]    | Downloading package crubadan to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package crubadan is already up-to-date!\n",
      "[nltk_data]    | Downloading package dependency_treebank to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package dependency_treebank is already up-to-date!\n",
      "[nltk_data]    | Downloading package dolch to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package dolch is already up-to-date!\n",
      "[nltk_data]    | Downloading package europarl_raw to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package europarl_raw is already up-to-date!\n",
      "[nltk_data]    | Downloading package floresta to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package floresta is already up-to-date!\n",
      "[nltk_data]    | Downloading package framenet_v15 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package framenet_v15 is already up-to-date!\n",
      "[nltk_data]    | Downloading package framenet_v17 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package framenet_v17 is already up-to-date!\n",
      "[nltk_data]    | Downloading package gazetteers to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package gazetteers is already up-to-date!\n",
      "[nltk_data]    | Downloading package genesis to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package genesis is already up-to-date!\n",
      "[nltk_data]    | Downloading package gutenberg to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package gutenberg is already up-to-date!\n",
      "[nltk_data]    | Downloading package ieer to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package ieer is already up-to-date!\n",
      "[nltk_data]    | Downloading package inaugural to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package inaugural is already up-to-date!\n",
      "[nltk_data]    | Downloading package indian to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package indian is already up-to-date!\n",
      "[nltk_data]    | Downloading package jeita to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package jeita is already up-to-date!\n",
      "[nltk_data]    | Downloading package kimmo to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package kimmo is already up-to-date!\n",
      "[nltk_data]    | Downloading package knbc to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package knbc is already up-to-date!\n",
      "[nltk_data]    | Downloading package lin_thesaurus to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package lin_thesaurus is already up-to-date!\n",
      "[nltk_data]    | Downloading package mac_morpho to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package mac_morpho is already up-to-date!\n",
      "[nltk_data]    | Downloading package machado to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package machado is already up-to-date!\n",
      "[nltk_data]    | Downloading package masc_tagged to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package masc_tagged is already up-to-date!\n",
      "[nltk_data]    | Downloading package moses_sample to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package moses_sample is already up-to-date!\n",
      "[nltk_data]    | Downloading package movie_reviews to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package movie_reviews is already up-to-date!\n",
      "[nltk_data]    | Downloading package names to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package names is already up-to-date!\n",
      "[nltk_data]    | Downloading package nombank.1.0 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package nombank.1.0 is already up-to-date!\n",
      "[nltk_data]    | Downloading package nps_chat to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package nps_chat is already up-to-date!\n",
      "[nltk_data]    | Downloading package omw to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package omw is already up-to-date!\n",
      "[nltk_data]    | Downloading package opinion_lexicon to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package opinion_lexicon is already up-to-date!\n",
      "[nltk_data]    | Downloading package paradigms to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package paradigms is already up-to-date!\n",
      "[nltk_data]    | Downloading package pil to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package pil is already up-to-date!\n",
      "[nltk_data]    | Downloading package pl196x to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package pl196x is already up-to-date!\n",
      "[nltk_data]    | Downloading package ppattach to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package ppattach is already up-to-date!\n",
      "[nltk_data]    | Downloading package problem_reports to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package problem_reports is already up-to-date!\n",
      "[nltk_data]    | Downloading package propbank to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package propbank is already up-to-date!\n",
      "[nltk_data]    | Downloading package ptb to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package ptb is already up-to-date!\n",
      "[nltk_data]    | Downloading package product_reviews_1 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package product_reviews_1 is already up-to-date!\n",
      "[nltk_data]    | Downloading package product_reviews_2 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package product_reviews_2 is already up-to-date!\n",
      "[nltk_data]    | Downloading package pros_cons to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package pros_cons is already up-to-date!\n",
      "[nltk_data]    | Downloading package qc to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package qc is already up-to-date!\n",
      "[nltk_data]    | Downloading package reuters to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package reuters is already up-to-date!\n",
      "[nltk_data]    | Downloading package rte to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package rte is already up-to-date!\n",
      "[nltk_data]    | Downloading package semcor to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package semcor is already up-to-date!\n",
      "[nltk_data]    | Downloading package senseval to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package senseval is already up-to-date!\n",
      "[nltk_data]    | Downloading package sentiwordnet to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package sentiwordnet is already up-to-date!\n",
      "[nltk_data]    | Downloading package sentence_polarity to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package sentence_polarity is already up-to-date!\n",
      "[nltk_data]    | Downloading package shakespeare to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package shakespeare is already up-to-date!\n",
      "[nltk_data]    | Downloading package sinica_treebank to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package sinica_treebank is already up-to-date!\n",
      "[nltk_data]    | Downloading package smultron to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package smultron is already up-to-date!\n",
      "[nltk_data]    | Downloading package state_union to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package state_union is already up-to-date!\n",
      "[nltk_data]    | Downloading package stopwords to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package stopwords is already up-to-date!\n",
      "[nltk_data]    | Downloading package subjectivity to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package subjectivity is already up-to-date!\n",
      "[nltk_data]    | Downloading package swadesh to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package swadesh is already up-to-date!\n",
      "[nltk_data]    | Downloading package switchboard to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package switchboard is already up-to-date!\n",
      "[nltk_data]    | Downloading package timit to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package timit is already up-to-date!\n",
      "[nltk_data]    | Downloading package toolbox to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package toolbox is already up-to-date!\n",
      "[nltk_data]    | Downloading package treebank to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package treebank is already up-to-date!\n",
      "[nltk_data]    | Downloading package twitter_samples to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data]    |   Package twitter_samples is already up-to-date!\n",
      "[nltk_data]    | Downloading package udhr to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package udhr is already up-to-date!\n",
      "[nltk_data]    | Downloading package udhr2 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package udhr2 is already up-to-date!\n",
      "[nltk_data]    | Downloading package unicode_samples to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package unicode_samples is already up-to-date!\n",
      "[nltk_data]    | Downloading package universal_treebanks_v20 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package universal_treebanks_v20 is already up-to-\n",
      "[nltk_data]    |       date!\n",
      "[nltk_data]    | Downloading package verbnet to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package verbnet is already up-to-date!\n",
      "[nltk_data]    | Downloading package verbnet3 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package verbnet3 is already up-to-date!\n",
      "[nltk_data]    | Downloading package webtext to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package webtext is already up-to-date!\n",
      "[nltk_data]    | Downloading package wordnet to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package wordnet is already up-to-date!\n",
      "[nltk_data]    | Downloading package wordnet_ic to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package wordnet_ic is already up-to-date!\n",
      "[nltk_data]    | Downloading package words to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package words is already up-to-date!\n",
      "[nltk_data]    | Downloading package ycoe to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package ycoe is already up-to-date!\n",
      "[nltk_data]    | Downloading package rslp to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package rslp is already up-to-date!\n",
      "[nltk_data]    | Downloading package maxent_treebank_pos_tagger to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package maxent_treebank_pos_tagger is already up-\n",
      "[nltk_data]    |       to-date!\n",
      "[nltk_data]    | Downloading package universal_tagset to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package universal_tagset is already up-to-date!\n",
      "[nltk_data]    | Downloading package maxent_ne_chunker to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package maxent_ne_chunker is already up-to-date!\n",
      "[nltk_data]    | Downloading package punkt to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package punkt is already up-to-date!\n",
      "[nltk_data]    | Downloading package book_grammars to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package book_grammars is already up-to-date!\n",
      "[nltk_data]    | Downloading package sample_grammars to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package sample_grammars is already up-to-date!\n",
      "[nltk_data]    | Downloading package spanish_grammars to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package spanish_grammars is already up-to-date!\n",
      "[nltk_data]    | Downloading package basque_grammars to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package basque_grammars is already up-to-date!\n",
      "[nltk_data]    | Downloading package large_grammars to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package large_grammars is already up-to-date!\n",
      "[nltk_data]    | Downloading package tagsets to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package tagsets is already up-to-date!\n",
      "[nltk_data]    | Downloading package snowball_data to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package snowball_data is already up-to-date!\n",
      "[nltk_data]    | Downloading package bllip_wsj_no_aux to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Package bllip_wsj_no_aux is already up-to-date!\n",
      "[nltk_data]    | Downloading package word2vec_sample to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping models/word2vec_sample.zip.\n",
      "[nltk_data]    | Downloading package panlex_swadesh to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    | Downloading package mte_teip5 to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping corpora/mte_teip5.zip.\n",
      "[nltk_data]    | Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
      "[nltk_data]    | Downloading package averaged_perceptron_tagger_ru to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping\n",
      "[nltk_data]    |       taggers/averaged_perceptron_tagger_ru.zip.\n",
      "[nltk_data]    | Downloading package perluniprops to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping misc/perluniprops.zip.\n",
      "[nltk_data]    | Downloading package nonbreaking_prefixes to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping corpora/nonbreaking_prefixes.zip.\n",
      "[nltk_data]    | Downloading package vader_lexicon to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    | Downloading package porter_test to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping stemmers/porter_test.zip.\n",
      "[nltk_data]    | Downloading package wmt15_eval to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping models/wmt15_eval.zip.\n",
      "[nltk_data]    | Downloading package mwa_ppdb to\n",
      "[nltk_data]    |     /Users/ishansharma/nltk_data...\n",
      "[nltk_data]    |   Unzipping misc/mwa_ppdb.zip.\n",
      "[nltk_data]    | \n",
      "[nltk_data]  Done downloading collection all\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('all')\n",
    "stopwords = set(stopwords.words(\"english\"))\n",
    "exclude = set(string.punctuation)\n",
    "translate = Translator()\n",
    "sno = SnowballStemmer(\"english\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_words(sent):\n",
    "    sent = sent.lower()\n",
    "    temp = ''\n",
    "    for ch in sent:\n",
    "        if ch not in exclude:\n",
    "            temp +=ch\n",
    "        else:\n",
    "            temp+=''\n",
    "    arr = word_tokenize(temp)\n",
    "    sent = \"\"\n",
    "    for i in arr:\n",
    "        if i!='' and i!=' ' and (i not in stopwords):\n",
    "            sent +=i+' '\n",
    "    return sent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "inp = pd.Series([df.empty])\n",
    "inp_processed = []\n",
    "err = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/1 [00:00<?, ?it/s]\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "cannot use a string pattern on a bytes-like object",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-40-1b69b744b151>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0;31m#sent = unidecode(inp[i].decode('utf-8'))\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0msent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minp\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m     \u001b[0;32mif\u001b[0m \u001b[0mdetect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msent\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;34m'en'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m             \u001b[0msent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtranslate\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranslate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.8/site-packages/langdetect/detector_factory.py\u001b[0m in \u001b[0;36mdetect\u001b[0;34m(text)\u001b[0m\n\u001b[1;32m    127\u001b[0m     \u001b[0minit_factory\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    128\u001b[0m     \u001b[0mdetector\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_factory\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 129\u001b[0;31m     \u001b[0mdetector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    130\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mdetector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    131\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.8/site-packages/langdetect/detector.py\u001b[0m in \u001b[0;36mappend\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m    102\u001b[0m         \u001b[0mDetector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_max_text_length\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mrest\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mcut\u001b[0m \u001b[0mdown\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    103\u001b[0m         '''\n\u001b[0;32m--> 104\u001b[0;31m         \u001b[0mtext\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mURL_RE\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m' '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    105\u001b[0m         \u001b[0mtext\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mMAIL_RE\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m' '\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    106\u001b[0m         \u001b[0mtext\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNGram\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnormalize_vi\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: cannot use a string pattern on a bytes-like object"
     ]
    }
   ],
   "source": [
    "for i in tqdm(range(len(inp))):\n",
    "    #sent = unidecode(inp[i].decode('utf-8'))\n",
    "    sent = inp[i]\n",
    "    if detect(sent) != 'en':\n",
    "        try:\n",
    "            sent = translate.translate(sent).text\n",
    "        except:\n",
    "            err +=1\n",
    "    sent = get_words(sent)\n",
    "    stemmed = ''\n",
    "    for j in sent.split(\" \"):\n",
    "        stemmed+= sno.stem(j)+\" \"\n",
    "    df.loc[i,'text'] = stemmed.encode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"stemmed_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "from gensim.models import Word2Vec\n",
    "import multiprocessing\n",
    "cores = multiprocessing.cpu_count()\n",
    "from sklearn import utils\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"stemmed_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>5</th>\n",
       "      <th>dr. goldberg offers everything i look for in a general practitioner.  he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first.  really, what more do you need?  i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Unfortunately, the frustration of being Dr. Go...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>Been going to Dr. Goldberg for over 10 years. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>Got a letter in the mail last week that said D...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>I don't know what Dr. Goldberg was like before...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>Top notch doctor in a top notch practice. Can'...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  5  \\\n",
       "0           0  2   \n",
       "1           1  4   \n",
       "2           2  4   \n",
       "3           3  1   \n",
       "4           4  5   \n",
       "\n",
       "  dr. goldberg offers everything i look for in a general practitioner.  he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first.  really, what more do you need?  i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.  \n",
       "0  Unfortunately, the frustration of being Dr. Go...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "1  Been going to Dr. Goldberg for over 10 years. ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "2  Got a letter in the mail last week that said D...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "3  I don't know what Dr. Goldberg was like before...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
       "4  Top notch doctor in a top notch practice. Can'...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      "
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = pd.Series([df])\n",
    "y=[]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Found input variables with inconsistent numbers of samples: [1, 0]",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-104-e1d6dd4864e8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_selection\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mSEED\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m2000\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mx_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx_validation_and_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_validation_and_test\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m.02\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_state\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mSEED\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mx_validation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_validation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_validation_and_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_validation_and_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m.5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_state\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mSEED\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.8/site-packages/sklearn/model_selection/_split.py\u001b[0m in \u001b[0;36mtrain_test_split\u001b[0;34m(*arrays, **options)\u001b[0m\n\u001b[1;32m   2125\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Invalid parameters passed: %s\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2126\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2127\u001b[0;31m     \u001b[0marrays\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mindexable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0marrays\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2128\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2129\u001b[0m     \u001b[0mn_samples\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_num_samples\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marrays\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36mindexable\u001b[0;34m(*iterables)\u001b[0m\n\u001b[1;32m    290\u001b[0m     \"\"\"\n\u001b[1;32m    291\u001b[0m     \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0m_make_indexable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mX\u001b[0m \u001b[0;32min\u001b[0m \u001b[0miterables\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 292\u001b[0;31m     \u001b[0mcheck_consistent_length\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    293\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    294\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36mcheck_consistent_length\u001b[0;34m(*arrays)\u001b[0m\n\u001b[1;32m    253\u001b[0m     \u001b[0muniques\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munique\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlengths\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    254\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muniques\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 255\u001b[0;31m         raise ValueError(\"Found input variables with inconsistent numbers of\"\n\u001b[0m\u001b[1;32m    256\u001b[0m                          \" samples: %r\" % [int(l) for l in lengths])\n\u001b[1;32m    257\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Found input variables with inconsistent numbers of samples: [1, 0]"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "SEED = 2000\n",
    "x_train, x_validation_and_test, y_train, y_validation_and_test = train_test_split(x, y, test_size=.02, random_state=SEED)\n",
    "x_validation, x_test, y_validation, y_test = train_test_split(x_validation_and_test, y_validation_and_test, test_size=.5, random_state=SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Negative% Moderate% Positive%\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'x_train' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-96-0ae9dd0c8f30>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Total Negative% Moderate% Positive%\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_train\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_train\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_train\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_validation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_validation\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_validation\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_validation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_validation\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_validation\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_validation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_validation\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_validation\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_validation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m1.\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_test\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_test\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my_test\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'x_train' is not defined"
     ]
    }
   ],
   "source": [
    "print(\"Total Negative% Moderate% Positive%\")\n",
    "print(len(x_train),(len(x_train[y_train == 0]) / (len(x_train)))*100,(len(x_train[y_train == 1]) / (len(x_train)))*100,(len(x_train[y_train == 2]) / (len(x_train)))*100)\n",
    "print(len(x_validation),(len(x_validation[y_validation == 0]) / (len(x_validation)))*100,(len(x_validation[y_validation == 1]) / (len(x_validation)))*100,(len(x_validation[y_validation == 2]) / (len(x_validation)*1.))*100)\n",
    "print(len(x_test),(len(x_test[y_test == 0]) / (len(x_test)))*100,(len(x_test[y_test == 1]) / (len(x_test)))*100,(len(x_test[y_test == 2]) / (len(x_test))*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def labelize(text,label):\n",
    "    result = []\n",
    "    prefix = label\n",
    "    for i, t in zip(text.index, text):\n",
    "        result.append(TaggedDocument(t.split(), [prefix + '_%s' % i]))\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_x = pd.concat([x_train,x_validation,x_test])\n",
    "all_x_w2v = labelize(all_x, 'all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Word2Vec(sg=1, size=300, negative=5, window=2, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "model.build_vocab([x.words for x in tqdm(all_x_w2v)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "for epoch in range(10):\n",
    "    model.train(utils.shuffle([x.words for x in tqdm(all_x_w2v)]), total_examples=len(all_x_w2v), epochs=1)\n",
    "    model.alpha -= 0.002\n",
    "    model.min_alpha = model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('w2v_model_sg.word2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Word2Vec(sg=0, size=300, negative=5, window=2, min_count=2, workers=cores, alpha=0.065, min_alpha=0.065)\n",
    "model.build_vocab([x.words for x in tqdm(all_x_w2v)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "for epoch in range(10):\n",
    "    model.train(utils.shuffle([x.words for x in tqdm(all_x_w2v)]), total_examples=len(all_x_w2v), epochs=1)\n",
    "    model.alpha -= 0.002\n",
    "    model.min_alpha = model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'model' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-80-9857a63bfed1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'w2v_model_cbow.word2vec'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'model' is not defined"
     ]
    }
   ],
   "source": [
    "model.save('w2v_model_cbow.word2vec') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CBOW means continuous bag of words. SG means Skip Gram. \n",
    "With a corpus, CBOW model predicts the current word from a window of surrounding context words, while Skip-gram model predicts surrounding context words given the current word.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, let's say we have the following sentence: \"I love dogs\". CBOW model tries to predict the word \"love\" when given \"I\", \"dogs\" as inputs, on the other hand, Skip-gram model tries to predict \"I\", \"dogs\" when given the word \"love\" as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  gensim.models import KeyedVectors\n",
    "model_sg = KeyedVectors.load('w2v_model_sg.word2vec')\n",
    "model_cbow = KeyedVectors.load('w2v_model_cbow.word2vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(model_sg.wv.vocab.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(num_words=30000)\n",
    "tokenizer.fit_on_texts(x_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenizing Train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequences = tokenizer.texts_to_sequences(x_train)\n",
    "x_train_seq = pad_sequences(sequences, maxlen=80)\n",
    "print('Shape of data tensor:', x_train_seq.shape)\n",
    "partial_data = pd.DataFrame({\"text_vectors\":np.ndarray.tolist(x_train_seq),'label':y_train},dtype = 'object')\n",
    "partial_data.to_pickle(\"train_tokanized.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenizing Validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequences = tokenizer.texts_to_sequences(x_validation)\n",
    "x_validation_seq = pad_sequences(sequences, maxlen=80)\n",
    "print('Shape of data tensor:', x_validation_seq.shape)\n",
    "partial_data = pd.DataFrame({\"text_vectors\":np.ndarray.tolist(x_validation_seq),'label':y_validation},dtype = 'object')\n",
    "partial_data.to_pickle(\"validation_tokanized.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenizing Test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequences = tokenizer.texts_to_sequences(x_test)\n",
    "x_test_seq = pad_sequences(sequences, maxlen=80)\n",
    "print('Shape of data tensor:', x_test_seq.shape)\n",
    "partial_data = pd.DataFrame({\"text_vectors\":np.ndarray.tolist(x_test_seq),'label':y_test},dtype = 'object')\n",
    "partial_data.to_pickle(\"test_tokanized.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_index = {}\n",
    "for w in model_cbow.wv.vocab.keys():\n",
    "    embeddings_index[w] = np.append(model_cbow.wv[w],model_sg.wv[w])\n",
    "print('Found %s word vectors.' % len(embeddings_index))\n",
    "num_words  = 30000\n",
    "embedding_matrix = np.zeros((num_words, 600))\n",
    "for word, i in tokenizer.word_index.items():\n",
    "    if i >= num_words:\n",
    "        continue\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import to_categorical\n",
    "y_train = to_categorical(y_train,num_classes = 3)\n",
    "y_validation = to_categorical(y_validation,num_classes = 3)\n",
    "y_test = to_categorical(y_test,num_classes = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix,classification_report\n",
    "seed = 7\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import LSTM, Activation, Dense, Dropout, Flatten , Conv2D, MaxPooling2D, Reshape, ConvLSTM2D\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "model = Sequential()\n",
    "e = Embedding(30000, 600,  input_length=80, trainable=True) #weights=[embedding_matrix]\n",
    "model.add(e)\n",
    "model.add(Reshape((1,80,600),input_shape=(100000,)))\n",
    "model.add(Conv2D(filters=16, kernel_size=3, padding='same', activation='relu',))\n",
    "model.add(MaxPooling2D(pool_size=(3, 3), padding='same',strides = 2))\n",
    "model.add(Dropout(0.3))\n",
    "model.add(Conv2D(filters=16, kernel_size=3, padding='same', activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(3, 3), padding='same',strides = 2))              \n",
    "model.add(Dropout(0.3))\n",
    "model.add(Flatten())\n",
    "#model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(3, activation='sigmoid'))\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.callbacks import ModelCheckpoint\n",
    "filepath=\"CNN_weights.h5\"\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model.fit(x_train_seq, y_train, validation_data=(x_validation_seq, y_validation), epochs=3, batch_size=8, callbacks = [checkpoint])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import load_model\n",
    "loaded_CNN_model = load_model('CNN_weights.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_CNN_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#To plot model\n",
    "from keras.utils.vis_utils import plot_model\n",
    "plot_model(loaded_CNN_model, to_file='CNN_plot.png', show_shapes=True, show_layer_names=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_CNN_model.evaluate(x=x_test_seq, y=y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predict = loaded_CNN_model.predict(x=x_test_seq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix(np.argmax(y_test,axis = 1),np.argmax(y_predict,axis = 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_report(np.argmax(y_test,axis = 1),np.argmax(y_predict,axis = 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}