sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Sentiment Analyis and Prediction\n",
    "\n",
    "Index:\n",
    "\n",
    "-   [Sources](#Sources)\n",
    "-   [PreProcessing](#PreProcessing)\n",
    "-   [Model](#Model)\n",
    "-   [Results](#Results)\n",
    "-   [Conclusion](#Conclusion)\n",
    "-   [Usage](#Usage)\n",
    "-   [Bonus](#Bonus)\n",
    "\n",
    "## Sources\n",
    "\n",
    "As the grade isn't abouyt how we elaborate the dataset we used a [DataSet](https://www.kaggle.com/maxjon/complete-tweet-sentiment-extraction-data) originally made for a competition\n",
    "Thanks to the author of this dataset\n",
    "To interact with the data we will use [numpy](https://numpy.org/), [pandas](https://pandas.pydata.org/), [nltk](https://www.nltk.org/), [pickle](https://docs.python.org/3/library/pickle.html) and [sklearn](https://scikit-learn.org/stable/)\n",
    "We also used [tweepy](https://www.tweepy.org/) to try our trained model\n",
    "\n",
    "## PreProcessing\n",
    "\n",
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/romain/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /home/romain/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /home/romain/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n",
      "[nltk_data] Downloading package stopwords to /home/romain/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": "True"
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn as sk\n",
    "import nltk\n",
    "import re\n",
    "\n",
    "## Tokenization\n",
    "nltk.download('punkt')\n",
    "\n",
    "# Normalization\n",
    "nltk.download('wordnet')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "\n",
    "#Cleaning\n",
    "nltk.download('stopwords')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Importation\n",
    "\n",
    "We import the train data in order to preprocess it"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 27480 entries, 0 to 27480\n",
      "Data columns (total 4 columns):\n",
      " #   Column         Non-Null Count  Dtype \n",
      "---  ------         --------------  ----- \n",
      " 0   textID         27480 non-null  object\n",
      " 1   text           27480 non-null  object\n",
      " 2   selected_text  27480 non-null  object\n",
      " 3   sentiment      27480 non-null  object\n",
      "dtypes: object(4)\n",
      "memory usage: 1.0+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "train = pd.read_csv(\"data/train.csv\")\n",
    "# Remove null data\n",
    "train = train.dropna()\n",
    "\n",
    "print(train.info())"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Tokenization\n",
    "\n",
    "In order analyze the string data we 'tokenize' it\n",
    "It means that we split every sentence into an array of word and ponctuation"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       textID                                               text  \\\n",
      "0  cb774db0d1  [I, `, d, have, responded, ,, if, I, were, going]   \n",
      "1  549e992a42  [Sooo, SAD, I, will, miss, you, here, in, San,...   \n",
      "2  088c60f138                  [my, boss, is, bullying, me, ...]   \n",
      "3  9642c003ef             [what, interview, !, leave, me, alone]   \n",
      "4  358bd9e861  [Sons, of, *, *, *, *, ,, why, couldn, `, t, t...   \n",
      "\n",
      "                         selected_text sentiment  \n",
      "0  I`d have responded, if I were going   neutral  \n",
      "1                             Sooo SAD  negative  \n",
      "2                          bullying me  negative  \n",
      "3                       leave me alone  negative  \n",
      "4                        Sons of ****,  negative  \n"
     ]
    }
   ],
   "source": [
    "train[\"text\"] = [nltk.tokenize.word_tokenize(i) for i in train[\"text\"]]\n",
    "print(train.head())\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Remove Noise from our dataSet\n",
    "\n",
    "The noises are things like empty word, spaces, single letter or special characters.\n",
    "What we'll need to do now is to remove all the unwanted data with RegEx for example\n",
    "We may want to lower the words too"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "outputs": [],
   "source": [
    "def cleanData(word : str):\n",
    "    return re.sub(r'[^A-Za-z0-9_]','',word).lower()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Normalization\n",
    "\n",
    "We now have all our sentence sliced up into words, but we do face a problem.\n",
    "The same word can have multiple forms depending on the context.\n",
    "Our goal will be to transform all those version into the radical in order to get a smaller dictionnary.\n",
    "The process of grouping together forms of a word is called **Lemmatisation**.\n",
    "Firstly we tag the words to identify their type and secondly we use a dictionary to transform them to a simplest form.\n",
    "\n",
    "### StopWords\n",
    "\n",
    "What we want to do at the same time is removing useless words.\n",
    "Some words might not be helpful for us to understand the whole sentence, those are called stopwords\n",
    "For example: \"the\", \"an\", \"in\" doesn't help and would weaken the model as they appear to be really common in all type of sentence.\n",
    "They doesn't help us taking a decision"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "outputs": [],
   "source": [
    "tag_map = dict()\n",
    "tag_map['V'] = nltk.corpus.wordnet.VERB\n",
    "tag_map['J'] = nltk.corpus.wordnet.ADJ\n",
    "tag_map['N'] = nltk.corpus.wordnet.NOUN\n",
    "tag_map['R'] = nltk.corpus.wordnet.ADV\n",
    "tag_map['S'] = nltk.corpus.wordnet.ADJ_SAT\n",
    "\n",
    "# map the given posTag to the matching postag for nltk\n",
    "def convertPostagToLemmitizationTag(pos_tag):\n",
    "    return tag_map.get(pos_tag, nltk.corpus.wordnet.NOUN)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "outputs": [],
   "source": [
    "def lemmatize_sentence(sent : str):\n",
    "    lemmatizer = nltk.stem.wordnet.WordNetLemmatizer()\n",
    "    lemmatized_sentence = []\n",
    "    for word, tag in nltk.pos_tag(sent):\n",
    "        ## Clean our words with the function we made earlier\n",
    "        cleanedWord = cleanData(lemmatizer.lemmatize(word, convertPostagToLemmitizationTag(tag[0])))\n",
    "        ## If the word isn't null or a stopword then we add it to our final array\n",
    "        if cleanedWord is not None and len(cleanedWord) > 0 and cleanedWord not in nltk.corpus.stopwords.words('english'):\n",
    "            lemmatized_sentence.append(cleanedWord)\n",
    "    return lemmatized_sentence\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       textID                                text  \\\n",
      "0  cb774db0d1                       [respond, go]   \n",
      "1  549e992a42       [sooo, sad, miss, san, diego]   \n",
      "2  088c60f138                        [bos, bully]   \n",
      "3  9642c003ef           [interview, leave, alone]   \n",
      "4  358bd9e861  [sons, put, release, already, buy]   \n",
      "\n",
      "                         selected_text sentiment  \n",
      "0  I`d have responded, if I were going   neutral  \n",
      "1                             Sooo SAD  negative  \n",
      "2                          bullying me  negative  \n",
      "3                       leave me alone  negative  \n",
      "4                        Sons of ****,  negative  \n"
     ]
    }
   ],
   "source": [
    "## lemmatize and clean the sentence\n",
    "train[\"text\"] = [lemmatize_sentence(i) for i in train[\"text\"]]\n",
    "print(train.head())\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now to get a quick idea of the words repartition, we iterate through all the dataset, take the most used words for each sentiment\n",
    "We do have a pretty good idea of what type of words are mostly used"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "neutral: [('get', 1264), ('go', 1166), ('day', 645), ('work', 640), ('http', 596), ('lol', 494), ('u', 479), ('like', 474), ('time', 471), ('know', 456)]\n",
      "positive: [('day', 1345), ('good', 1191), ('love', 1012), ('happy', 866), ('get', 774), ('go', 636), ('thanks', 559), ('mother', 533), ('great', 495), ('like', 471)]\n",
      "negative: [('get', 922), ('go', 836), ('miss', 636), ('work', 494), ('like', 490), ('day', 420), ('feel', 408), ('sad', 405), ('im', 370), ('bad', 369)]\n"
     ]
    }
   ],
   "source": [
    "commonDic = dict()\n",
    "dataSet = {}\n",
    "\n",
    "for i in [\"neutral\", \"positive\", \"negative\"]:\n",
    "    commonDic[i] = []\n",
    "    for sentence in train[train[\"sentiment\"] == i][\"text\"]:\n",
    "        commonDic[i] = np.concatenate((commonDic[i], sentence))\n",
    "\n",
    "    commonDic[i] = nltk.FreqDist(commonDic[i])\n",
    "    print(i + \": \" + str(commonDic.get(i).most_common(10)))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Encoding\n",
    "\n",
    "To process the data easier we'll use an Encoder, firstly we'll encode the Y data (the sentiment).\n",
    "In order to do that we'll use the sk LabelEncoder\n",
    "\n",
    "In a second time we do the same for our our words by creating a dictionary of all our worlds and encoding it\n",
    "After that we encode each row\n",
    "But that time we will not use the LabelEncoder, instead we use the CountVectorizer\n",
    "Basically it create a dictionnary of known words, a matrix where each word is a row in the matrix.\n",
    "This way it's way easier to process data in our futur model\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "outputs": [],
   "source": [
    "sentimentEncoder = sk.preprocessing.LabelEncoder()\n",
    "\n",
    "encodedSentiment = sentimentEncoder.fit_transform(train[\"sentiment\"])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "outputs": [],
   "source": [
    "textEncoder = sk.feature_extraction.text.CountVectorizer()\n",
    "encodedText = textEncoder.fit_transform([ ' '.join(i) for i in train[\"text\"]])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Model\n",
    "\n",
    "Now that our data is cleaned up, we want to run our Model.\n",
    "As we want to put a label on a given sentence we conclude that we'll be using a classification algortihm.\n",
    "\n",
    "But Firstly we'll split up the data we previously preprocessed in order to get a training and testing set"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "outputs": [],
   "source": [
    "Xtrain, Xtest, Ytrain, Ytest = sk.model_selection.train_test_split(encodedText, encodedSentiment, test_size=0.15)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "To find the best matching algorithms, we decide to test out a few classificaiton algorithms on a lightweight dataset"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn import svm\n",
    "from sklearn.neighbors import KNeighborsClassifier"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GaussianNB: 0.397\n",
      "MultinomialNB: 0.56\n",
      "DecisionTreeClassifier: 0.608\n",
      "RandomForestClassifier: 0.611\n",
      "SVC: 0.585\n",
      "KNeighborsClassifier: 0.473\n"
     ]
    }
   ],
   "source": [
    "for model in [GaussianNB(), MultinomialNB(), DecisionTreeClassifier(), RandomForestClassifier(), svm.SVC(), KNeighborsClassifier()]:\n",
    "    model.fit(Xtrain.toarray()[:1000],Ytrain[:1000])\n",
    "    prediction = model.predict(Xtest.toarray()[:1000])\n",
    "    accuracy = sk.metrics.accuracy_score(Ytest[:1000], prediction)\n",
    "    with open(type(model).__name__ + \".result\", \"w\") as f:\n",
    "        f.write(\"\\n\\n\" + type(model).__name__ + \": \" + str(accuracy))\n",
    "        f.write(\"\\nPrediction:\\n\")\n",
    "        f.write(str(prediction))\n",
    "        f.write(\"\\nReal:\\n\")\n",
    "        f.write(str(Ytest[:1000]))\n",
    "    print(type(model).__name__ + \": \" + str(accuracy))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now have a clearer idea of which type of algorithm we want to use.\n",
    "In order to know which one will best fit our data we decide to train the Top3 with the full set of data."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MultinomialNB: 0.6491994177583698\n",
      "DecisionTreeClassifier: 0.6669092673459486\n",
      "RandomForestClassifier: 0.6933527413876759\n"
     ]
    }
   ],
   "source": [
    "for model in [MultinomialNB(), DecisionTreeClassifier(), RandomForestClassifier()]:\n",
    "    model.fit(Xtrain.toarray(),Ytrain)\n",
    "    prediction = model.predict(Xtest.toarray())\n",
    "    accuracy = sk.metrics.accuracy_score(Ytest, prediction)\n",
    "    with open(type(model).__name__ + \".bigResult\", \"w\") as f:\n",
    "        f.write(\"\\n\\n\" + type(model).__name__ + \": \" + str(accuracy))\n",
    "    print(type(model).__name__ + \": \" + str(accuracy))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now have to decide which algorithm to use in our definitive version, in our case we will use the RandomForestClassifier.\n",
    "By the way, this algorithm is well known to work perfectly with text analysis so it's pretty logic.\n",
    "\n",
    "### Reusability\n",
    "\n",
    "As we don't want to train our model each time, we train a model and save it with pickle.\n",
    "That way we can load it quickly and make predictions without retraining the model every time."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForestClassifier: 0.6926249393498302\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "\n",
    "model = RandomForestClassifier()\n",
    "model.fit(Xtrain.toarray(), Ytrain)\n",
    "prediction = model.predict(Xtest.toarray())\n",
    "accuracy = sk.metrics.accuracy_score(Ytest, prediction)\n",
    "with open(type(model).__name__ + \".finalResult\", \"w\") as f:\n",
    "    f.write(\"\\n\\n\" + type(model).__name__ + \": \" + str(accuracy))\n",
    "print(type(model).__name__ + \": \" + str(accuracy))\n",
    "\n",
    "pickle.dump(model, open('trainedModel.sav', 'wb'))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now have a pre-trained model that we can use on the run like so:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9539058709364386\n"
     ]
    }
   ],
   "source": [
    "model = pickle.load(open('trainedModel.sav', 'rb'))\n",
    "result = model.score(Xtest.toarray(), Ytest)\n",
    "print(result)\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Results"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9539058709364386\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "\n",
    "model = pickle.load(open('trainedModel.sav', 'rb'))\n",
    "result = model.score(Xtest.toarray(), Ytest)\n",
    "print(result)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As we can see there, our model, once trained is very efficient and as an accuracy score of 95% which is pretty good\n",
    "We are satisfied with this accuracy and keep our train model like so\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "To conclude, we do have now a model trained to identify the sentiment of a tweet.\n",
    "The most important part is how we processed the data to make a more precise and efficient model.\n",
    "We had to make a few decisions about how we process the data, removing stopwords or not, keeping special characters, encoding algorithm\n",
    "The model we decided to use is also really important, and we do think that the RandomForestClassifier fits our need.\n",
    "\n",
    "## Usage\n",
    "\n",
    "We can use this tweet sentiment recognizer for multiple usages if combined with a tweet scraper\n",
    "For example, it could be interesting to determine the whole sentiment of a twitter account or by month to know the mindset of a given person\n",
    "But most scrapers are done lately so we'll have to use the official API\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "negative\n",
      "neutral\n",
      "positive\n"
     ]
    }
   ],
   "source": [
    "def get_sentiment(tweet : str):\n",
    "    encoded_text = textEncoder.transform([' '.join(lemmatize_sentence(nltk.tokenize.word_tokenize(tweet)))]) #([ ' '.join(i) for i in lemmatize_sentence(tweet)])\n",
    "    return sentimentEncoder.inverse_transform(model.predict(encoded_text.toarray()))[0]\n",
    "\n",
    "print(get_sentiment(\"i'm really sad those day\"))\n",
    "print(get_sentiment(\"do french fries have a soul ?\"))\n",
    "print(get_sentiment(\"so happy to see my friend today\"))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Bonus\n",
    "\n",
    "Here is a sample code to interact with a user profile to get his tweets positivity score"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "outputs": [],
   "source": [
    "\n",
    "import tweepy\n",
    "def get_profile_positivity(username : str, nb_tweet : int = 100, detailled : bool = False):\n",
    "    auth = tweepy.OAuthHandler(\n",
    "        open(\"./info/api.key\",\"r\").read(),\n",
    "        open(\"./info/apikey.secret\",\"r\").read()\n",
    "    )\n",
    "    auth.set_access_token(\n",
    "        open(\"./info/access.token\",\"r\").read(),\n",
    "        open(\"./info/accesssecret.token\",\"r\").read()\n",
    "    )\n",
    "\n",
    "    api = tweepy.API(auth)\n",
    "    positivity = {\n",
    "        \"negative\":0,\n",
    "        \"neutral\":0,\n",
    "        \"positive\":0\n",
    "    }\n",
    "    if detailled:\n",
    "        positivity[\"negative\"] = []\n",
    "        positivity[\"positive\"] = []\n",
    "        positivity[\"neutral\"] = []\n",
    "\n",
    "\n",
    "    for t in api.user_timeline(username, count=nb_tweet, include_rts=False):\n",
    "        if not detailled:\n",
    "            positivity[get_sentiment(t.text)] += 1\n",
    "        else:\n",
    "            positivity[get_sentiment(t.text)].append(t.text)\n",
    "\n",
    "    return positivity"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'negative': ['Sleepy Eyes Chuck Todd is so happy with the fake voter tabulation process that he can’t even get the words out straight. Sad to watch!', 'The Vice President has the power to reject fraudulently chosen electors.', 'How can you certify an election when the numbers being certified are verifiably WRONG. You will see the real number… https://t.co/jfBOEEVjX7', 'Sorry, but the number of votes in the Swing States that we are talking about is VERY LARGE and totally OUTCOME DETE… https://t.co/KZKiATT1lB', 'Why haven’t they done signature verification in Fulton County, Georgia. Why haven’t they deducted all of the dead p… https://t.co/Bkz8kFz41u', 'Some States are very slow to inoculate recipients despite successful and very large scale distribution of vaccines… https://t.co/uCpPoqzWuA', 'Our Republican Senate just missed the opportunity to get rid of Section 230, which gives unlimited power to Big Tec… https://t.co/bevkn4zsNf', 'Watching @FoxNews is almost as bad as watching Fake News @CNN. New alternatives are developing!', 'New Lott study estimates 11,350 absentee votes lost to Trump in Georgia. Another 289,000 \"excess (fraudulent) votes… https://t.co/V6cH6HYifI', 'The Wall Street Journal’s very boring &amp; incoherent Editorial fails to mention my big &amp; easy wins in Texas, Florida,… https://t.co/JZaL1rH3FB', '....Can you imagine if the Republicans stole a Presidential Election from the Democrats - All hell would break out.… https://t.co/v4WDkxludM', 'Weak and tired Republican “leadership” will allow the bad Defense Bill to pass. Say goodbye to VITAL Section 230 te… https://t.co/fzmmkQWmsv'], 'neutral': ['These are the things and events that happen when a sacred landslide election victory is so unceremoniously &amp; viciou… https://t.co/uYLdmiz5Ka', 'https://t.co/Pm2PKV0Fp3', 'I am asking for everyone at the U.S. Capitol to remain peaceful. No violence! Remember, WE are the Party of Law &amp; O… https://t.co/XWZnbZWwze', 'Mike Pence didn’t have the courage to do what should have been done to protect our Country and our Constitution, gi… https://t.co/pNct6O9uuy', 'https://t.co/izItBeFE6G', 'Even Mexico uses Voter I.D.', 'The States want to redo their votes. They found out they voted on a FRAUD. Legislatures never approved. Let them do it. BE STRONG!', 'They just happened to find 50,000 ballots late last night. The USA is embarrassed by fools. Our Election Process is… https://t.co/cLaYvIvzab', 'THE REPUBLICAN PARTY AND, MORE IMPORTANTLY, OUR COUNTRY, NEEDS THE PRESIDENCY MORE THAN EVER BEFORE - THE POWER OF THE VETO. STAY STRONG!', 'States want to correct their votes, which they now know were based on irregularities and fraud, plus corrupt proces… https://t.co/i1ZARUmtHr', 'If Vice President @Mike_Pence comes through for us, we will win the Presidency. Many States want to decertify the m… https://t.co/9snX9u3SrA', 'Get smart Republicans. FIGHT! https://t.co/3fs1oPVnAx', 'Just happened to have found another 4000 ballots from Fulton County. Here we go!', 'Looks like they are setting up a big “voter dump” against the Republican candidates. Waiting to see how many votes they need?', 'BIG NEWS IN PENNSYLVANIA! https://t.co/7JqTWYUgOr', 'I will be speaking at the SAVE AMERICA RALLY tomorrow on the Ellipse at 11AM Eastern. Arrive early — doors open at… https://t.co/0OjqxuqF3Q', 'Antifa is a Terrorist Organization, stay out of Washington. Law enforcement is watching you very closely!… https://t.co/v0zJSzt2BI', 'Washington is being inundated with people who don’t want to see an election victory stolen by emboldened Radical Le… https://t.co/u4LEyfrt5W', 'GEORGIA! Get out today and VOTE for @KLoeffler and @Perduesenate! https://t.co/YKiSx7d7lp', 'Reports are coming out of the 12th Congressional District of Georgia that Dominion Machines are not working in cert… https://t.co/TdklHEmTnx', 'See you in D.C. https://t.co/ti4bChnPKz', 'https://t.co/tJkmEhSqY5', 'https://t.co/X0CIj5dFlV', 'https://t.co/zkFfhX0hOQ', 'https://t.co/gWj2obEKm8', 'https://t.co/9fRvjIgqKE', 'On my way, see you soon! https://t.co/7QW23k5b9r', 'https://t.co/3LNZWTO8NY', 'Heading to Georgia now. See you soon!', 'The “Surrender Caucus” within the Republican Party will go down in infamy as weak and ineffective “guardians” of ou… https://t.co/gDEuykZxdB', '“We are not acting to thwart the Democratic process, we are acting to protect it.” @SenRonJohnson', 'https://t.co/PXNO26lpoZ', 'https://t.co/2rLJfoKmSS', 'https://t.co/tuGfzK3QPE', 'https://t.co/JEzkcjgLSy', 'https://t.co/NmdLNsfDU2', 'https://t.co/dN2P40kJLW', '....@SenTedCruz @HawleyMO @Jim_Jordan @senatemajldr @GOPLeader &amp; THE WORLD!', 'The Swing States did not even come close to following the dictates of their State Legislatures. These States  “elec… https://t.co/6NsOU46HF6', 'Trump Speaks to State Legislators on Call About Decertifying Election https://t.co/z6BgCAe3zX via @BreitbartNews', 'https://t.co/FDUVk8cm9S', 'I will be there. Historic day! https://t.co/k6LStsWpfy', '“Georgia election data, just revealed, shows that over 17,000 votes illegally flipped from Trump to Biden.” @OANN… https://t.co/Meq7ioowNB', 'I spoke to Secretary of State Brad Raffensperger yesterday about Fulton County and voter fraud in Georgia. He was u… https://t.co/crqOHxwPtq', 'Republicans in Georgia must be careful of the political corruption in Fulton County, which is rampant. The Governor… https://t.co/ZKjiUIFD7l', 'The number of cases and deaths of the China Virus is far exaggerated in the United States because of @CDCgov’s ridi… https://t.co/6B8A4OOW1e', 'The vaccines are being delivered to the states by the Federal Government far faster than they can be administered!', 'GOP Senators Join Hawley in Objecting to Electoral College Votes https://t.co/f0JHfN4UUb via @BreitbartNews', 'https://t.co/moYUIcRq56', 'https://t.co/JLq7hHI42o', 'https://t.co/nslWcFwkCj', 'An attempt to steal a landslide win. Can’t let it happen! https://t.co/sKn4iTjUy0', '.@senatemajldr Mitch M, and all! https://t.co/zLIEKzfpFv', 'Civil War: Tucker Carlson Hits His Own Network in Epic Post-Election Monologue https://t.co/xUSQQWCa8q', '....Just a small portion of these votes give US a big and conclusive win in Georgia. Have they illegally destroyed… https://t.co/13AvDqYADp', 'TRANSPARENCY in medical # will be one of the biggest and most important things done for the American citizen.… https://t.co/YZGY38AYeV', 'For historical purposes remember, I was able to get rid of the INDIVIDUAL MANDATE, the most unpopular and expensive… https://t.co/JcQ7TodsDV', 'https://t.co/t6nWHJjJAN', '....Please remember who got it done!!!', 'Because of the Trump Administration, hospitals are now required, effective immediately, to publish their REAL PRICE… https://t.co/mqYtZXlBjP', 'Only because Biden got very few votes, just like the Election! https://t.co/sNIAJ1i5hu', 'Herschel is speaking the truth! https://t.co/6x9VLsc9qf', 'Republicans should have gotten rid of Section 230 in the Defense Bill, and you wouldn’t have had this problem. Neve… https://t.co/KZmam0zhAZ', 'January 6th. See you in D.C. https://t.co/vynZTv9lHb', '....is therefore both illegal and invalid, and that would include the two current Senatorial Elections. In Wisconsi… https://t.co/26IjrqlUOp', '....changes made to the voting process, rules and regulations, many made hastily before the election, and therefore… https://t.co/9OnMCZAn4e', 'Before even discussing the massive corruption which took place in the 2020 Election, which gives us far more votes… https://t.co/2seIXXRoIl', '.@FoxNews Weekend Daytime is not watchable. Switching over to @OANN!', 'NOW! https://t.co/hjC1WW64O6', 'Massive amounts of evidence will be presented on the 6th. We won, BIG! https://t.co/ymncRrNR5t', 'The BIG Protest Rally in Washington, D.C., will take place at 11.00 A.M. on January 6th. Locational details to follow. StopTheSteal!', 'https://t.co/RBgT5SMWGT', 'https://t.co/7YHKqgjAub', 'https://t.co/FDUVk8cm9S', 'Sen. Josh Hawley Slams Walmart Tweet Calling Him a ‘Sore Loser‘ https://t.co/wmShnPcrzj via @BreitbartNews America… https://t.co/H38fgKR5BS', 'Finished off the year with the highest Stock Market in history. Setting records with your 401k’s, just like I said… https://t.co/v4mhFx0TdM', 'https://t.co/2kYtZF8Mei', 'https://t.co/S6fKpzoXZZ', 'We now have far more votes than needed to flip Georgia in the Presidential race. Massive VOTER FRAUD took place. Th… https://t.co/9xjslmlGf4', '.@BrianKempGA, his puppet Lt. Governor @GeoffDuncanGA, and Secretary of State, are disasters for Georgia. Won’t let… https://t.co/AzUchjw5uN', 'The United States had more votes than it had people voting, by a lot. This travesty cannot be allowed to stand. It… https://t.co/dH1j93GtiD', 'JANUARY SIXTH, SEE YOU IN DC!', 'Twitter is shadow banning like never before. A disgrace that our weak and ineffective political leadership refuses… https://t.co/bZPpwdsODN', 'Hearings from Atlanta on the Georgia Election overturn now being broadcast LIVE via @RSBNetwork! \\nhttps://t.co/ogBvLbKfqG', 'Hearings from Atlanta on the Georgia Election overturn now being broadcast. Check it out. @OANN @newsmax and many m… https://t.co/Ejjc4SEieM', '$2000 ASAP!', 'The Federal Government has distributed the vaccines to the states. Now it is up to the states to administer. Get moving!', '...millions more votes than Trump, but can’t get anywhere close to him in this poll. No incoming president has ever… https://t.co/XDTBmSlzbH', '“Barack Obama was toppled from the top spot and President Trump claimed the title of the year’s Most Admired Man. T… https://t.co/aKtP3lCSpz', '....other acts of fraud and irregularities as well. STAY TUNED!', '...The consent decree signed by the “Secretary”, with the consent of Kemp, is perhaps even more poorly negotiated t… https://t.co/gfDa0RcV2V', 'https://t.co/Dcm41ThF5k', 'When are we going to be allowed to do signature verification in Fulton County, Georgia? The process is going VERY s… https://t.co/gkDts3Q0n7', 'It is up to the States to distribute the vaccines once brought to the designated areas by the Federal Government. W… https://t.co/eg0GJTW71l', 'Loeffler, Perdue Support Increasing Relief Payments to $2K https://t.co/xLFHNDOBIR via @BreitbartNews. Republicans… https://t.co/WukNDe4R4z', 'Unless Republicans have a death wish, and it is also the right thing to do, they must approve the $2000 payments AS… https://t.co/Ur6BCexxt1', '....and Congressmen/Congresswomen Elected. I do believe they forgot!', '...more votes than is needed by me to win Pennsylvania, not to mention hundreds of thousands of votes in other cate… https://t.co/Uo18Qad6JE', '“A group of Republican lawmakers in Pennsylvania say 200,000 more votes were counted in the 2020 Election than vote… https://t.co/onHvrHbUpN', '....being removed and brought home from foreign lands who do NOTHING for us. A disgraceful act of cowardice and tot… https://t.co/CsscO7MIj6', 'Give the people $2000, not $600. They have suffered enough! https://t.co/2jOVCnGtXS', 'This Tweet from @realDonaldTrump has been withheld in response to a report from the copyright holder. Learn more.', '“Breaking News: In Pennsylvania there were 205,000 more votes than there were voters. This alone flips the state to President Trump.”'], 'positive': ['Please support our Capitol Police and Law Enforcement. They are truly on the side of our Country. Stay peaceful!', 'These scoundrels are only toying with the @sendavidperdue (a great guy) vote. Just didn’t want to announce quite ye… https://t.co/PWkWMR8npN', 'I hope the Democrats, and even more importantly, the weak and ineffective RINO section of the Republican Party, are… https://t.co/qOXn3ASfvH', 'Georgia, get out and VOTE for two great Senators, @KLoeffler and @sendavidperdue. So important to do so!', 'Pleased to announce that @KLoeffler &amp; @sendavidperdue have just joined our great #StopTheSteal group of Senators. T… https://t.co/rBcAsw9bx6', '“We’ve seen in the last few months, unprecedented amounts of Voter Fraud.” @SenTedCruz  True!', 'Great! https://t.co/f0nKNPgqWg', 'Something how Dr. Fauci is revered by the LameStream Media as such a great professional, having done, they say, suc… https://t.co/xEpEJBWmKu', 'So true. Thanks Josh! https://t.co/lacUQC6IHh', '...And after they see the facts, plenty more to come...Our Country will love them for it! #StopTheSteal https://t.co/0IdbiACLIb', 'Wow, I guess it’s not good to go against a President who everyone in Georgia knows got you into office! https://t.co/4xUNdOncoB', 'MAKE AMERICA GREAT AGAIN!', 'Will be in Georgia on Monday night, 9:00 P.M. to RALLY for two GREAT people, @sendavidperdue &amp; @KLoeffler. GET READY TO VOTE ON TUESDAY!!!', 'Thank you Madison! https://t.co/XyaAHO9CwC', 'A great honor! https://t.co/U4WFBWrnF7', 'I hope to see the great Governor of South Dakota @KristiNoem, run against RINO @SenJohnThune, in the upcoming 2022… https://t.co/3z3zTYTvH7', 'HAPPY NEW YEAR!', 'Thank you, a great honor! https://t.co/9GfT5c8hY0', '....that, quite frankly, didn’t have much of a chance, like 7, 8 or 9. The Presidential Election was Rigged with hu… https://t.co/56MgHJkCYU', 'I love the Great State of Georgia, but the people who run it, from the Governor, @BrianKempGA, to the Secretary of… https://t.co/HL2Fh8oO7F', '$2000 for our great people, not $600! They have suffered enough from the China Virus!!!']}\n"
     ]
    }
   ],
   "source": [
    "print(get_profile_positivity(\"realDonaldTrump\", 5000, detailled=True))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'negative': 5, 'neutral': 65, 'positive': 1}\n"
     ]
    }
   ],
   "source": [
    "print(get_profile_positivity(\"the_weird_weeb\", 5000))\n",
    "\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}