hws/hw5_twitter_api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Total points 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a numpy review question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b= np.array([[2, 5 ,8, 9] ])\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q 1(.5 point) Define a numpy array a such that np.dot(a, b) = \n",
    "\n",
    "array([\n",
    "\n",
    "       [2, 5, 8, 9],\n",
    "\n",
    "       [2, 5, 8, 9],\n",
    "       \n",
    "       [2, 5, 8, 9],\n",
    "       \n",
    "       [2, 5, 8, 9]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here. print the value of np.dot(a,b).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In Rest of the homework\n",
    "- We'll collect climate change related tweet from twitter using REST API\n",
    "- Use tweets to populate Pandas DataFrames\n",
    "- Perform some elementary pandas operation to check data sanity\n",
    "\n",
    "\n",
    "**Please read**\n",
    "\n",
    "- https://developer.twitter.com/en/developer-terms/more-on-restricted-use-cases\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before proceeding with homework.\n",
    "\n",
    "<font size=\"6\" color='red'> Create new app at https://apps.twitter.com/app/new  (You may have to create twitter account before this) </font>\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[twitter api reference index](https://developer.twitter.com/en/docs/api-reference-index)\n",
    "\n",
    "\n",
    "# POST oauth2/token\n",
    "Allows a registered application to obtain an OAuth 2 Bearer Token, which can be used to make API requests on an application's own behalf, without a user context.\n",
    "https://developer.twitter.com/en/docs/basics/authentication/api-reference/token\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have create app, click on app details and keys and token tab\n",
    "\n",
    "<font size=\"6\" color='red'> write API key value for client_key and API secret key value for client_secret in the next cell</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client_key = #write your API key\n",
    "client_secret =  #write your API secret key "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import base64\n",
    "\n",
    "key_secret = '{}:{}'.format(client_key, client_secret).encode('ascii')\n",
    "b64_encoded_key = base64.b64encode(key_secret)\n",
    "b64_encoded_key = b64_encoded_key.decode('ascii')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_url = 'https://api.twitter.com/'\n",
    "auth_endpoint = base_url+'oauth2/token'\n",
    "\n",
    "auth_headers = { 'Authorization': 'Basic {}'.format(b64_encoded_key),\n",
    "                'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'}\n",
    "\n",
    "auth_data = { 'grant_type': 'client_credentials'}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's post to authentication endpoint and get the token to be used in subsequent REST API call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(auth_endpoint, headers=auth_headers, data=auth_data)\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_data =  response.json()\n",
    "json_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "access_token = json_data['access_token']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's do some search on climate change\n",
    "This is how we got search endpoint\n",
    "\n",
    "https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_tweets=10000\n",
    "tw_limit=100# we can get only 100 tweet per call with standard search api\n",
    "tweets = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_headers = {'Authorization': 'Bearer {}'.format(access_token)    \n",
    "}\n",
    "\n",
    "parameters = { 'q': 'climate change',\n",
    "                    'result_type': 'recent',\n",
    "                'count': 100 }\n",
    "\n",
    "search_url = base_url+'1.1/search/tweets.json'\n",
    "\n",
    "response = requests.get(search_url, headers=search_headers, params=parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "climate_tweets = response.json()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(climate_tweets['statuses'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets +=climate_tweets['statuses']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    " ids = [tw['id'] for tw in climate_tweets['statuses']]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_id = min(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tw_ids = []\n",
    "tw_ids+=ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As per twitter api documentation we can use\n",
    "\n",
    "max_id:Returns results with an ID less than (that is, older than) or equal to the specified ID.\n",
    "\n",
    "to get older tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q2 (.5 points) Modify only *parameters* dictionary in the following code so that subsequent call to climate change search returns older tweets with id less than minimum of previous tweets ids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for  i in range(max_tweets//tw_limit -1):\n",
    "    parameters = { 'q': 'climate change',\n",
    "                    'result_type': 'recent',\n",
    "                'count': tw_limit\n",
    "                 }\n",
    "    print(\"searching tweets with id  < {}\".format(min_id))\n",
    "    search_url = base_url+'1.1/search/tweets.json'\n",
    "    response = requests.get(search_url, headers=search_headers, params=parameters)\n",
    "    climate_tweets = response.json()\n",
    "    ids = [tw['id'] for tw in climate_tweets['statuses']]\n",
    "    tw_ids+=ids\n",
    "    print(i, len(climate_tweets['statuses']))\n",
    "    min_id = min(ids)\n",
    "    tweets +=climate_tweets['statuses']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Total tweets retrieved= {}'.format(len(tweets)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's print some tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pprint #pretty-print” arbitrary Python data structures. Tweets object has hierarchical struture\n",
    "max_tweet_to_print=4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint.pprint(tweets[0:max_tweet_to_print])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Link to understand [tweet-object](https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/tweet-object)\n",
    "\n",
    "Let'see some tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for status in tweets[0:max_tweet_to_print]:\n",
    "    print(status['text'] + '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas DataFrame for storing tweet feed\n",
    "\n",
    "Create following date frame for storing tweets\n",
    "\n",
    "- **climate_feed_df** DataFrame for storing tweet\n",
    "- **user_df** DataFrame for storing user detail\n",
    "- **retweeted_status_df** for storing original tweet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Printing keys in tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tweets[1].keys())\n",
    "print(len(tweets[1].keys()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generators for building DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q3 (2 point) Complete following   python generator *tweet_generator*. Generator yield modified tweet .\n",
    "For each tweet in tweets\n",
    "- Replace **user** attribute with **user_id** and value with **tweet['user']['id']** value.\n",
    "- Replace **retweeted_status** attribute with **retweeted_status_id** and value with **tweet['retweeted_status']['id']** value.\n",
    "- Keep other attributes in the tweet as such."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tweet_generator(tweets):\n",
    "    for tweet in tweets:\n",
    "        modified_tweet= {}\n",
    "        \n",
    "        #Write your code here\n",
    "        \n",
    "        \n",
    "        yield modified_tweet       \n",
    "            \n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checking  generator output\n",
    "list(tweet_generator(tweets[0:2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "climate_feed_df = pd.DataFrame(tweet_generator(tweets))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "climate_feed_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q4(1 points) List datatype using climate_feed_df. Describe which attributes doesn't have right data types. What should be correct datatype."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write code here datatype listing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# write columns with wrong datatype and mention correct data type too.\n",
    "\n",
    "*double click this cell to edit*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q5 (1 point) Complete following tweet_user_generator. Generator should yield only *user* value from a tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tweet_user_generator(tweets):\n",
    "# write code here  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_df = pd.DataFrame(tweet_user_generator(tweets))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q6(1 point) List datatype of columns using user_df. Describe which attributes doesn't have right data types. What should be correct datatype."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write code here for datatype listing\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# write columns with wrong datatype and mention correct data type too.\n",
    "\n",
    "*double click this cell to edit*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q7 (1 point) Complete following retweeted_status_generator. Generator should yield dictionary of retweeted_status value from each tweet. If a tweet doesn't have this key return empty dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retweeted_status_generator(tweets):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retweeted_status_df = pd.DataFrame(retweeted_status_generator(tweets))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retweeted_status_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q8 (1 point) Write code to find number of rows in retweeted_status_df with all the values NaN. print number of such rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write code here\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}