Collecting_Jobs_data_Using_API-Questions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Collecting Job Data Using APIs**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **45 to 60** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After completing this lab, you will be able to:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Collect job data from Jobs API\n",
    "*   Store the collected data into an excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "><strong>Note: Before starting with the assignment make sure to read all the instructions and then move ahead with the coding part.</strong>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instructions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the actual lab, firstly you need to click on the [Jobs_API](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Jobs_API.ipynb) notebook link. The file contains flask code which is required to run the Jobs API data.\n",
    "\n",
    "Now, to run the code in the file that opens up follow the below steps.\n",
    "\n",
    "Step1: Download the file. \n",
    "\n",
    "Step2: Upload it on the IBM Watson studio. (If IBM Watson Cloud service does not work in your system, follow the alternate Step 2 below)\n",
    "\n",
    "Step2(alternate): Upload it in your SN labs environment using the upload button which is highlighted in red in the image below:\n",
    "Remember to upload this Jobs_API file in the same folder as your current .ipynb file\n",
    "\n",
    "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Upload.PNG\">\n",
    "\n",
    "Step3:  Run all the cells of the Jobs_API file. (Even if you receive an asterik sign after running the last cell, the code works fine.)\n",
    "\n",
    "If you want to learn more about flask, which is optional, you can click on this link [here](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/FLASK_API.md.html).\n",
    "\n",
    "Once you run the flask code, you can start with your assignment.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Used in this Assignment\n",
    "\n",
    "The dataset used in this lab comes from the following source: https://www.kaggle.com/promptcloud/jobs-on-naukricom under the under a **Public Domain license**.\n",
    "\n",
    "> Note: We are using a modified subset of that dataset for the lab, so to follow the lab instructions successfully please use the dataset provided with the lab, rather than the dataset from the original source.\n",
    "\n",
    "The original dataset is a csv. We have converted the csv to json as per the requirement of the lab.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warm-Up Exercise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you attempt the actual lab, here is a fully solved warmup exercise that will help you to learn how to access an API.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using an API, let us find out who currently are on the International Space Station (ISS).<br> The API at [http://api.open-notify.org/astros.json](http://api.open-notify.org/astros.json?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2021-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ) gives us the information of astronauts currently on ISS in json format.<br>\n",
    "You can read more about this API at [http://open-notify.org/Open-Notify-API/People-In-Space/](http://open-notify.org/Open-Notify-API/People-In-Space?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2021-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests # you need this module to make an API call\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "api_url = \"http://api.open-notify.org/astros.json\" # this url gives use the astronaut data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = requests.get(api_url) # Call the API using the get method and store the\n",
    "                                # output of the API call in a variable called response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if response.ok:             # if all is well() no errors, no network timeouts)\n",
    "    data = response.json()  # store the result in json format in a variable called data\n",
    "                            # the variable data is of type dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'people': [{'craft': 'ISS', 'name': 'Oleg Kononenko'}, {'craft': 'ISS', 'name': 'Nikolai Chub'}, {'craft': 'ISS', 'name': 'Tracy Caldwell Dyson'}, {'craft': 'ISS', 'name': 'Matthew Dominick'}, {'craft': 'ISS', 'name': 'Michael Barratt'}, {'craft': 'ISS', 'name': 'Jeanette Epps'}, {'craft': 'ISS', 'name': 'Alexander Grebenkin'}, {'craft': 'ISS', 'name': 'Butch Wilmore'}, {'craft': 'ISS', 'name': 'Sunita Williams'}, {'craft': 'Tiangong', 'name': 'Li Guangsu'}, {'craft': 'Tiangong', 'name': 'Li Cong'}, {'craft': 'Tiangong', 'name': 'Ye Guangfu'}], 'number': 12, 'message': 'success'}\n"
     ]
    }
   ],
   "source": [
    "print(data)   # print the data just to check the output or for debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the number of astronauts currently on ISS.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12\n"
     ]
    }
   ],
   "source": [
    "print(data.get('number'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the names of the astronauts currently on ISS.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 12 astronauts on ISS\n",
      "And their names are :\n",
      "Oleg Kononenko\n",
      "Nikolai Chub\n",
      "Tracy Caldwell Dyson\n",
      "Matthew Dominick\n",
      "Michael Barratt\n",
      "Jeanette Epps\n",
      "Alexander Grebenkin\n",
      "Butch Wilmore\n",
      "Sunita Williams\n",
      "Li Guangsu\n",
      "Li Cong\n",
      "Ye Guangfu\n"
     ]
    }
   ],
   "source": [
    "astronauts = data.get('people')\n",
    "print(\"There are {} astronauts on ISS\".format(len(astronauts)))\n",
    "print(\"And their names are :\")\n",
    "for astronaut in astronauts:\n",
    "    print(astronaut.get('name'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hope the warmup was helpful. Good luck with your next lab!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab: Collect Jobs Data using Jobs API\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Objective: Determine the number of jobs currently open for various technologies  and for various locations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collect the number of job postings for the following locations using the API:\n",
    "\n",
    "* Los Angeles\n",
    "* New York\n",
    "* San Francisco\n",
    "* Washington DC\n",
    "* Seattle\n",
    "* Austin\n",
    "* Detroit\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Import required libraries\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write a function to get the number of jobs for the Python technology.<br>\n",
    "> Note: While using the lab you need to pass the **payload** information for the **params** attribute in the form of **key** **value** pairs.\n",
    "  Refer the ungraded **rest api lab** in the course **Python for Data Science, AI & Development**  <a href=\"https://www.coursera.org/learn/python-for-applied-data-science-ai/ungradedLti/P6sW8/hands-on-lab-access-rest-apis-request-http?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\">link</a>\n",
    "  \n",
    " ##### The keys in the json are \n",
    " * Job Title\n",
    " \n",
    " * Job Experience Required\n",
    " \n",
    " * Key Skills\n",
    " \n",
    " * Role Category\n",
    " \n",
    " * Location\n",
    " \n",
    " * Functional Area\n",
    " \n",
    " * Industry\n",
    " \n",
    " * Role \n",
    " \n",
    "You can also view  the json file contents  from the following <a href = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/jobs.json\">json</a> URL.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "api_url=\"http://127.0.0.1:5000/data\"\n",
    "def get_number_of_jobs_T(technology):\n",
    "    number_of_jobs = 0\n",
    "    payload={\"Key Skills\":technology}\n",
    "    response=requests.get(api_url, params=payload)\n",
    "    if response.ok:\n",
    "        data=response.json()\n",
    "        number_of_jobs = len(data)\n",
    "    return technology, number_of_jobs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling the function for Python and checking if it works.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Python', 1173)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_number_of_jobs_T(\"Python\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write a function to find number of jobs in US for a location of your choice\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "api_url=\"http://127.0.0.1:5000/data\"\n",
    "def get_number_of_jobs_L(location):\n",
    "    number_of_jobs=0\n",
    "    payload={\"Location\":location}\n",
    "    response=requests.get(api_url,params=payload)\n",
    "    if response.ok:\n",
    "        data=response.json()\n",
    "        number_of_jobs = len(data)\n",
    "    #your coe goes here\n",
    "    return location,number_of_jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the function for Los Angeles and check if it is working.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('San Francisco', 435)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code goes here\n",
    "get_number_of_jobs_L(\"San Francisco\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store the results in an excel file\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the API for all the given technologies above and write the results in an excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you do not know how create excel file using python, double click here for **hints**.\n",
    "\n",
    "<!--\n",
    "\n",
    "from openpyxl import Workbook        # import Workbook class from module openpyxl\n",
    "wb=Workbook()                        # create a workbook object\n",
    "ws=wb.active                         # use the active worksheet\n",
    "ws.append(['Country','Continent'])   # add a row with two columns 'Country' and 'Continent'\n",
    "ws.append(['Eygpt','Africa'])        # add a row with two columns 'Egypt' and 'Africa'\n",
    "ws.append(['India','Asia'])          # add another row\n",
    "ws.append(['France','Europe'])       # add another row\n",
    "wb.save(\"countries.xlsx\")            # save the workbook into a file called countries.xlsx\n",
    "\n",
    "\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a python list of all locations for which you need to find the number of jobs postings.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "city_list=[ 'New York', 'San Francisco', 'Los Angeles']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import libraries required to create excel spreadsheet\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: openpyxl in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (3.1.3)\n",
      "Requirement already satisfied: et-xmlfile in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from openpyxl) (1.1.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# your code goes here\n",
    "%pip install openpyxl\n",
    "from openpyxl import Workbook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a workbook and select the active worksheet\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# your code goes here\n",
    "wb=Workbook()                        \n",
    "ws=wb.active "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the number of jobs postings for each of the location in the above list.\n",
    "Write the Location name and the number of jobs postings into the excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New York 143\n",
      "San Francisco 17\n",
      "Los Angeles 24\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[143, 17, 24]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code goes here\n",
    "def get_number_of_jobs_TL(technology, city):\n",
    "    city_jobs=[]\n",
    "    number_of_jobs=0\n",
    "    for location in city:\n",
    "        payload={\"Key Skills\":technology,\"Location\":location}\n",
    "        response=requests.get(api_url,params=payload)\n",
    "        if response.ok:\n",
    "            data=response.json()\n",
    "            number_of_jobs = len(data)\n",
    "            print(location, number_of_jobs)\n",
    "            city_jobs.append(number_of_jobs)\n",
    "            \n",
    "    return city_jobs\n",
    "    \n",
    "get_number_of_jobs_TL('Python',city_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save into an excel spreadsheet named 'job-postings.xlsx'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "ws.append([ 'City', 'Number of Python Jobs'])\n",
    "ws.append(['New York', '143'])\n",
    "ws.append(['San Francisco', '17'])\n",
    "ws.append(['Los Angeles', '24'])\n",
    "wb.save('job-postings.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In the similar way, you can try for below given technologies and results  can be stored in an excel sheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collect the number of job postings for the following languages using the API:\n",
    "\n",
    "*   C\n",
    "*   C#\n",
    "*   C++\n",
    "*   Java\n",
    "*   JavaScript\n",
    "*   Python\n",
    "*   Scala\n",
    "*   Oracle\n",
    "*   SQL Server\n",
    "*   MySQL Server\n",
    "*   PostgreSQL\n",
    "*   MongoDB\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      Technology  New York  San Francisco  Los Angeles\n",
      "0              C      1622            214          296\n",
      "1             C#        41              3            5\n",
      "2            C++        43              3            3\n",
      "3           Java       326             38           43\n",
      "4     JavaScript        51              7            7\n",
      "5         Python       143             17           24\n",
      "6          Scala         8              0            0\n",
      "7         Oracle        95             19           17\n",
      "8     SQL Server        36              2            3\n",
      "9   MySQL Server         0              0            0\n",
      "10    PostgreSQL         1              0            0\n",
      "11       MongoDB        25              2            2\n"
     ]
    }
   ],
   "source": [
    "# your code goes here\n",
    "technology_list=['C','C#','C++','Java','JavaScript','Python','Scala','Oracle','SQL Server','MySQL Server','PostgreSQL','MongoDB']\n",
    "def get_number_of_jobs_TL(technology, city):\n",
    "    job_list = []\n",
    "    for tech in technology:\n",
    "        number_of_jobs_list = [tech]\n",
    "        for location in city:\n",
    "            payload={\"Key Skills\": tech, \"Location\": location}\n",
    "            response=requests.get(api_url, params=payload)\n",
    "            if response.ok:\n",
    "                data=response.json()\n",
    "                number_of_jobs = len(data)\n",
    "                number_of_jobs_list.append(number_of_jobs)\n",
    "        job_list.append(number_of_jobs_list)\n",
    "    return job_list\n",
    "\n",
    "df=pd.DataFrame(data=get_number_of_jobs_TL(technology_list, city_list))\n",
    "column_names=['Technology','New York','San Francisco','Los Angeles']\n",
    "df.columns = column_names\n",
    "print(df)\n",
    "df.to_excel('job-postings.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Author\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ayushi Jain\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rav Ahuja\n",
    "\n",
    "Lakshmi Holla\n",
    "\n",
    "Malika\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Date (YYYY-MM-DD) | Version | Changed By        | Change Description                 |\n",
    "| ----------------- | ------- | ----------------- | ---------------------------------- | \n",
    "| 2022-01-19        | 0.3     | Lakshmi Holla        | Added changes in the markdown      |\n",
    "| 2021-06-25        | 0.2     | Malika            | Updated GitHub job json link       |\n",
    "| 2020-10-17        | 0.1     | Ramesh Sannareddy | Created initial version of the lab |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2022 IBM Corporation. All rights reserved. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}