BeautifulSoup-Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computitional Social Science - Assignment 2\n",
    "## Name: Daniyal Saleem\n",
    "## Matriculation Number: 219203408\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 1.1\n",
    "- Collect the name of all the countries per continent from English Wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "page = requests.get(\"https://simple.wikipedia.org/wiki/List_of_countries_by_continents\")\n",
    "from bs4 import BeautifulSoup #library for parsing HTML code\n",
    "soup = BeautifulSoup(page.content, 'html.parser') \n",
    "# out = open(\"wiki_Countries.html\", \"w\", encoding=\"utf-8\")\n",
    "# out.write(str(soup))\n",
    "# out.close()\n",
    "\n",
    "## Read in the HTML file\n",
    "# soup = BeautifulSoup(open(\"wiki_Countries.html\", encoding='utf-8'), \"html.parser\")\n",
    "\n",
    "liContinents = soup.find_all(\"li\",{\"class\":\"toclevel-1\"})  # getting all li of continetns\n",
    "# print(liContinents[:7])\n",
    "\n",
    "continents_list=[]\n",
    "continents_list_byContinent=[]\n",
    "\n",
    "for item in liContinents[:7]: #only traversing starting 7 items i.e  only continents\n",
    "    q=item.find(\"a\") # getting first anchor tag of each li\n",
    "    href=q.get(\"href\") # getting href attribute  of each anchor tag\n",
    "    continents_list.append(href[1:])\n",
    "# print(continents_list)\n",
    "\n",
    "#For every continent get list of countries\n",
    "all_uls = soup.find(\"div\",{\"class\":\"mw-parser-output\"}).find_all(\"ul\", recursive=False)\n",
    "# print(all_uls)\n",
    "for ul in all_uls[:7]: #each individual ul contains list of countries of single continent\n",
    "    countries_list=[]  #populating countries of individual continent \n",
    "    for li in ul.find_all('li', recursive=False):\n",
    "        a = li.find('a') # getting anchor tag of individual country line item\n",
    "        if(a):\n",
    "            country_name = a[\"title\"]\n",
    "        countries_list.append(country_name)\n",
    "    continents_list_byContinent.append(countries_list)\n",
    "\n",
    "continents_list_byContinent.insert(1, []) # adjusting list because ther is no country list (i.e. no ul) in Antarctica\n",
    "\n",
    "# merging last two lists because there are two lists (i.e. two ul;s) in Oceania/Australia continent\n",
    "continents_list_byContinent[-2]= continents_list_byContinent[-2] + continents_list_byContinent[-1] \n",
    "\n",
    "del continents_list_byContinent[-1] # deleting extra last list after merging\n",
    "# print(continents_list_byContinent)\n",
    "\n",
    "dict_countries_by_Continents = {}\n",
    "for i,item in enumerate(continents_list):\n",
    "    dict_countries_by_Continents[continents_list[i]] = continents_list_byContinent[i]\n",
    "\n",
    "dict_countries_by_Continents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 1.2\n",
    "- Create countries-continents dataframe, this dataframe should have two columns: country , continent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame(columns=['Country','Continent'])\n",
    "for continent in dict_countries_by_Continents.keys():\n",
    "    for country in dict_countries_by_Continents[continent]:\n",
    "        df = df.append({\n",
    "             \"Country\":  country,\n",
    "             \"Continent\": continent\n",
    "              }, ignore_index=True)\n",
    "\n",
    "df.rename(columns={'Country': 'Country_or_region'}, inplace=True)\n",
    "df\n",
    "# df.to_csv('x.csv', sep=',')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 1.3\n",
    "- Collect the happiness score, GDP per capital, social support, healthy\n",
    "life expectancy, freedom to make life choices, generosity, and perceptions of corruption per country in 2019 from English Wikipedia and\n",
    "put all collected information in a dataframe ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page2 = requests.get(\"https://en.wikipedia.org/wiki/World_Happiness_Report#2019_report\")\n",
    "soup2 = BeautifulSoup(page2.content, 'html.parser') \n",
    "\n",
    "table = soup2.find(\"table\",{\"class\":\"wikitable\"}) #getting the main table (2019 Report)\n",
    "df2 = pd.read_html(str(table))[0] # Pandas method to convert html table to a dataframe:\n",
    "df2.columns = [c.replace(' ', '_') for c in df2.columns]\n",
    "df2\n",
    "\n",
    "# table = soup.find(\"table\",{\"class\":\"wikitable\"}) #getting the main table\n",
    "# table_rows = table.find_all('tr')\n",
    "\n",
    "# l []\n",
    "# for tr in table_rows:\n",
    "# #     print(tr)\n",
    "#     td = tr.find_all('td')\n",
    "#     row = [Cell.text for Cell in td]\n",
    "#     l.append(row)\n",
    "# pd.DataFrame(l, columns=[\"Rank\",\"Country\", \"Happiness Score\", \"GDP per Capital\", \"Social Support\", \"Healthy Life Expectancy\", \"Freedom to make Life Choices\", \"Generosity\",\"Perceptions of Corruption\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 1.4\n",
    "- Create a new dataframe with all the information that you collected\n",
    "and save it in a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# result = df.append(df2, sort=False)\n",
    "\n",
    "result=pd.merge(df, df2, on='Country_or_region')\n",
    "result=result.sort_values(by=['Overall_rank']) # sort by Hapiness Score\n",
    "\n",
    "result.to_csv('All_Info.csv', sep=',')\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 2.1\n",
    "- Plot the distribution of happiness score per country."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "data=result\n",
    "Countries =data['Country_or_region']\n",
    "H_Score = data['Score']\n",
    "\n",
    "plt.figure(figsize=(12,35))\n",
    "\n",
    "plt.barh(Countries, H_Score,height=0.4,align='edge')\n",
    "plt.ylabel(\"Countries\")\n",
    "plt.xlabel(\"Happiness Score\")\n",
    "plt.title(\"Distribution of Happiness Score Per Country.\")\n",
    "    \n",
    "plt.margins(y=0.005)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 2.2\n",
    "- Find the top 10 happiest countries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.sort_values(by=['Score']) # sort by Hapiness Score\n",
    "data.head(10) # Display top 10 Happiest Countries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 2.3\n",
    "- What is the average of happiness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting the average value of Happiness Score Column\n",
    "avg_Happiness = data[\"Score\"].mean() \n",
    "print(\"The average of Happiness is\" , avg_Happiness)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 2.4\n",
    "- What are the countries below the median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "median_Happiness = data[\"Score\"].median() # getting the median value of Happiness Score Column\n",
    "\n",
    "# Create variable with TRUE if Score is less than the Median of Score\n",
    "less_than_median = data['Score'] < median_Happiness\n",
    "\n",
    "#Selecting all row where Happiness Score is less than the Median of Score\n",
    "data[less_than_median]['Country_or_region']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 2.5\n",
    "- Plot the correlation between different variables and give an interpretation. \n",
    "- For each continent, give the happiest country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# correlation between Score and GDP_per_capita\n",
    "\n",
    "x1=data[\"Score\"]\n",
    "y1=data[\"GDP_per_capita\"]\n",
    "\n",
    "plt.scatter(x1, y1)\n",
    "plt.title(\"correlation between GDP_per_capita and Happiness Score.\")\n",
    "plt.ylabel(\"GDP_per_capita\")\n",
    "plt.xlabel(\"Happiness Score\")\n",
    "plt.show()\n",
    "\n",
    "print(\"Interpretation: There is weak positive correlation between Score and GDP_per_capita variables.\") \n",
    "print(\"It means that one variable will increase if we increase the other variable but the the relationship is not very strong.\")\n",
    "\n",
    "\n",
    "print(\" \\nrefference: https://www.verywellmind.com/what-is-correlation-2794986\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For each continent, give the happiest country\n",
    "\n",
    "# max happines score of country by continent \n",
    "happiest_Country = data.groupby(['Continent'])['Score'].transform(max) == data['Score']\n",
    "data[happiest_Country]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}