Web_Scraping_Exercises_Jose_BootCamp.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    },
    "colab": {
      "name": "Web-Scraping-Exercises-Jose_BootCamp.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DebjitHore/Python-Codes/blob/master/Web_Scraping_Exercises_Jose_BootCamp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v2Oph896zfwK"
      },
      "source": [
        "___\n",
        "\n",
        "<a href='https://www.udemy.com/user/joseportilla/'><img src='https://github.com/DebjitHore/Complete-Python-3-Bootcamp/blob/master/Pierian_Data_Logo.png?raw=1'/></a>\n",
        "___\n",
        "<center><em>Content Copyright by Pierian Data</em></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7EPe2bgAzfwP"
      },
      "source": [
        "# Web Scraping Exercises \n",
        "\n",
        "## Complete the Tasks Below"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ksuSzgcOzfwR"
      },
      "source": [
        "**TASK: Import any libraries you think you'll need to scrape a website.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "OAkg6BV6zfwS"
      },
      "source": [
        "# CODE HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "ehz8Zpd1zfwU"
      },
      "source": [
        "import requests\n",
        "import bs4\n",
        "import lxml"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bbB525fzzfwV"
      },
      "source": [
        "**TASK: Use requests library and BeautifulSoup to connect to http://quotes.toscrape.com/ and get the HMTL text from the homepage.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "bm6JSduozfwW"
      },
      "source": [
        "# CODE HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "HY2tAZkJzfwX"
      },
      "source": [
        "results=  requests.get('http://quotes.toscrape.com/')"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "voo6nQX0zfwY"
      },
      "source": [
        "soup= bs4.BeautifulSoup(results.text, 'lxml')\n",
        "soup"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kx6Ddt90zfwa"
      },
      "source": [
        "**TASK: Get the names of all the authors on the first page.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "vuvPLz3Xzfwb"
      },
      "source": [
        "# CODE HERE\n",
        "authors=[]\n",
        "for item in soup.select('.author'):\n",
        "  authors.append(item.getText())\n",
        "authors=list(set(authors))\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NDgIMs55zfwc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2d790522-6536-4bf7-c6fb-ccf4ac1bbca5"
      },
      "source": [
        "authors"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Eleanor Roosevelt',\n",
              " 'Jane Austen',\n",
              " 'Steve Martin',\n",
              " 'J.K. Rowling',\n",
              " 'André Gide',\n",
              " 'Thomas A. Edison',\n",
              " 'Albert Einstein',\n",
              " 'Marilyn Monroe']"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeVF4CzZzfwc"
      },
      "source": [
        "**TASK: Create a list of all the quotes on the first page.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "y2GCBYKqzfwd"
      },
      "source": [
        "#CODE HERE\n",
        "quotes=[]\n",
        "for item in soup.select('.text'):\n",
        "  quotes.append(item.getText())\n",
        "quotes=list(set(quotes))\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q4gOiMZpzfwd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "47dc0edc-a855-4e5e-e088-fb0564b84f10"
      },
      "source": [
        "quotes"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"“I have not failed. I've just found 10,000 ways that won't work.”\",\n",
              " '“A day without sunshine is like, you know, night.”',\n",
              " \"“A woman is like a tea bag; you never know how strong it is until it's in hot water.”\",\n",
              " \"“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”\",\n",
              " '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',\n",
              " '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”',\n",
              " '“It is better to be hated for what you are than to be loved for what you are not.”',\n",
              " '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',\n",
              " '“Try not to become a man of success. Rather become a man of value.”',\n",
              " '“It is our choices, Harry, that show what we truly are, far more than our abilities.”']"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6qJZfl4zfwe"
      },
      "source": [
        "**TASK: Inspect the site and use Beautiful Soup to extract the top ten tags from the requests text shown on the top right from the home page (e.g Love,Inspirational,Life, etc...). HINT: Keep in mind there are also tags underneath each quote, try to find a class only present in the top right tags, perhaps check the span.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "9EOsOwvuzfwf"
      },
      "source": [
        "# CODE HERE\n",
        "tags=[]\n",
        "for item in soup.select('.tag-item'):\n",
        "  tags.append(item.getText())\n"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DJZ8Joyozfwf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f3e74101-5077-4d42-a04d-3448b3f78b78"
      },
      "source": [
        "for item in tags:\n",
        "  print(item)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "love\n",
            "\n",
            "\n",
            "inspirational\n",
            "\n",
            "\n",
            "life\n",
            "\n",
            "\n",
            "humor\n",
            "\n",
            "\n",
            "books\n",
            "\n",
            "\n",
            "reading\n",
            "\n",
            "\n",
            "friendship\n",
            "\n",
            "\n",
            "friends\n",
            "\n",
            "\n",
            "truth\n",
            "\n",
            "\n",
            "simile\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JgjulSYYzfwg"
      },
      "source": [
        "**TASK: Notice how there is more than one page, and subsequent pages look like this http://quotes.toscrape.com/page/2/. Use what you know about for loops and string concatenation to loop through all the pages and get all the unique authors on the website. Keep in mind there are many ways to achieve this, also note that you will need to somehow figure out how to check that your loop is on the last page with quotes. For debugging purposes, I will let you know that there are only 10 pages, so the last page is http://quotes.toscrape.com/page/10/, but try to create a loop that is robust enough that it wouldn't matter to know the amount of pages beforehand, perhaps use try/except for this, its up to you!**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "AE5ymemMzfwg"
      },
      "source": [
        "# CODE HERE\n",
        "base_url= ' http://quotes.toscrape.com/page/'\n",
        "n=1\n",
        "authors=set()"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z8tG1v6k5CKk"
      },
      "source": [
        "while True:\n",
        "  page_url= base_url+str(n)\n",
        "  n+=1\n",
        "  res= requests.get(page_url)\n",
        "  if \"No quotes found!\" in res.text:\n",
        "    break\n",
        "  soup=bs4.BeautifulSoup(res.text, 'lxml')\n",
        "  \n",
        "  # CODE HERE\n",
        "  for item in soup.select('.author'):\n",
        "    authors.add(item.getText())"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bhc7KNHx8sJC",
        "outputId": "008d0333-64fc-4e60-8ac6-a5ee6e0de828"
      },
      "source": [
        "authors"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Albert Einstein',\n",
              " 'Alexandre Dumas fils',\n",
              " 'Alfred Tennyson',\n",
              " 'Allen Saunders',\n",
              " 'Ayn Rand',\n",
              " 'Bob Marley',\n",
              " 'C.S. Lewis',\n",
              " 'Charles Bukowski',\n",
              " 'Charles M. Schulz',\n",
              " 'Douglas Adams',\n",
              " 'Dr. Seuss',\n",
              " 'E.E. Cummings',\n",
              " 'Eleanor Roosevelt',\n",
              " 'Elie Wiesel',\n",
              " 'Ernest Hemingway',\n",
              " 'Friedrich Nietzsche',\n",
              " 'Garrison Keillor',\n",
              " 'George Bernard Shaw',\n",
              " 'George Carlin',\n",
              " 'George Eliot',\n",
              " 'George R.R. Martin',\n",
              " 'Harper Lee',\n",
              " 'Haruki Murakami',\n",
              " 'Helen Keller',\n",
              " 'J.D. Salinger',\n",
              " 'J.K. Rowling',\n",
              " 'J.M. Barrie',\n",
              " 'J.R.R. Tolkien',\n",
              " 'James Baldwin',\n",
              " 'Jane Austen',\n",
              " 'Jim Henson',\n",
              " 'Jimi Hendrix',\n",
              " 'John Lennon',\n",
              " 'Jorge Luis Borges',\n",
              " 'Khaled Hosseini',\n",
              " \"Madeleine L'Engle\",\n",
              " 'Marilyn Monroe',\n",
              " 'Mark Twain',\n",
              " 'Martin Luther King Jr.',\n",
              " 'Mother Teresa',\n",
              " 'Pablo Neruda',\n",
              " 'Ralph Waldo Emerson',\n",
              " 'Stephenie Meyer',\n",
              " 'Suzanne Collins',\n",
              " 'Terry Pratchett',\n",
              " 'W.C. Fields',\n",
              " 'William Nicholson'}"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cZ38SUQbzfwh"
      },
      "source": [
        "There are lots of other potential solutions that are even more robust and flexible, the main idea is the same though, use a while loop to cycle through potential pages and have a break condition based on the invalid page."
      ]
    }
  ]
}