b_data_collection_methods/3_dynamic_web_scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "26acf247",
   "metadata": {},
   "source": [
    "<img src='images/gesis.png' style='height: 50px; float: left'>\n",
    "<img src='images/social_comquant.png' style='height: 50px; float: left; margin-left: 40px'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6d67c10",
   "metadata": {},
   "source": [
    "## Introduction to Computational Social Science methods with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "494a8570",
   "metadata": {},
   "source": [
    "# Session B3: Dynamic web scraping\n",
    "\n",
    "In the previous [Session B2](2_data_parsing_and_static_web_scraping.ipynb), we have introduced [Data parsing and static web scraping](2_data_parsing_and_static_web_scraping.ipynb). To reiterate, the idea behind web scraping is to extract content from web pages by parsing their semi-structured HTML source code. This is fairly simple for pages whose content is generated statically, that is, whose content is entirely stored in the source code. The Beautiful Soup package is a user-friendly package that allows for fast learning and beginning static web scraping in an efficient way.\n",
    "\n",
    "However, many websites change their content in a browsing session. Content changes can occur on the client side (your side) that do not necessarily change the website's source code but change its appearance, such as expanding text boxes. In that case, classical scraping methods are sufficient for grabbing the entire text because the displayed truncated version of a text might already be entirely stored in the source code. But changes can also occur on the server side. For example, when you scroll to the bottom of a page and more content is being displayed, the source code itself changes. Such content is generated dynamically by users interacting with websites and is hard to capture with data collection methods used for static websites. This is when you need **dynamic web scraping**.\n",
    "\n",
    "Dynamic websites change their content (*i.e.*, source code) due to various reasons, such as:\n",
    "\n",
    "- clicking, scrolling, mouse hovering  \n",
    "- screen sizes, languages (IP-based), devices, time of day \n",
    "- previous visits (user's browsing history) \n",
    "- and more...\n",
    "\n",
    "<img src='images/selenium.png' style='height: 100px; float: right; margin-left: 100px'>\n",
    "\n",
    "Usually, user interactions are registered and source code is updated via code pieces of JavaScript. Besides HTML, the programming language [JavaScript](https://en.wikipedia.org/wiki/JavaScript) is one of the core technologies of the web. Website interactions that lead to content changes are often challenging or not obtainable through classical scraping approaches since they require JavaScript executions initiated by user interactions. Thus, other methods such as browser automation tools are needed to help us imitate user interactions and make dynamic web scraping possible. [Selenium](https://www.selenium.dev/) is such a browser automation software. We can use Selenium to control every major web browser such as Chrome, Firefox, or Edge. Actions are not limited to loading web pages, we can also perform other actions that allow for better interaction with the websites such as mouse clicks, handling pop-up windows, or filling forms.\n",
    "\n",
    "Beautiful Soup made for a new experience in Jupyter Notebooks and Python: you had a webpage in another browser tab and had to inspect its source code to build a scraper. Now, with Selenium, we will go one step further: we will actually control the webpage from within the Jupyter Notebook. Beautiful Soup can then be used on top of Selenium. Dynamic web scraping is typically discussed in the literature as a complication of static web scraping, for example, by Bosse *et al.* (2022).\n",
    "\n",
    "<div class='alert alert-block alert-success'>\n",
    "<b>In this session</b>, \n",
    "\n",
    "you will learn how to collect webpage content that is dynamically generated. In subsession **B3.1**, we will get to know the Selenium package and learn how to automate web browsing with it. In subsession **B3.2**, we will work on practical examples: scraping questions and answers from the Quora platform.\n",
    "</div>\n",
    "\n",
    "<div class='alert alert-block alert-danger'>\n",
    "<b>Caution</b>\n",
    "\n",
    "This Jupyter Notebook demonstrates a workflow that consists of a **sequence of processing steps**. The notebook must be executed from top to bottom. Going back up from a certain code cell and trying to execute a cell that precedes it may not work.\n",
    "</div>\n",
    "\n",
    "<div class='alert alert-block alert-warning'>\n",
    "<b>Additional resources</b>\n",
    "\n",
    "If you would like to explore web scraping further, [Scrapy](https://scrapy.org/) can be your next address for more complex web tasks. It uses less memory and CPU storage and supports data extraction from HTML sources as well. We can even extend its functionality. As we mentioned, it is great a great library for complex and larger projects and we can easily transfer existing projects into another project.\n",
    "</div>\n",
    "\n",
    "\"If the approaches we’ve covered thus far\n",
    "won’t work (which can happen when a website is dynamically generated or interactive, for instance), then we’ll have to call in the cavalry. In this case, the cavalry is a Python package called selenium. Since my editors at Sage feel it would be best if one could carry this book without the assistance of a hydraulic lift, we’re not going to have room to cover selenium in-text. If you want to read more about how to scrape the interactive web, we’ve prepared an **online supplement** that will guide you through the process.\" (McLevey)\n",
    "\n",
    "Also Patel (2020)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6182d3a1",
   "metadata": {},
   "source": [
    "## B3.1. Introducing Selenium \n",
    "\n",
    "### 3.1.1. Setting up basic configurations\n",
    "\n",
    "Selenium is a [browser automation software](https://www.selenium.dev/) that can interface with many different browser types and programming languages. Thus, we can write programming scripts that control the browser and imitate our behavior, such as clicking or scrolling. Before we can start writing a programming script, we need to set up Selenium by downloading a driver. Depending on the browser we want to use (e.g., Firefox, Chrome), we need a different driver, which could be found [here](https://www.selenium.dev/downloads/). In this notebook, we will go through instructions for using both Google Chrome and Mozilla Firefox, for which you can find the drivers [here](https://chromedriver.chromium.org/downloads) and [here](https://github.com/mozilla/geckodriver/releases), respectively. To download the correct driver, you need to know which operating system (e.g. Windows, Linux, Mac) your machine runs on, and which browser version you have. For Chrome, you can find the browser version under Settings > About Chrome (see screenshot below):\n",
    "\n",
    "<img src='images/chrome_version.JPEG' width=\"1000\" height=\"1000\" align=\"center\"/>\n",
    "\n",
    "Download the correct driver, and after unpacking the zip folder, place the *.exe* driver file in the same folder we are running this script.\n",
    "\n",
    "The Selenium webpage contains documentation for all the programming languages, which you can find [here](https://www.selenium.dev/documentation/). However, the documentation is not as concise, and since we are using python, we can also find a separate documentation [here](https://selenium-python.readthedocs.io/).\n",
    "\n",
    "Both documentations are very handy and should be kept close when working with Selenium. When you inspect the documentation, you will recognize that besides sending specific behavioral commands to the browser, accessing web elements is very similar to other approaches, such as beautiful soup. You will need XPATH, CSS selectors, and other properties of web elements to interact with them.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc61e0c3",
   "metadata": {},
   "source": [
    "### B3.1.2. Extracting relavent information from dynamic webpages\n",
    "\n",
    "<img src='images/quora_logo.png' style='height: 120px; float: right; margin-left: 10px' >\n",
    "\n",
    "As we already discussed in the section of extracting relavent information from static webpages, each and every project might require data from different sources by web scarping to answer some of our research questions. Or simply, we might like to make some tasks of our lives easier and faster by automated web scraping compared to manual browsing or scrolling; such as searching jobs on internet, finding your favorite bands' histories, or reading different minds over various questions. \n",
    "\n",
    "\n",
    "## B3.2. Getting practical with on Quora\n",
    "\n",
    "Along with the last example, let's explore scraping [Quora](www.quora.com), which is a social question-and-answer website where users can collaborate by editing questions and commenting on answers that have been submitted by other users. \n",
    "\n",
    "In this section, we will showcase how you can use Selenium with Chrome/Firefox to collect data from the dynamic website Quora. Before collecting data, we need to check whether we are allowed to collect data from the website. Quora states that we are permitted to employ scrapers but must adhere to the [robots.txt](https://www.quora.com/robots.txt), which specifies the allowed and disallowed contents for scraping, and that we make ourself known to the website so that they can contact us if they want to. We can give Quora our contact information by adding them to the user-agent, the information the browser sends to the website. *Section 4-d: Permitted uses of Quora’s terms of service* specifies the rules for scraping (see the screenshot below):\n",
    "\n",
    "<img src='images/quora.png' width=\"700\" height=\"700\" align=\"center\"/>\n",
    "\n",
    "### B3.2.1 Example: Scraping posts\n",
    "\n",
    "Before scraping any information, we need to create an account on Quora. We would recommend creating a new account for your scraping project. Go to www.quora.com and create a new account.\n",
    "\n",
    "After creating the account, make sure to import all the necessary libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9825d4ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd # to work with data frames; you may have already imported it in this notebook\n",
    "from time import sleep # to slow down our scraper\n",
    "\n",
    "# all selenium specific packages:\n",
    "from selenium import webdriver # to load the browser\n",
    "from selenium.webdriver.common.keys import Keys # necessary to automate typings, like filling out the forms\n",
    "from selenium.webdriver.common.by import By # necessary to search for web elements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba8d142b",
   "metadata": {},
   "source": [
    "In case you are using Chrome, import the first line in the next cell, if it is Firefox, import the second one (note that if you import both of them at the same time, it will only work for the last one- Firefox!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69b20c82",
   "metadata": {},
   "outputs": [],
   "source": [
    "#from selenium.webdriver.chrome.options import Options # necessary to change our user agent when working with Chrome\n",
    "\n",
    "from selenium.webdriver.firefox.options import Options # necessary to change our user agent when working with Firefox"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2191cfe4",
   "metadata": {},
   "source": [
    "We can now start the driver (i.e., the browser), which should appear as a separate window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e10e4999",
   "metadata": {},
   "outputs": [],
   "source": [
    "# starting the driver\n",
    "\n",
    "# For Chrome:\n",
    "#driver = webdriver.Chrome()\n",
    "\n",
    "# For Firefox:\n",
    "driver = webdriver.Firefox()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e266b822",
   "metadata": {},
   "source": [
    "As you can see, a new browser window opens, which is *being controlled by automated test software:*\n",
    "\n",
    "<img src='images/chrome.png' width=\"700\" height=\"700\" align=\"center\"/>\n",
    "\n",
    "For Firefox, it looks something like this:\n",
    "\n",
    "<img src='images/firefox.png' width=\"700\" height=\"700\" align=\"center\"/>\n",
    "\n",
    "The *driver* instance is the browser we will use to navigate the website and find web elements. \n",
    "We can now check what our user-agent for our browser is with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80a2f36a",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = driver.execute_script(\"return navigator.userAgent\")\n",
    "print(agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73dd7706",
   "metadata": {},
   "source": [
    "We can change the user-agent information to make ourself identifiable, and Quora can contact us if they want. We need to initiate a new driver with the changed information. Hence, we first need to quit our current session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8ccc18d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# quit current session\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "782503e7",
   "metadata": {},
   "source": [
    "Make sure to run the correct line of code when restarting the driver; in the middle two lines, the first line is for Chrome and second one (which is commented by default) is for Firefox."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "972aa7fc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# adding our e-mail address to the user-agent\n",
    "opts = Options()\n",
    "opts.add_argument(\"user-agent=Getting news feed data; contact me through: [e-mail address]\")\n",
    "\n",
    "\n",
    "#driver = webdriver.Chrome(options=opts)# initaite driver with new user-agent for Chrome\n",
    "driver = webdriver.Firefox(options=opts)# initaite driver with new user-agent for Firefox\n",
    "\n",
    "# lets check if we changed our user-agent\n",
    "agent = driver.execute_script(\"return navigator.userAgent\")\n",
    "print(agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01247368",
   "metadata": {},
   "source": [
    "If we all have come to this point, now, we can start with our new project with scraping [Quora](https://www.quora.com/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e414312b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# url to visit\n",
    "url_search = \"https://www.quora.com/\"\n",
    "# go to url\n",
    "driver.get(url_search)\n",
    "sleep(1.5) # set sleep time for 1.5 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e138c05",
   "metadata": {},
   "source": [
    "We will set some pauses occasionally to slow down the scraping process and give the browser some time to load the website. Next, we want to #to the website. With Selenium, we can automate the step and fill in all the text fields. \n",
    "\n",
    "Similarly, when working with other scarping approaches, we need to find the web elements by inspecting the HTML structure of the website and locating them through their paths, class, or names. Ideally, the elements have an ID we can identify them with, as in the case of the e-mail address and password fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08ed5fe6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# providing log-in credentials to the website\n",
    "\n",
    "EMail_field = driver.find_element(By.XPATH, '//*[@id=\"email\"]') # Find e-mail field\n",
    "# specify your e-mail address\n",
    "my_email = \"ENTER YOUR E-MAIL ADDRESS\"\n",
    "\n",
    "EMail_field.send_keys(my_email) # sending the string to the e-mail field\n",
    "sleep(1.5)\n",
    "\n",
    "PW_field = driver.find_element(By.XPATH, '//*[@id=\"password\"]') # find password field\n",
    "# specify your password \n",
    "my_password = \"ENTER YOUR PASSWORD\"\n",
    "\n",
    "PW_field.send_keys(my_password) # sending the string to the password field\n",
    "sleep(1.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cae5a80",
   "metadata": {},
   "source": [
    "After we fill in all our information, we can find the log-in button and click on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92ef6afe",
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_elements(By.CLASS_NAME, \"iyYUZT\")[4].click()\n",
    "sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d17aa4e",
   "metadata": {},
   "source": [
    "<div class='alert alert-block alert-danger'>\n",
    "<b>Caution</b>\n",
    "    \n",
    "In case you encounter a recaptcha, you can click on the check box and then log in.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f3ad65",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Clicking on the checkbox\n",
    "driver.find_elements(By.CSS_SELECTOR, \"div.qu-mb--medium\")[3].click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6717f20a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Logging in\n",
    "driver.find_elements(By.CLASS_NAME, \"iyYUZT\")[4].click()\n",
    "sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "239a6929",
   "metadata": {},
   "source": [
    "After we log into our account, we can see the cookie notification. We can also interact with pop-ups and accept or reject them. We will reject the cookies by finding the *Reject All* button and clicking on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a180b39",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rejecting cookies\n",
    "driver.find_element(By.ID, \"onetrust-reject-all-handler\").click()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "780855c4",
   "metadata": {},
   "source": [
    "<div class='alert alert-block alert-info'>\n",
    "<b>Insight</b>\n",
    "\n",
    "Your website might not be in English, depending on the region you are accessing Quora from.\n",
    "\n",
    "In our case, we are accessing the website from Germany. However, through the language settings at the top of the website, we can change the language to English. We also can automate that step as in the following cell.\n",
    "    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bae26a60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# click on menu\n",
    "driver.find_elements(By.CLASS_NAME, \"puppeteer_popper_reference\")[1].click()\n",
    "# click to select \"English\"\n",
    "driver.find_element(By.CSS_SELECTOR, \"div.qu-dynamicFontSize--button\").click()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15732cb6",
   "metadata": {},
   "source": [
    "Next, we want to collect some of the information present in our news feed. We need to find the container for the entire feed to collect individual posts, answers, or questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da3c8eda",
   "metadata": {},
   "outputs": [],
   "source": [
    "# access whole feed containing various forms of posts/questions/answers etc.\n",
    "NewsFeed = driver.find_element(By.CLASS_NAME, \"dom_annotate_multifeed_home\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16178dee",
   "metadata": {},
   "source": [
    "When we inspect the structure of the news feed, we can see that posts, questions, answers, or advertisements have different classes. Thus, we can leverage the class names to access the information we are interested in. For this guide, we only want to collect data from answered questions, which are the predominant elements in our feed.\n",
    "\n",
    "By inspecting the website, we know that one element with the class \"dom_annotate_multifeed_bundle_AnswersBundle\" contains the answers we are interested in.\n",
    "\n",
    "Let's have a look at the first answer in our feed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca83900e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# find all answers\n",
    "Answers = NewsFeed.find_elements(By.CLASS_NAME, \"dom_annotate_multifeed_bundle_AnswersBundle\")\n",
    "# select first answer and print all containing texts\n",
    "answer1 = Answers[0]\n",
    "print(answer1.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e02efd3",
   "metadata": {},
   "source": [
    "The text shows that we have several different elements, which are separated by a new line (\\n).\n",
    "The first separated element seems to be the author's name. We can also spot the story title and the number of shares and comments at the end of the string.\n",
    "\n",
    "Since newline characters in the current string separate all the information, we could just split the text by \"\\n\" and deduce by the order what each text section associates to which information. However, it is likely, that some answers will contain more or less data points, making the order of split elements not generalizable. Thus, a better way to obtain each piece of information is by selecting their elements.\n",
    "\n",
    "For example, we can find the title of the answer through the class name \"qu-userSelect--text\" with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5ea1677",
   "metadata": {},
   "outputs": [],
   "source": [
    "title = answer1.find_element(By.CLASS_NAME, \"qu-userSelect--text\").text\n",
    "print(\"the title is: \", title)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cb7beb9",
   "metadata": {},
   "source": [
    "Similarly, we can find the number of shares or comments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53726a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "shares = answer1.find_element(By.CLASS_NAME, \"dom_annotate_answer_action_bar_share\").text\n",
    "comments = answer1.find_element(By.CLASS_NAME, \"dom_annotate_answer_action_bar_comment\").text\n",
    "\n",
    "print(\"number of shares: \", shares)\n",
    "print(\"number of comments: \", comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5b73f95",
   "metadata": {},
   "source": [
    "The numbers of shares and comments are still a string, but we can convert them into integers when we save the data in a data frame.\n",
    "\n",
    "Since we saved all answers of our feed in one variable, we can loop over it and extract all the information we are interested in. To make this process easier, we can define a function that extracts the information from an answer element and returns a list with the information we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2502cf3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining the function to extract information from each answer post\n",
    "def get_post_info(PostInfo):\n",
    "    AuthorInfo = PostInfo.find_element(By.CLASS_NAME, \"qu-alignItems--flex-start\") # Container for author information \n",
    "    authorName = AuthorInfo.find_element(By.CLASS_NAME, \"qu-wordBreak--break-word\").text # author name\n",
    "    authorLink = AuthorInfo.find_element(By.TAG_NAME, \"a\").get_attribute(\"href\") # author link\n",
    "   \n",
    "    title = PostInfo.find_element(By.CLASS_NAME, \"qu-userSelect--text\").text # answer title\n",
    "    \n",
    "    StoryLinkContainer = PostInfo.find_element(By.CLASS_NAME, \"qu-mb--tiny\").find_element(By.TAG_NAME, \"a\") # container for story link \n",
    "    StoryLink = StoryLinkContainer.get_attribute(\"href\") # get story link\n",
    "    \n",
    "    upvotes = PostInfo.find_element(By.CLASS_NAME, \"dom_annotate_answer_action_bar_upvote\").text # number of upvotes\n",
    "    shares = PostInfo.find_element(By.CLASS_NAME, \"dom_annotate_answer_action_bar_share\").text # number of shares\n",
    "    comments = PostInfo.find_element(By.CLASS_NAME, \"dom_annotate_answer_action_bar_comment\").text # number of comments\n",
    "    \n",
    "    # aggregate the answer data into a list\n",
    "    post_info = [authorName, authorLink, title, StoryLink, upvotes, shares, comments]\n",
    "    return post_info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b04affc3",
   "metadata": {},
   "source": [
    "Now, we need a loop to save the data into another list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc12fca9",
   "metadata": {},
   "outputs": [],
   "source": [
    "AnswersInfo = [] # initiate list to save data\n",
    "for post in Answers: # loop over all answers\n",
    "    AnswersInfo.append(get_post_info(post)) # add answer info to the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecc013a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets have a look at the number of answers we gathered:\n",
    "len(AnswersInfo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84118b6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "AnswersInfo[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31069acc",
   "metadata": {},
   "source": [
    "We collected only very few answers. Why is that?\n",
    "\n",
    "Because the website only loaded very few answers, and we need to scroll down on the website to generate more content. \n",
    "Luckily, Selenium can help us!\n",
    "\n",
    "We have several methods to imitate scrolling:\n",
    "\n",
    "   - scrolling by pixel \n",
    "   - scrolling to the bottom of the page\n",
    "\n",
    "Let's start with the first one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "503cdaf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. scoll down incrementally\n",
    "driver.execute_script(\"window.scrollTo(0, 1000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 2000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 3000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 4000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 5000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 6000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 7000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 8000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 9000)\")\n",
    "sleep(1) \n",
    "driver.execute_script(\"window.scrollTo(0, 10000)\")\n",
    "sleep(1) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aa72459",
   "metadata": {},
   "source": [
    "We can also use another approach, but first, we can scroll to the top of the page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c1c2053",
   "metadata": {},
   "outputs": [],
   "source": [
    "# to the top of the page\n",
    "driver.execute_script(\"window.scrollTo(0, -document.body.scrollHeight);\")\n",
    "sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc8bdfed",
   "metadata": {},
   "source": [
    "For option two, we scroll to the bottom of the page, indicated by the document height of the website. Similarly, as before, we can repeat that process to load more content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44d95482",
   "metadata": {},
   "outputs": [],
   "source": [
    "# to the bottom of the page - here we give the browser a bit more time to load all the content\n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2)\n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2) \n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2) \n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2) \n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2)\n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2)\n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2) \n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2) \n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2) \n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "sleep(2) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69d88e97",
   "metadata": {},
   "source": [
    "Of course, we can also implement loops for each of those processes, but for now, we loaded enough content.\n",
    "\n",
    "To grab the newly loaded content, we need to find the newsfeed again and find all answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5349940",
   "metadata": {},
   "outputs": [],
   "source": [
    "NewsFeed = driver.find_element(By.CLASS_NAME, \"dom_annotate_multifeed_home\")\n",
    "Answers = NewsFeed.find_elements(By.CLASS_NAME, \"dom_annotate_multifeed_bundle_AnswersBundle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4191270",
   "metadata": {},
   "source": [
    "Finally, we can re-run our loop to grab all the info."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b46d3f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "AnswersInfo = []\n",
    "for post in Answers:\n",
    "    AnswersInfo.append(get_post_info(post))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3639a670",
   "metadata": {},
   "source": [
    "Let's check the number of answers we collected this time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a59bebc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(Answers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6da6014f",
   "metadata": {},
   "source": [
    "Better! If we want to collect more data, we can implement more scrolling, but we collected enough information for demonstration purposes now.\n",
    "\n",
    "Next, we can convert the list into a data frame, making it easier for us to work with the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61f1e947",
   "metadata": {},
   "outputs": [],
   "source": [
    "AnswersInfo_df = pd.DataFrame(AnswersInfo, columns=[\"author\", \"author_link\",\n",
    "                                                    \"title\", \"story_link\",\n",
    "                                                    \"num_upvotes\", \"num_shares\", \n",
    "                                                    \"num_comments\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e523e400",
   "metadata": {},
   "outputs": [],
   "source": [
    "AnswersInfo_df.head(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15c9ae48",
   "metadata": {},
   "source": [
    "Once we are done working with the driver, we can close the current session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebf1c526",
   "metadata": {},
   "outputs": [],
   "source": [
    "# close driver\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b2f254f",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Hint:</b> \n",
    "    \n",
    "We can do similar operations to  gather advertisements, posts, or questions information by accessing other classes, such as:\n",
    "- \"dom_annotate_multifeed_bundle_AdBundle\" for Ads\n",
    "- \"dom_annotate_multifeed_bundle_PostBundle\" for posts\n",
    "\n",
    "However, our function for accessing author and post information might have to be adapted for those classes.\n",
    "\n",
    "It is also worth noting that instead of looking at the browser and its behavior, we can also implement a headless browser that will function in the background without us seeing it (have a look at [this Stackoverflow link](https://stackoverflow.com/questions/53657215/running-selenium-with-headless-chrome-webdriver). You can specify those settings at the beginning with options like `options.add_argument(\"--headless\")`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2833822",
   "metadata": {},
   "source": [
    "### B3.2.2 Example: Searching for a specific question and gathering all answers to that question on Quora\n",
    "\n",
    "In this example, we want to log into quora, search for a specific question, click on it, specify all answers to it and then scrape them. We will do this using Google Chrome.\n",
    "\n",
    "Like before, we begin with starting the driver:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dce749d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# adding our e-mail address to the user-agent\n",
    "opts = Options()\n",
    "opts.add_argument(\"user-agent=Getting answers data; contact me through: [e-mail address]\")\n",
    "\n",
    "#driver = webdriver.Chrome(options=opts)# initaite driver with new user-agent for Chrome\n",
    "driver = webdriver.Firefox(options=opts)# initaite driver with new user-agent for Firefox\n",
    "\n",
    "# lets check if we changed our user-agent\n",
    "agent = driver.execute_script(\"return navigator.userAgent\")\n",
    "print(agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef83af46",
   "metadata": {},
   "source": [
    "Then we go to the website:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69338044",
   "metadata": {},
   "outputs": [],
   "source": [
    "# url to visit\n",
    "url_search = \"https://www.quora.com/\"\n",
    "# go to url\n",
    "driver.get(url_search)\n",
    "sleep(1.5) # set sleep time for 1.5 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e16b8433",
   "metadata": {},
   "source": [
    "We enter our credentials:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5c916bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# providing log-in credentials to the website\n",
    "\n",
    "EMail_field = driver.find_element(By.XPATH, '//*[@id=\"email\"]') # Find e-mail field\n",
    "# specify your e-mail address\n",
    "my_email = \"ENTER YOUR E-MAIL ADDRESS\"\n",
    "\n",
    "EMail_field.send_keys(my_email) # sending the string to the e-mail field\n",
    "sleep(1.5)\n",
    "\n",
    "PW_field = driver.find_element(By.XPATH, '//*[@id=\"password\"]') # find password field\n",
    "# specify your password \n",
    "my_password = \"ENTER YOUR PASSWORD\"\n",
    "\n",
    "PW_field.send_keys(my_password) # sending the string to the password field\n",
    "sleep(1.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07a07200",
   "metadata": {},
   "source": [
    "And then we click on the login button:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7deb5330",
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_elements(By.CLASS_NAME, \"iyYUZT\")[4].click()\n",
    "sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff89facd",
   "metadata": {},
   "source": [
    "Like before, in case you encountered a recaptcha, just click on the check box by hand and then log in."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b047e9aa",
   "metadata": {},
   "source": [
    "Rejecting cookies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bda675f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rejecting cookies\n",
    "driver.find_element(By.ID, \"onetrust-reject-all-handler\").click()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaa0351f",
   "metadata": {},
   "source": [
    "Changing the language to English:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f83c23a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# click on menu\n",
    "driver.find_elements(By.CLASS_NAME, \"puppeteer_popper_reference\")[1].click()\n",
    "# click to select \"English\"\n",
    "driver.find_element(By.CSS_SELECTOR, \"div.qu-dynamicFontSize--button\").click()\n",
    "sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30d718ad",
   "metadata": {},
   "source": [
    "Now we should find the search bar, and pass our question to it for searching."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6323738",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_field = driver.find_element(By.XPATH, '//*[@enterkeyhint=\"search\"]') # finding search field\n",
    "\n",
    "question = \"Will AI kill the art industry?\"\n",
    "\n",
    "search_field.send_keys(question) # sending the question string to the search field\n",
    "sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f4350c8",
   "metadata": {},
   "source": [
    "It will look like this:\n",
    "\n",
    "<img src='images/search_question.png' width=\"900\" height=\"700\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de74e077",
   "metadata": {},
   "source": [
    "As you can see, a list of related questions show up. The first one is the one we are looking for, So we click on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb1f21a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clicking on the question link\n",
    "driver.find_elements(By.CLASS_NAME, \"iyYUZT\")[4].click()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec48d295",
   "metadata": {},
   "source": [
    "Now, a list of results are shown on the page. These results contain both the answers to the question and also some related topics. We want to have the answers only, so we need to click on the \"All related\" button, and then select the \"Answers\" option. You can see how it looks like here:\n",
    "\n",
    "<img src='images/answers.png' width=\"900\" height=\"700\" align=\"center\"/>\n",
    "\n",
    "We first click on the \"All related\" button:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "545d6f53",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Selecting answers to be shown\n",
    "driver.find_elements(By.CLASS_NAME, \"iyYUZT\")[17].click()\n",
    "sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "001cb99d",
   "metadata": {},
   "source": [
    "Now, before we click on the Answers option, let's find the number of answers to the question and save it in a variable. We will need this later for scrolling down to the last answer, so we don't miss out any of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b2217f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = driver.find_elements(By.CLASS_NAME, \"iyYUZT\")[19].text\n",
    "text = text.split()[1]\n",
    "text = text.replace(\"(\", \"\")\n",
    "text = text.replace(\")\", \"\")\n",
    "answers_count = int(text)\n",
    "\n",
    "answers_count"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "189e7bcd",
   "metadata": {},
   "source": [
    "Now we can click on the Answers option:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b85be8a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_elements(By.CLASS_NAME, \"iyYUZT\")[19].click()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08e5b96e",
   "metadata": {},
   "source": [
    "If we scroll down to the end of the page, it will show us the first 15 answers to the question. If we want to scroll down to the last answer, we need to scroll down to the end of the page, and we need to repeat this process n times, with n being the number of answers divided by 15, plus 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "988fdba7",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(answers_count // 15 + 1):\n",
    "    driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "    sleep(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "538dc73f",
   "metadata": {},
   "source": [
    "Now that we have completely scrolled down to the end of the page, we can extract the infromation from the answers. We get help from BeautifulSoup to ease the process, and we keep the author names, dates, number of upvotes and the answer texts in their corresponding lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae316b3d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "authors = []\n",
    "dates = []\n",
    "upvotes = []\n",
    "texts = []\n",
    "\n",
    "author_urls = []\n",
    "\n",
    "for i in range (answers_count):\n",
    "    \n",
    "    # Finding the answers html code:\n",
    "    answer = driver.find_element(By.CLASS_NAME, \"dom_annotate_question_answer_item_\" + str(i))\n",
    "    soup = BeautifulSoup(answer.get_attribute('innerHTML'), 'lxml')\n",
    "    \n",
    "    # Finding the desired information and saving them to their lists:\n",
    "    authors.append(soup.find_all('a', {'class': \"dFkjrQ\"})[1].text)\n",
    "    dates.append(soup.find('a', {'class': \"answer_timestamp\"}).text)\n",
    "    texts.append(soup.find_all('div', {'class': \"iyYUZT\"})[3].text)\n",
    "    \n",
    "    # We also keep the URLs of the authors in a list, we will need that later in this section.\n",
    "    author_urls.append(soup.find_all('a', {'class': \"dFkjrQ\"})[1]['href'])\n",
    "    \n",
    "    # Some of the answers do not have upvotes, we add None objects to the list for them:\n",
    "    upvote = soup.find('span', {'class': \"q-text qu-whiteSpace--nowrap qu-display--inline-flex qu-alignItems--center qu-justifyContent--center\"})    \n",
    "    if upvote == None:\n",
    "        upvotes.append(upvote)\n",
    "    else:\n",
    "        upvotes.append(upvote.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00991c96",
   "metadata": {},
   "source": [
    "Now that we have all the information, we can make a dataframe to keep the data in a more structured way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc76aee5",
   "metadata": {},
   "outputs": [],
   "source": [
    "quora_search_question_df = pd.DataFrame([authors, dates, texts, upvotes]).transpose()\n",
    "quora_search_question_df.columns = ['author', 'date', 'text', 'upvotes']\n",
    "\n",
    "quora_search_question_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a040929",
   "metadata": {},
   "source": [
    "We will save the dataframe to the outputs folder as a csv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eaebdada",
   "metadata": {},
   "outputs": [],
   "source": [
    "quora_search_question_df.to_csv('./outputs/quora search question dataframe.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "070c9ed2",
   "metadata": {},
   "source": [
    "#### Getting users' information in more details:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73835d6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Openning a new tab:\n",
    "driver.execute_script(\"window.open('https://www.quora.com/profile/Mohammed-1745')\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a996a40",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Current tab:\n",
    "driver.current_window_handle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab0bc554",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Available tabs:\n",
    "driver.window_handles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bd8d42b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Switching to a child tab:\n",
    "\n",
    "child = driver.window_handles[-1]\n",
    "\n",
    "driver.switch_to.window(child)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95cb0585",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Switching to parent tab:\n",
    "parent = driver.window_handles[0]\n",
    "\n",
    "driver.switch_to.window(parent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46c9a396",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting all users' info:\n",
    "\n",
    "no_of_followers = []\n",
    "no_of_followings = []\n",
    "no_of_answers = []\n",
    "no_of_questions = []\n",
    "no_of_posts = []\n",
    "date_joined = []\n",
    "total_content_views = []\n",
    "this_month_content_views = []\n",
    "\n",
    "parent_tab = driver.current_window_handle\n",
    "\n",
    "\n",
    "for i in author_urls:\n",
    "    \n",
    "    if 'www.quora.com' in i:\n",
    "        \n",
    "        driver.execute_script(\"window.open('\"+i+\"')\")        \n",
    "        sleep(3)\n",
    "        \n",
    "        child_tab = driver.window_handles[-1]\n",
    "        driver.switch_to.window(child_tab)\n",
    "        \n",
    "        iyYUZT = driver.find_elements(By.CLASS_NAME, \"iyYUZT\")\n",
    "        \n",
    "        if len(iyYUZT) < 22:\n",
    "            no_of_followers.append('N/A')\n",
    "            no_of_followings.append('N/A')\n",
    "            no_of_answers.append('N/A')\n",
    "            no_of_questions.append('N/A')\n",
    "            no_of_posts.append('N/A')\n",
    "            date_joined.append('N/A')\n",
    "            total_content_views.append('N/A')\n",
    "            this_month_content_views.append('N/A')\n",
    "            \n",
    "            driver.close()\n",
    "            driver.switch_to.window(parent_tab)\n",
    "            continue\n",
    "            \n",
    "        \n",
    "        no_of_followers.append(iyYUZT[11].text.split()[0])\n",
    "        no_of_followings.append(iyYUZT[12].text.split()[0])\n",
    "        no_of_answers.append(iyYUZT[18].text.split()[0])\n",
    "        no_of_questions.append(iyYUZT[19].text.split()[0])\n",
    "        no_of_posts.append(iyYUZT[20].text.split()[0])\n",
    "\n",
    "        for i in driver.find_elements(By.CLASS_NAME, \"qu-truncateLines--2\"):\n",
    "            if 'content views' in i.text:\n",
    "                content_views = i.text\n",
    "            if 'Joined' in i.text:\n",
    "                date_joined.append(i.text.split('Joined ')[1])\n",
    "\n",
    "        total_content_views.append(content_views.split()[0])\n",
    "        this_month_content_views.append(content_views.split('views')[1].split()[0])\n",
    "        \n",
    "        driver.close()\n",
    "        driver.switch_to.window(parent_tab)\n",
    "\n",
    "\n",
    "    else:        \n",
    "        \n",
    "        no_of_followers.append('N/A')\n",
    "        no_of_followings.append('N/A')\n",
    "        no_of_answers.append('N/A')\n",
    "        no_of_questions.append('N/A')\n",
    "        no_of_posts.append('N/A')\n",
    "        date_joined.append('N/A')\n",
    "        total_content_views.append('N/A')\n",
    "        this_month_content_views.append('N/A')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee9e533b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making the dataframe:\n",
    "\n",
    "users_info_df = pd.DataFrame([authors, no_of_followers, no_of_followings, no_of_answers, no_of_questions, no_of_posts,\n",
    "                             date_joined, total_content_views, this_month_content_views]).transpose()\n",
    "users_info_df.columns = ['authors', 'no_of_followers', 'no_of_followings', 'no_of_answers', 'no_of_questions', 'no_of_posts',\n",
    "                        'date_joined', 'total_content_views', 'this_month_content_views']\n",
    "\n",
    "users_info_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a04dbf5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "users_info_df.to_csv('./outputs/users_info_df.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16476b2b",
   "metadata": {},
   "source": [
    "<div class='alert alert-block alert-info'>\n",
    "<b>Insight</b>\n",
    "    \n",
    "We can also start considering whatelse we can do after learning these practices, such as searching for questions with a specific keywords or looking for multiple questions and their answers.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "446860c0",
   "metadata": {},
   "source": [
    "<div class='alert alert-block alert-warning'>\n",
    "<b>Additional resources</b>\n",
    "\n",
    "If you would like to explore web scraping further, [Scrapy](https://scrapy.org/) can be your next address for more complex web tasks. It uses less memory and CPU storage and supports data extraction from HTML sources as well. We can even extend its functionality. As we mentioned, it is great a great library for complex and larger projects and we can easily transfer existing projects into another project.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fe29c81",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "### Recommended readings\n",
    "\n",
    "Bosse, S., Dahlhaus, L., & Engel, U. (2022) \"Web data mining: Collecting textual data from web pages using R.\" In: Engel, U. & Quan-Haase, A. (eds), *Handbook of Computational Social\n",
    "Science* 2 (p. 46–70). Abingdon: Routledge. https...\n",
    "\n",
    "<a id='mclevey_doing_2022'></a>\n",
    "McLevey, J. (2022). *Doing Computational Social Science: A Practical Introduction*. SAGE. https://us.sagepub.com/en-us/nam/doing-computational-social-science/book266031. *A rather complete introduction to the field with well-structured and insightful chapters also on using Pandas. The [website](https://github.com/UWNETLAB/dcss_supplementary) offers the code used in the book.*\n",
    "\n",
    "___\n",
    "\n",
    "https://realpython.com/beautiful-soup-web-scraper-python/#reasons-for-web-scraping\n",
    "\n",
    "https://medium.com/pythoneers/the-fundamentals-of-web-scraping-using-python-its-libraries-6f146b91efb4\n",
    "\n",
    "https://www.crummy.com/software/BeautifulSoup/bs4/doc/\n",
    "\n",
    "https://www.dataquest.io/blog/web-scraping-python-using-beautiful-soup/#tve-jump-1788432a71d\n",
    "\n",
    "https://developer.mozilla.org/en-US/docs/Web/HTML/Element\n",
    "\n",
    "https://medium.com/geekculture/web-scraping-cheat-sheet-2021-python-for-web-scraping-cad1540ce21c#b81d\n",
    "\n",
    "https://trends.google.com/trends/yis/2021/DE/\n",
    "\n",
    "https://blog.google/products/search/15-tips-getting-most-out-google-trends/\n",
    "\n",
    "https://limeproxies.netlify.app/blog/selenium-vs-beautifulsoup\n",
    "\n",
    "https://github.com/strohne/autocol"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ec15403",
   "metadata": {},
   "source": [
    "<div class='alert alert-block alert-success'>\n",
    "<b>Document information</b>\n",
    "\n",
    "Contact and main author: Pouria Mirelmi\n",
    "\n",
    "Contributors: Felix Beck-Soldner, N. Gizem Bacaksizlar Turbic, & Haiko Lietz\n",
    "\n",
    "Acknowledgements: Fabian Flöck\n",
    "\n",
    "Version date: 18 August 2023\n",
    "\n",
    "License: Creative Commons Attribution 4.0 International (CC BY 4.0)\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}