Text Summarizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bs4 as bs\n",
    "import urllib.request\n",
    "import re\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scrap Data From Wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "scraped_data = urllib.request.urlopen('https://en.wikipedia.org/wiki/Data_science')\n",
    "article = scraped_data.read()\n",
    "parsed_article =  bs.BeautifulSoup(article, 'lxml')\n",
    "paragraphs = parsed_article.find_all('p')\n",
    "article_text = \"\"\n",
    "for p in paragraphs:\n",
    "    article_text += p.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "article_text = re.sub(r'\\[[0-9]*\\]', ' ', article_text)\n",
    "article_text = re.sub(r'\\s+', ' ', article_text)\n",
    "formatted_article = re.sub('[^a-zA-Z]', ' ', article_text)\n",
    "formatted_article = re.sub(r'\\s+', ' ', formatted_article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenize_sentence = nltk.sent_tokenize(article_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequency of Each Word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "stopwords = nltk.corpus.stopwords.words('english')\n",
    "word_frequency = {}\n",
    "for word in nltk.word_tokenize(formatted_article):\n",
    "    if word not in stopwords:\n",
    "        if word not in word_frequency.keys():\n",
    "            word_frequency[word] = 1\n",
    "        else: \n",
    "            word_frequency[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "maximum_frequncy = max(word_frequency.values())\n",
    "for word in word_frequency.keys():\n",
    "    word_frequency[word] = (word_frequency[word] / maximum_frequncy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentence Score\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_score = {}\n",
    "for sent in tokenize_sentence:\n",
    "    for word in nltk.word_tokenize(sent.lower()):\n",
    "        if word in word_frequency.keys():\n",
    "            if len(sent.split(' ')) < 50:\n",
    "                if sent not in sentence_score.keys():\n",
    "                    sentence_score[sent] = word_frequency[word]\n",
    "                else:\n",
    "                    sentence_score[sent] += word_frequency[word]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of Whole Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"Data science\" became more widely used in the next few years: in 2002, the Committee on Data for Science and Technology launched Data Science Journal. In 2014, the American Statistical Association's Section on Statistical Learning and Data Mining changed its name to the Section on Statistical Learning and Data Science, reflecting the ascendant popularity of data science. Data science is a \"concept to unify statistics, data analysis, machine learning, domain knowledge and their related methods\" in order to \"understand and analyze actual phenomena\" with data. Stanford professor David Donoho writes that data science is not distinguished from statistics by the size of datasets or use of computing, and that many graduate programs misleadingly advertise their analytics and statistics training as the essence of a data science program. Data science is an interdisciplinary field focused on extracting knowledge from data sets, which are typically large (see big data). Turing award winner Jim Gray imagined data science as a \"fourth paradigm\" of science (empirical, theoretical, computational and now data-driven) and asserted that \"everything about science is changing because of the impact of information technology\" and the data deluge. Data science is related to data mining, machine learning and big data.  Data science is an inter-disciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from many structural and unstructured data. Later, attendees at a 1992 statistics symposium at the University of Montpellier II acknowledged the emergence of a new discipline focused on data of various origins and forms, combining established concepts and principles of statistics and data analysis with computing. Andrew Gelman of Columbia University and data scientist Vincent Granville have described statistics as a nonessential part of data science.\n"
     ]
    }
   ],
   "source": [
    "import heapq \n",
    "sentence_summary = heapq.nlargest(10, sentence_score, key = sentence_score.get)\n",
    "summary = ' '.join(sentence_summary)\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}