BDTA Lesson 10 Basic CSV Extracting Contexts to File.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic CSV: Extracting Column to File\n",
    "This Notebook is for extracting a column of text comments from a CSV. It assumes you have **Zapped Gremlins**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we see what files we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AdamSavageComments.csv   MockInterviewCorpus.csv\r\n"
     ]
    }
   ],
   "source": [
    "%ls *.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the appropriate column\n",
    "\n",
    "Now we import colum 4 (the 5th column) of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "585"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import csv\n",
    "comments = []\n",
    "with open('AdamSavageComments.csv', 'r') as file: # This makes sure that file is closed after reading\n",
    "    data = csv.reader(file)\n",
    "    for row in data:\n",
    "        comments.append(row[4]) # This puts all the data from column 5 into a list\n",
    "    file.close()\n",
    "\n",
    "len(comments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['commentText',\n",
       " 'Adam avoids question.',\n",
       " \"Adam Savage's stock went way down in my book. Sometimes it's best not to get to know celebrities or what they think... 3:36 Ahh, he's bordering and battling with white guilt. We just saw it's mental evolution. Compare the contradiction 1:45 and 4:04.\",\n",
       " 'not sure what side Adam is talking about... sounds more like the SJW and not GG...']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comments[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing with conditions\n",
    "\n",
    "We can also make decisions based on other columns. In this case we check to see if column 5 is a decimal (because in the the first row it is not) **and** whether it is over 3. This pulls the comments liked by more than three people."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "174"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import csv\n",
    "comments = []\n",
    "with open('AdamSavageComments.csv', 'r') as file: # This makes sure that file is closed after reading\n",
    "    data = csv.reader(file)\n",
    "    for row in data:\n",
    "        if row[5].isdecimal() and int(row[5]) > 3:\n",
    "            comments.append(row[4]) # This puts all the data into a list\n",
    "    file.close()       \n",
    "\n",
    "len(comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check what comments we got."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i waited until the end to down vote this video.', '0:35 - \"I don\\'t understand the anger...\" You don\\'t even understand the premise.']\n"
     ]
    }
   ],
   "source": [
    "print(comments[0:2]) # Here we check the list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting Comments\n",
    "\n",
    "We can further process the comments selecting those we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selectedComms = []\n",
    "\n",
    "for comment in comments:\n",
    "    if \"book\" in comment or \"stock\" in comment: # Here we check if the words we want are in the comment.\n",
    "        selectedComms.append(comment)\n",
    "        \n",
    "len(selectedComms)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert list to string\n",
    "\n",
    "If we want to use our text tools we need to convert the items in the list into a nice text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "0:35 - \"I don't understand the anger...\" You don't even understand the premise.\n",
      "\n",
      "He doesnt know th\n"
     ]
    }
   ],
   "source": [
    "theWholeText = \"\"\n",
    "\n",
    "for comment in comments[1:]:\n",
    "    theWholeText = theWholeText + \"\\n\\n\" + comment\n",
    "    \n",
    "print(theWholeText[0:100]) # We check by printing first 100 characters of the string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Here we append all the comments to get a single text. We could save that out or search it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving file\n",
    "Now we save the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(\"FullText.txt\", \"w\") as myfile: # Note that we overwrite to the file. That is the \"w\"\n",
    "    myfile.write(theWholeText)\n",
    "    myfile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FullText.txt                performanceConcordance.txt\r\n",
      "Hume Enquiry.txt            theWritingStory.txt\r\n",
      "StoryOfWriting.txt          truthConcordance.txt\r\n",
      "bigdata.txt\r\n"
     ]
    }
   ],
   "source": [
    "%ls *.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\n\\nRT @rickkytheG: https://t.co/aq0wC7bOuE     Responding to feminists who oppose prostitution #GamerGate #opskynet #notyourshield\\n\\nhttps://t.co/aq0wC7bOuE     Responding to feminists who oppose prostitution #GamerGate #opskynet #notyourshield\\n\\nRT @cringe_channel: So, what isn't #gamergate's fault to\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# theWholeText[0:300]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Exercise\n",
    "\n",
    "Download the MockInterviewData.csv file. That has interview data from 4 interviews with 2 questions each interview. Write a notebook that can:\n",
    "\n",
    "* Extract the answers by people with graduate degrees (MA or PhD)\n",
    "\n",
    "**Optional**\n",
    "Can you also get the high frequency words from those with just BAs and compare them to those by those with advanced degrees?\n",
    "\n",
    "* Calculate the top high-frequency words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}