Headline Cleaning Part 2 - Stub.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Headline Cleaning Part 2: Tidying the Data\n",
    "## An incomplete notebook to get you started. \n",
    "---\n",
    "The purpose of this file is to get you started on the full headline cleaning challenge.\n",
    "\n",
    "---\n",
    "\n",
    "Now that we're familiar with the headline data, we're ready to _really_ clean this data up.\n",
    "\n",
    "In this workbook, we're going to make two output files. Here's what should go in them:\n",
    "* `clean_headlines.txt`: This should be a tab-delimited text file with a column for the paper, the date (in `YYYY-MM-DD` format), and the text of the headline. \n",
    "* `headline_word_count.txt`: This file should be tab-delimited with columns for the paper, the date (same format as above), each word the paper used on that date, and the count of those words. The words are defined as alphanumeric characters separated by whitespace from other characters, should have punctuation removed, and should be cast to lowercase. For example, \"Mr. Bean\" would be two words, \"mr\" and \"bean\". \n",
    "\n",
    "Let's work through an example so you can see what I mean. Here's a list of the headlines for the _Missoulian_ for 2015-09-23:\n",
    "* After EWU air raid, Bobcats switch focus to Cal Poly option\n",
    "* Alternative art gallery FrontierSpace to hold annual art auction\n",
    "* Does state gets passing grade for education funding? Group aims to find out\n",
    "* Fall films have issues\n",
    "* Family in need after Bonner fire destroyed home, killed cat\n",
    "* Headwaters Dance Company's \"last hurrah\" concerts are next week\n",
    "* Montana coaching tree gathers for fundraiser\n",
    "* Mountain Line to detour Saturday due to UM Homecoming Parade\n",
    "* Rabid bat found at picnic area north of Helena\n",
    "* Ravalli County attorney's daughter jailed in sibling assault\n",
    "* Search continues today for missing Kalispell bow hunter\n",
    "* Tester to escort Pope Francis to House chambers\n",
    "\n",
    "In `clean_headlines.txt` we'd expect to see a row that looks like this:\n",
    "Missoulian    2015-09-23    After EWU air raid, Bobcats switch focus to Cal Poly option\n",
    "\n",
    "In `headline_word_count.txt` we'd expect to see a row that looks like this:\n",
    "Missoulian    2015-09-23    after    2\n",
    "\n",
    "(Note that the gaps between fields should be a tab.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "from pprint import pprint\n",
    "from string import punctuation\n",
    "\n",
    "\n",
    "working_dir =  \"C:\\\\Users\\\\jchan\\\\Dropbox\\\\Teaching\\\\AppliedDataAnalytics\\\\Code\\\\headline-cleaning\\\\\"\n",
    "\n",
    "input_files = [\"missoula.txt\",\"sidney.txt\",\"butte.txt\",\"bozeman.txt\",\"billings.txt\"]\n",
    "paper_names = [\"Missoulian\",\"Sidney Herald\",\"Montana Standard\",\"Bozeman Daily Chronicle\",\"Billings Gazette\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's open up one of these files and see what we have. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23-Sep\t30-Sep\t7-Oct\t14-Oct\t21-Oct\t28-Oct\t4-Nov\t11-Nov\t18-Nov\t25-Nov\t2-Dec\t9-Dec\t16-Dec\t23-Dec\t30-Dec\t6-Jan\t13-Jan\t20-Jan\t27-Jan\t3-Feb\t10-Feb\t17-Feb\t24-Feb\t2-Mar\t9-Mar\t16-Mar\t23-Mar\t30-Mar\t6-Apr\t13-Apr\t20-Apr\t27-Apr\t4-May\t11-May\t18-May\t25-May\t1-Jun\t8-Jun\t15-Jun\t22-Jun\t29-Jun\t6-Jul\t13-Jul\t20-Jul\t27-Jul\t3-Aug\t10-Aug\t17-Aug\t24-Aug\t31-Aug\t7-Sep\t14-Sep\t21-Sep\n",
      "\n",
      "After EWU air raid, Bobcats switch focus to Cal Poly option\tStudent accidentally discharges gun at Tech student housing; no one hurt\tBillings woman suspects antifreeze poisoning killed Knuckles, her English bulldog\t'Two Rivers' takes impressionistic look at Milltown Dam's removal\tBig Sky notebook: Turnover margin a make-or-break component to success\t\t'Tall building lawyers' in court with Carlyle director\tComedy group reunites for sketch-improv show\tBourbon glazed carrots\tBison plan draws comments from around nation, world\t'Curiosities' at Butterfly Herbs print show\tAs leaders fret in Paris, climate-change concerns grow in Montana\tBig Sky XC champion Morley transferring from Griz\tChristmas Eve service to be held at Faith Baptist Church\tGriz hope to celebrate New Year in Flagstaff\tHot-shooting Griz look to continue streak at home\tGriz guards learn lesson in Kansas loss\tFormer Griz Kose to play in NFLPA Bowl this weekend\tBillings woman denies embezzling $36K from Wells Fargo Bank\tGriz defense has them atop Big Sky\tDepleted Lady Griz to battle win-starved Thunderbirds\tFormer Griz Gregory signs with Australian club\tLady Griz eye 6th straight win\tFighting Hawks soar past Lady Griz\tBe Wary of Imported Supplements: FDA\tCoach Selvig reflects on 31st 20-win season, job that keeps on giving\tHigh turnover yields wide-open receiver competitions for Griz\tGriz prepare for first spring scrimmage at 7 p.m. Thursday\tAdvocates push state to extend age limit for students with disabilities\tExpectations high for Montana's seasoned offensive line\tMontana golf finishes tied for 8th at Big Sky tournament\t3 Griz named to Hampshire Honor Society\tBillings birthday brawl at pizzeria ends with one woman hospitalized, another arrested\tGriz face Southern Utah to open first Big Sky softball tourney\tMontana volleyball finalizes fall 2016 schedule\t'Fat Shaming' Begins in First Grade\tBullock meets with Colstrip leaders about its energy future\tFormer Griz Wagenmann again injures foot in NFL camp\tBig Sky preseason poll, all-conference team: Why I voted the way I did\tBillings man sentenced to 5 years incarceration after using position of power to abuse teen girl\tGriz football opens season with night game; kickoff scheduling complicated by TV\tChicago O'Hare International Airport (ORD)\tGriz softball adds Gomez as graduate assistant\tBackpacks that stand out\tCards place Wagenmann on PUP list\tFormer Griz WR Jamaal Jones injured, waived by Chargers\tDT Davidson brings size, strength to Griz line\tKalispell product Petrino joins Lady Griz coaching staff\tCoaches rank Griz soccer 10th in Pacific region\tGreen Griz defense looking for quick start vs. Red Flash offense\tAfter EWU air raid, Bobcats switch focus to Cal Poly option\tStudent accidentally discharges gun at Tech student housing; no one hurt\tBillings woman suspects antifreeze poisoning killed Knuckles, her English bulldog\n",
      "\n",
      "Alternative art gallery FrontierSpace to hold annual art auction\tYouth detention facility at Galen will close, not relocate\tCorvallis woman arrested for fleeing Missoula police on bike, possessing meth\tBillings man accused of killing neighbor: 'It's Halloween. We're just having some fun' after shooting\tCar chase along Flathead Lake ends in arrest\tArtist Isabelle Johnson bucked tradition, introduced modernism to students\t2 Billings City Council incumbents lose to challengers; Cromley keeps seat\tDeputy shoots driver during traffic stop near Dillon\tBullock noncommittal on opening Superfund talks\tCold nights, cautious travel in forecast for northwestern Montana\t'Legroom'\t\tGlacier's Bennett understands what Gregorak is going through\t\tLady Griz look to usher in 2016 in style\tYoung Vikings to test Lady Griz\tSurging Bears to test Lady Griz\tGriz add two JuCo transfers to defensive haul\tCeramic artist Doug Baldwin to talk about his life, work\tLady Griz hungry for victory\tGriz adopt playoff mindset with tournament looming\tGrizzlies' Champions Center construction set for April\tMontana's tough road trip will test Oguine\tAugusta-area man says he feared for life when he shot, killed neighbor\tBig Sky women: Idaho State stuns Montana State on last-second shot\tNevada erases halftime deficit, ends Grizzlies' season\tThe Most Outdoorsy Jobs\tKicking, punting competitions open and even this spring for Griz\tAmerica's 50 favorite dog breeds\tMontana State to pay O'Neill's canceled $28,000 speaking fee\tMontana women's athletic programs lead way in APR\tCollege softball: Stensby pitches another gem as Griz sweep Argonauts\tBillings sergeant on paid leave since February resigns\tGriz track and field: McLeod posts 3 personal bests in 1st day of heptathlon\t21-year-old man stabbed to death Wednesday in Lame Deer\t\tBurn rate\tBus passenger arrested in Billings with marijuana in luggage\tFormer Griz guard Gregory signs to play in Holland\tBritney Spears' gum\tStewart promoted to assistant, completing DeCuire's staff\tConfirmed microcephaly cases in Brazil\tBillings Animal Rescue Kare supports protest over sentence in dog death case\tBackpacks with your kids' favorite characters\tFor Selvig, success came at a price\tGriz see strength in revamped D-line\tGriz mine Thompson Falls' 'diamond in the rough'\tStitt: Chalich, Phillips remain too close to call in battle for backup QB\tInjuries have Griz still sorting through options at receiver\tGriz goalkeeper Kailey Norman named Big Sky Defensive Player of the Week\tAlternative art gallery FrontierSpace to hold annual art auction\tYouth detention facility at Galen will close, not relocate\tCorvallis woman arrested for fleeing Missoula police on bike, possessing meth\n",
      "\n",
      "Does state gets passing grade for education funding? Group aims to find out\tGiant Geyser, 2nd largest in Yellowstone, erupts in rare display\tDistemper in dog temporarily closes Helena animal shelter; stray cats accepted by appointment\tBillings vet receives keys to new family home\tChess tournament planned in Kalispell\tBillings Clinic increases security after shooting, does not blame officers\tAround the Big Sky\tDramatic dashcam footage: 3 close calls, woman ejected from vehicle\tClass AA All-State volleyball: Big Sky's Gragg, Sentinel's Godwin make first team\tEngesser scores 25; Denver beats Idaho State 79-69\t'Lolo' subject of Travelers' Rest Chapter presentation\tBillings man accused of attempting to steal alcohol from bar\tGriz sign 2 transfers for spring football\tLow commodity prices create anxiety in southwest Montana, worldwide\tBernie Sanders takes on Trump in Iowa campaign stop\tCity Club Missoula discusses development office\tBarry Beach throws party for Billings supporters\tLady Griz begin big road swing\tDivorcing husband wary of wife's online deception\tMontana signing Class of 2016 bios\tBig Sky commissioner Doug Fullerton to retire in June\tBig Sky Film Fest: Lucy Walker discusses new VR documentary\tWestern A boys' basketball: Columbia Falls outlasts Corvallis in OT\tCareers with the Highest Projected Employment Growth\tBrain Bleed Risk From Warfarin May Be Higher Than Thought\t'Stuttering comedian' Nina G to give performance, 'Going Beyond Inspirational'\tBelgian MSUB basketball player to friends, family: ?Don?t jeopardize your freedom just because you?re scared?\tMontana men's basketball: Bradshaw will transfer for senior season\tAmerica: A carb-loving nation\tNew UM football helmet to feature clawing-out Grizzly\tShifty Z receivers add new element to Griz offense\tHolmes leads Griz group awaiting draft fate\tDebt Free Degrees proposal\t\tADHD Can First Appear in Young Adulthood for Some, Study Suggests\t2 New Findings Offer Hope for Those With Severe Hemophilia\tButte counselor, campaign treasurer faces felony drug charges\tCancer's Heavy Financial Burden\tWa-Griz Champions Center: 'It's on its way'\tBritney Spears' pregnancy test\tB Mavs to host Firecracker tournament\tDallas/Fort Worth International Airport (DFW)\tBillings' St. Vincent Healthcare to lay off 49, close employee day care\tBig Timber wind farm in jeopardy\tLady Griz coach Robin Selvig retires after 38 years\tBoyfriend fights woman's effort to end her 10-year obsession\tShannon Schweyen officially named next Lady Griz coach\tAttorney General Tim Fox wants to hear from victims\tCommunity meeting set for tonight on Copper King fire\tGriz release first fall football depth chart\tDoes state gets passing grade for education funding? Group aims to find out\tGiant Geyser, 2nd largest in Yellowstone, erupts in rare display\tDistemper in dog temporarily closes Helena animal shelter; stray cats accepted by appointment\n",
      "\n"
     ]
    }
   ],
   "source": [
    "this_input = input_files[0]\n",
    "\n",
    "with open(working_dir + this_input) as infile :\n",
    "    for idx,row in enumerate(infile.readlines()) :\n",
    "        print(row)\n",
    "        if idx > 2 :\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Still a mess. I'm going to write a function that takes the paper in and builds a well-formatted date from these ugly dates. \n",
    "\n",
    "First, I'm just going to read in all the dates and keep them in order so that I can see what's going on. This step could be done in Excel too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dates = defaultdict(list) # paper keying a list of dates\n",
    "\n",
    "for this_input in input_files : # notice, by not hardcoding the input file I can quickly wrap the above cell in a `for`\n",
    "    with open(working_dir + this_input,'r',encoding=\"Latin-1\") as infile :\n",
    "        \n",
    "        these_dates = infile.readline().strip().split(\"\\t\")\n",
    "        paper = paper_names[input_files.index(this_input)] # ?index if this doesn't make sense. \n",
    "        \n",
    "        dates[paper] = these_dates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missoulian : 23-Sep 30-Sep 7-Oct 14-Oct 21-Oct 28-Oct 4-Nov 11-Nov 18-Nov 25-Nov 2-Dec 9-Dec 16-Dec 23-Dec 30-Dec 6-Jan 13-Jan 20-Jan 27-Jan 3-Feb 10-Feb 17-Feb 24-Feb 2-Mar 9-Mar 16-Mar 23-Mar 30-Mar 6-Apr 13-Apr 20-Apr 27-Apr 4-May 11-May 18-May 25-May 1-Jun 8-Jun 15-Jun 22-Jun 29-Jun 6-Jul 13-Jul 20-Jul 27-Jul 3-Aug 10-Aug 17-Aug 24-Aug 31-Aug 7-Sep 14-Sep 21-Sep\n",
      "\n",
      "\n",
      "\n",
      "Sidney Herald : 22-Sep 29-Sep 6-Oct 15-Oct 20-Oct 27-Oct 3-Nov 10-Nov 17-Nov 25-Nov 1-Dec 8-Dec 15-Dec 22-Dec 29-Dec 5-Jan 12-Jan 20-Jan 26-Jan 2-Feb 10-Feb 17-Feb 24-Feb 2-Mar 9-Mar 16-Mar 23-Mar 30-Mar 8-Apr 15-Apr 20-Apr 27-Apr 4-May 11-May 18-May 25-May 1-Jun 8-Jun 15-Jun 22-Jun 29-Jun 6-Jul 13-Jul 19-Jul 27-Jul 3-Aug 10-Aug 17-Aug 24-Aug 30-Aug 6-Sep 13-Sep 21-Sep\n",
      "\n",
      "\n",
      "\n",
      "Montana Standard : 22-Sep 30-Sep 7-Oct 14-Oct 21-Oct 28-Oct 4-Nov 11-Nov 18-Nov 25-Nov 2-Dec 9-Dec 16-Dec 23-Dec 30-Dec 6-Jan 13-Jan 20-Jan 27-Jan 3-Feb 10-Feb 17-Feb 24-Feb 3-Mar 9-Mar 16-Mar 22-Mar 30-Mar 6-Apr 13-Apr 20-Apr 27-Apr 4-May 11-May 18-May 25-May 1-Jun 8-Jun 15-Jun 22-Jun 29-Jun 6-Jul 12-Jul 20-Jul 27-Jul 3-Aug 10-Aug 17-Aug 25-Aug 31-Aug 7-Sep 14-Sep 21-Sep\n",
      "\n",
      "\n",
      "\n",
      "Bozeman Daily Chronicle : 23-Sep 30-Sep 7-Oct 14-Oct 21-Oct 28-Oct 4-Nov 11-Nov 18-Nov 25-Nov 2-Dec 9-Dec 16-Dec 23-Dec 30-Dec 6-Jan 13-Jan 20-Jan 27-Jan 3-Feb 10-Feb 17-Feb 24-Feb 2-Mar 9-Mar 16-Mar 23-Mar 30-Mar 6-Apr 13-Apr 20-Apr 27-Apr 4-May 11-May 18-May 25-May 1-Jun 8-Jun 15-Jun 22-Jun 29-Jun 6-Jul 13-Jul 20-Jul 27-Jul 3-Aug 10-Aug 17-Aug 24-Aug 31-Aug 7-Sep 14-Sep 21-Sep 28-Sep 5-Oct\n",
      "\n",
      "\n",
      "\n",
      "Billings Gazette : 22-Sep 30-Sep 7-Oct 14-Oct 21-Oct 28-Oct 4-Nov 11-Nov 18-Nov 25-Nov 2-Dec 9-Dec 16-Dec 23-Dec 30-Dec 6-Jan 13-Jan 20-Jan 27-Jan 3-Feb 10-Feb 17-Feb 24-Feb 2-Mar 9-Mar 16-Mar 23-Mar 30-Mar 6-Apr 13-Apr 20-Apr 27-Apr 4-May 11-May 18-May 25-May 1-Jun 8-Jun 15-Jun 22-Jun 29-Jun 6-Jul 13-Jul 20-Jul 27-Jul 3-Aug 10-Aug 17-Aug 24-Aug 31-Aug 7-Sep 14-Sep 21-Sep\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now I'm going to print the paper and the dates in \"clean\" fashion \n",
    "# to try to figure out where the 2015/2016 break is for each paper.\n",
    "for paper in dates :\n",
    "    print(paper + \" : \" + \" \".join(dates[paper]))\n",
    "    print(\"\\n\\n\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now we're in a position to write a function that takes an ugly date and makes a pretty one. We're going to need the paper as an input into the function. I've put a stub here as well as a set of `assert` statements you can use to test your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def reformat_date(ugly_date, the_paper) :\n",
    "    '''\n",
    "        Takes as input a date of the form \"D-MMM\" and returns a date of the form\n",
    "        \"YYYY-MM-DD\". Note that we have to do some work on years. Dates in Oct, Nov,\n",
    "        Dec are *almost* all 2015. \n",
    "        \n",
    "        For every paper other than Bozeman Daily Chronicle, dates in a year\n",
    "        later than 21-Sep are in 2015. For BZN, 28-Sep and 5-Oct are in 2016.\n",
    "    '''\n",
    "    # Here's where you'll put your code to handle the formatting of \n",
    "    # ugly dates into pretty dates. \n",
    "    pass\n",
    "        \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Lots of testing here to make sure I've got what I expect.\n",
    "assert(\"2015-09-23\" == reformat_date(\"23-Sep\",\"Missoulian\"))\n",
    "assert(\"2015-09-30\" == reformat_date(\"30-Sep\",\"Missoulian\"))\n",
    "assert(\"2015-12-23\" == reformat_date(\"23-Dec\",\"Missoulian\"))\n",
    "assert(\"2016-01-06\" == reformat_date(\"6-Jan\",\"Missoulian\"))\n",
    "assert(\"2016-03-09\" == reformat_date(\"9-Mar\",\"Missoulian\"))\n",
    "assert(\"2016-06-01\" == reformat_date(\"1-Jun\",\"Missoulian\"))\n",
    "assert(\"2016-09-21\" == reformat_date(\"21-Sep\",\"Missoulian\"))\n",
    "\n",
    "assert(\"2015-09-22\" == reformat_date(\"22-Sep\",\"Sidney Herald\"))\n",
    "assert(\"2015-09-30\" == reformat_date(\"30-Sep\",\"Sidney Herald\"))\n",
    "assert(\"2016-06-01\" == reformat_date(\"1-Jun\",\"Sidney Herald\"))\n",
    "assert(\"2016-09-21\" == reformat_date(\"21-Sep\",\"Sidney Herald\"))\n",
    "\n",
    "assert(\"2015-09-22\" == reformat_date(\"22-Sep\",\"Montana Standard\"))\n",
    "assert(\"2015-09-30\" == reformat_date(\"30-Sep\",\"Montana Standard\"))\n",
    "assert(\"2015-10-28\" == reformat_date(\"28-Oct\",\"Montana Standard\"))\n",
    "assert(\"2016-06-01\" == reformat_date(\"1-Jun\",\"Montana Standard\"))\n",
    "assert(\"2016-09-21\" == reformat_date(\"21-Sep\",\"Montana Standard\"))\n",
    "\n",
    "assert(\"2015-09-22\" == reformat_date(\"22-Sep\",\"Billings Gazette\"))\n",
    "assert(\"2015-09-30\" == reformat_date(\"30-Sep\",\"Billings Gazette\"))\n",
    "assert(\"2015-10-28\" == reformat_date(\"28-Oct\",\"Billings Gazette\"))\n",
    "assert(\"2016-06-01\" == reformat_date(\"1-Jun\",\"Billings Gazette\"))\n",
    "assert(\"2016-09-21\" == reformat_date(\"21-Sep\",\"Billings Gazette\"))\n",
    "\n",
    "# Bozeman: Standard ones plus special cases\n",
    "assert(\"2015-09-23\" == reformat_date(\"23-Sep\",\"Bozeman Daily Chronicle\"))\n",
    "assert(\"2015-09-30\" == reformat_date(\"30-Sep\",\"Bozeman Daily Chronicle\"))\n",
    "assert(\"2015-10-28\" == reformat_date(\"28-Oct\",\"Bozeman Daily Chronicle\"))\n",
    "assert(\"2016-06-01\" == reformat_date(\"1-Jun\",\"Bozeman Daily Chronicle\"))\n",
    "assert(\"2016-09-21\" == reformat_date(\"21-Sep\",\"Bozeman Daily Chronicle\"))\n",
    "\n",
    "assert(\"2016-09-21\" == reformat_date(\"21-Sep\",\"Bozeman Daily Chronicle\"))\n",
    "assert(\"2016-09-28\" == reformat_date(\"28-Sep\",\"Bozeman Daily Chronicle\"))\n",
    "assert(\"2016-10-05\" == reformat_date(\"5-Oct\",\"Bozeman Daily Chronicle\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Okay, now it's time for you to shine. Write some code that loops through the files and creates our first output file. This file should have a column for newspaper, date, and the headline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Code for creating the first output file can start here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Here's a place to put code that writes out the data to `clean_headlines.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Having done that, we want to create the second output file, which does word counts by paper and date. Our lives may be easier if we start by writing a function that takes a headline as input and returns a list of the words in that headline with punctuation removed and everything in lowercase. For reasons that are unclear to me, I'm just giving you my function that does this. Let me know if you have any questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clean_headline(hl) :\n",
    "    ''' Removes punctuation, changes to lowercase, and splits on whitespace.\n",
    "        Returns a list of the words in a headline'''\n",
    "    \n",
    "    # Let's define our excluded characters. \n",
    "    excluded_chars = set(punctuation)\n",
    "    excluded_chars.add((\"`\",\"‘\",\"’\")) # some chars missed by `punctuation`\n",
    "\n",
    "    # now, take the headline, cast it to lower, remove the punctuation characters\n",
    "    # and return a list of words. \n",
    "    hl = hl.lower()\n",
    "    hl = ''.join([ch for ch in hl if ch not in excluded_chars])\n",
    "    return(hl.split())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# And, as always, let's test.\n",
    "assert(clean_headline(\"A headline test.\") == ['a','headline','test'])\n",
    "assert(clean_headline(\"Moar testin'.\") == ['moar','testin'])\n",
    "assert(clean_headline(\"Bobcats' woes.\") == ['bobcats','woes'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now take the headlines, split them into words, and record the counts by paper, date, and word. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# your code for building the word count data can go here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Here's a place to put code that writes out the data to `headline_word_count.txt`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}