1. Basics/5. Pattern Matching.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Pattern Matching\n",
    "Pattern matching allow us to match a string using specific rules."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "from spacy.matcher import Matcher\n",
    "matcher = Matcher(nlp.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Creating patterns"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Function to print matches"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "outputs": [],
   "source": [
    "def print_matches(matches, doc_):\n",
    "    print(f'Found {len(matches)} matches:')\n",
    "    for match_id, start, end in matches:\n",
    "        string_id = nlp.vocab.strings[match_id]     # string representation of the id\n",
    "        span = doc_[start:end]                      # matched text chunk +10 surrounding words\n",
    "        print(match_id, string_id, start, end, span.text)\n",
    "\n",
    "    print(f'Found {len(found_matches)} matches:')\n",
    "\n",
    "    for match_id, start, end in found_matches:\n",
    "        string_id = nlp.vocab.strings[match_id]  # string representation of the id\n",
    "        chunk = doc[start-5:end+5]               # matched text chunk +10 surrounding words\n",
    "        print(match_id, string_id, start, end, chunk.text)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In this example, we create three patterns. The third one:\n",
    "`pattern3 = [{'LOWER': 'solar'}, {'IS_PUNCT': True}, {'LOWER': 'power'}]`\n",
    "means: match the lower version of the token to the word 'solar' followed by a punctuation then by 'power'. E.g. 'solar-power', 'solar/power', or even a space: 'solar power', etc.\n",
    "The first two match: 'solarpower', and 'solar power'"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "outputs": [],
   "source": [
    "pattern1 = [{'LOWER': 'solarpower'}]\n",
    "# pattern2 = [{'LOWER': 'solar'}, {'LOWER': 'power'}]\n",
    "pattern3 = [{'LOWER': 'solar'}, {'IS_PUNCT': True}, {'LOWER': 'power'}]\n",
    "\n",
    "matcher.add('SolarPower', patterns=[pattern1, pattern3])"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2 matches:\n",
      "8656102463236116519 SolarPower 10 11 solarpower\n",
      "8656102463236116519 SolarPower 13 16 Solar-power\n"
     ]
    }
   ],
   "source": [
    "doc = nlp('The Solar Power industry continues to grow as demand for solarpower increases. Solar-power cars are gaining popularity.')\n",
    "found_matches = matcher(doc)\n",
    "print_matches(found_matches, doc)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Patterns Options\n",
    "We can use the 'OP' key to quantify patterns.\n",
    "\n",
    "The following quantifiers can be passed to the `'OP'` key:\n",
    "<table><tr><th>OP</th><th>Description</th></tr>\n",
    "\n",
    "<tr ><td><span >\\!</span></td><td>Negate the pattern, by requiring it to match exactly 0 times</td></tr>\n",
    "<tr ><td><span >?</span></td><td>Make the pattern optional, by allowing it to match 0 or 1 times</td></tr>\n",
    "<tr ><td><span >\\+</span></td><td>Require the pattern to match 1 or more times</td></tr>\n",
    "<tr ><td><span >\\*</span></td><td>Allow the pattern to match zero or more times</td></tr>\n",
    "</table>"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "outputs": [],
   "source": [
    "matcher.remove('SolarPower')\n",
    "\n",
    "pattern1 = [{'LOWER': 'solarpower'}]\n",
    "pattern2 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'power'}]\n",
    "matcher.add('SolarPower', patterns=[pattern1, pattern2])"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 3 matches:\n",
      "8656102463236116519 SolarPower 1 3 Solar Power\n",
      "8656102463236116519 SolarPower 10 11 solarpower\n",
      "8656102463236116519 SolarPower 13 16 Solar-power\n"
     ]
    }
   ],
   "source": [
    "found_matches = matcher(doc)\n",
    "print_matches(found_matches, doc)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Matching lemmas!\n",
    "If we wanted to match on both 'solar power' and 'solar powered', it might be tempting to look for the *lemma* of 'powered' and expect it to be 'power'. This is not always the case! The lemma of the *adjective* 'powered' is still 'powered':"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "outputs": [],
   "source": [
    "pattern1 = [{'LOWER': 'solarpower'}]\n",
    "pattern2 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LEMMA': 'power'}] # CHANGE THIS PATTERN\n",
    "\n",
    "# Remove the old patterns to avoid duplication:\n",
    "matcher.remove('SolarPower')\n",
    "\n",
    "# Add the new set of patterns to the 'SolarPower' matcher:\n",
    "matcher.add('SolarPower', patterns=[pattern1, pattern2])"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "outputs": [],
   "source": [
    "doc2 = nlp('Solar-powered energy runs solar-powered cars.')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2 matches:\n",
      "8656102463236116519 SolarPower 0 3 Solar-powered\n",
      "8656102463236116519 SolarPower 5 8 solar-powered\n"
     ]
    }
   ],
   "source": [
    "found_matches = matcher(doc2)\n",
    "print_matches(found_matches, doc2)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Phrase Matching"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Load PhraseMatcher and the text file"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "outputs": [],
   "source": [
    "# Import the PhraseMatcher library\n",
    "from spacy.matcher import PhraseMatcher\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "\n",
    "with open('text_files/reagonomics.txt', encoding='utf8') as f:\n",
    "    doc3 = nlp(f.read())"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "outputs": [
    {
     "data": {
      "text/plain": "[voodoo economics,\n supply-side economics,\n trickle-down economics,\n free-market economics]"
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First, create a list of match phrases:\n",
    "phrase_list = ['voodoo economics', 'supply-side economics', 'trickle-down economics', 'free-market economics']\n",
    "\n",
    "# Next, convert each phrase to a Doc object:\n",
    "phrase_patterns = [nlp(text) for text in phrase_list]\n",
    "\n",
    "phrase_patterns"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "outputs": [],
   "source": [
    "matcher.add('VoodooEconomics', phrase_patterns)\n",
    "\n",
    "# Build a list of matches:\n",
    "found_matches3 = matcher(doc3)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "outputs": [
    {
     "data": {
      "text/plain": "[(3473369816841043438, 41, 45),\n (3473369816841043438, 49, 53),\n (3473369816841043438, 54, 56),\n (3473369816841043438, 61, 65),\n (3473369816841043438, 673, 677),\n (3473369816841043438, 2990, 2994)]"
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "found_matches3"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The first four matches are where these terms are used in the definition of Reaganomics:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 6 matches:\n",
      "3473369816841043438 VoodooEconomics 41 45 supply-side economics\n",
      "3473369816841043438 VoodooEconomics 49 53 trickle-down economics\n",
      "3473369816841043438 VoodooEconomics 54 56 voodoo economics\n",
      "3473369816841043438 VoodooEconomics 61 65 free-market economics\n",
      "3473369816841043438 VoodooEconomics 673 677 supply-side economics\n",
      "3473369816841043438 VoodooEconomics 2990 2994 trickle-down economics\n"
     ]
    }
   ],
   "source": [
    "print_matches(found_matches3, doc3)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "outputs": [
    {
     "data": {
      "text/plain": "REAGANOMICS\nhttps://en.wikipedia.org/wiki/Reaganomics\n\nReaganomics (a portmanteau of [Ronald] Reagan and economics attributed to Paul Harvey)[1] refers to the economic policies promoted by U.S. President Ronald Reagan during the 1980s. These policies are commonly associated with supply-side economics, referred to as trickle-down economics or voodoo economics by political opponents, and free-market economics by political advocates.\n"
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc3[:70]"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## View Matching"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "outputs": [
    {
     "data": {
      "text/plain": "policies are commonly associated with supply-side economics, referred to as trickle"
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The first match in `matches`. we add surround it by 5 words\n",
    "doc3[41-5:45+5]"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Using a Sentencizer"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These policies are commonly associated with supply-side economics, referred to as trickle-down economics or voodoo economics by political opponents, and free-market economics by political advocates.\n",
      "\n",
      " 35 70\n"
     ]
    }
   ],
   "source": [
    "# Build a list of sentences\n",
    "sents = [sent for sent in doc3.sents]\n",
    "\n",
    "print(sents[1], sents[1].start, sents[1].end)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These policies are commonly associated with supply-side economics, referred to as trickle-down economics or voodoo economics by political opponents, and free-market economics by political advocates.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "first_match = found_matches3[0]\n",
    "first_match_start = first_match[1]\n",
    "first_match_end = first_match[2]\n",
    "\n",
    "for sent in sents:\n",
    "    if first_match_start > sent.start and first_match_end < sent.end:\n",
    "        print(sent)"
   ],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}