2. POS and NER/1. POS Tagging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Part-of-speech Tagging"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "outputs": [],
   "source": [
    "doc = nlp(\"The quick brown fox jumped over the lazy dog's back.\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Token Tags\n",
    "* pos_: give us the coarse POS tag\n",
    "* tag_: give us a fine-grained tag"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "outputs": [
    {
     "data": {
      "text/plain": "\"The quick brown fox jumped over the lazy dog's back.\""
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc.text"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jumped: VERB - VBD - verb, past tense \n"
     ]
    }
   ],
   "source": [
    "print(f'{doc[4].text}: {doc[4].pos_} - {doc[4].tag_} - {spacy.explain(doc[4].tag_)} ')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The        DET      DT     determiner\n",
      "quick      ADJ      JJ     adjective (English), other noun-modifier (Chinese)\n",
      "brown      ADJ      JJ     adjective (English), other noun-modifier (Chinese)\n",
      "fox        NOUN     NN     noun, singular or mass\n",
      "jumped     VERB     VBD    verb, past tense\n",
      "over       ADP      IN     conjunction, subordinating or preposition\n",
      "the        DET      DT     determiner\n",
      "lazy       ADJ      JJ     adjective (English), other noun-modifier (Chinese)\n",
      "dog        NOUN     NN     noun, singular or mass\n",
      "'s         PART     POS    possessive ending\n",
      "back       NOUN     NN     noun, singular or mass\n",
      ".          PUNCT    .      punctuation mark, sentence closer\n"
     ]
    }
   ],
   "source": [
    "for token in doc:\n",
    "    print(f'{token.text:{10}} {token.pos_:{8}} {token.tag_:{6}} {spacy.explain(token.tag_)}')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Same tokens with different meaning\n",
    "Some words, verbs, etc. can have different meaning depending on the context. spaCy is smart enough to understnad the difference as we will see in the following example."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "quick      ADJ      JJ     adjective (English), other noun-modifier (Chinese)\n"
     ]
    }
   ],
   "source": [
    "doc2 = nlp('I read books on NLP.')\n",
    "r = doc[1]\n",
    "\n",
    "print(f'{r.text:{10}} {r.pos_:{8}} {r.tag_:{6}} {spacy.explain(r.tag_)}')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "quick      ADJ      JJ     adjective (English), other noun-modifier (Chinese)\n"
     ]
    }
   ],
   "source": [
    "doc2 = nlp('I read a book on NLP.')\n",
    "r = doc[1]\n",
    "\n",
    "print(f'{r.text:{10}} {r.pos_:{8}} {r.tag_:{6}} {spacy.explain(r.tag_)}')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "spaCy understood that 'read' is in the past form in the second sentence."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Counting POS Tags"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "outputs": [
    {
     "data": {
      "text/plain": "The quick brown fox jumped over the lazy dog's back."
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Counting coarse tags"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "outputs": [
    {
     "data": {
      "text/plain": "{90: 2, 84: 3, 92: 3, 100: 1, 85: 1, 94: 1, 97: 1}"
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_counts = doc.count_by(spacy.attrs.POS)\n",
    "pos_counts"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   90. DET  : 2\n",
      "   84. ADJ  : 3\n",
      "   92. NOUN : 3\n",
      "  100. VERB : 1\n",
      "   85. ADP  : 1\n",
      "   94. PART : 1\n",
      "   97. PUNCT: 1\n"
     ]
    }
   ],
   "source": [
    "for k,v in pos_counts.items():\n",
    "    print(f'{k:{5}}. {doc.vocab[k].text:{5}}: {v}')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Counting fine-grained tags"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "74. POS : 1\n",
      "1292078113972184607. IN  : 1\n",
      "10554686591937588953. JJ  : 3\n",
      "12646065887601541794. .   : 1\n",
      "15267657372422890137. DT  : 2\n",
      "15308085513773655218. NN  : 3\n",
      "17109001835818727656. VBD : 1\n"
     ]
    }
   ],
   "source": [
    "tag_counts = doc.count_by(spacy.attrs.TAG)\n",
    "\n",
    "for k,v in sorted(tag_counts.items()):\n",
    "    print(f'{k}. {doc.vocab[k].text:{4}}: {v}')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Counting dependencies"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "400. advmod: 1\n",
      "402. amod: 3\n",
      "415. det : 2\n",
      "429. nsubj: 1\n",
      "439. pobj: 1\n",
      "443. prep: 1\n",
      "445. punct: 1\n",
      "8110129090154140942. case: 1\n",
      "8206900633647566924. ROOT: 1\n"
     ]
    }
   ],
   "source": [
    "dep_counts = doc.count_by(spacy.attrs.DEP)\n",
    "\n",
    "for k,v in sorted(dep_counts.items()):\n",
    "    print(f'{k}. {doc.vocab[k].text:{4}}: {v}')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Coarse-grained Part-of-speech Tags\n",
    "Every token is assigned a POS Tag from the following list:\n",
    "\n",
    "\n",
    "<table><tr><th>POS</th><th>DESCRIPTION</th><th>EXAMPLES</th></tr>\n",
    "\n",
    "<tr><td>ADJ</td><td>adjective</td><td>*big, old, green, incomprehensible, first*</td></tr>\n",
    "<tr><td>ADP</td><td>adposition</td><td>*in, to, during*</td></tr>\n",
    "<tr><td>ADV</td><td>adverb</td><td>*very, tomorrow, down, where, there*</td></tr>\n",
    "<tr><td>AUX</td><td>auxiliary</td><td>*is, has (done), will (do), should (do)*</td></tr>\n",
    "<tr><td>CONJ</td><td>conjunction</td><td>*and, or, but*</td></tr>\n",
    "<tr><td>CCONJ</td><td>coordinating conjunction</td><td>*and, or, but*</td></tr>\n",
    "<tr><td>DET</td><td>determiner</td><td>*a, an, the*</td></tr>\n",
    "<tr><td>INTJ</td><td>interjection</td><td>*psst, ouch, bravo, hello*</td></tr>\n",
    "<tr><td>NOUN</td><td>noun</td><td>*girl, cat, tree, air, beauty*</td></tr>\n",
    "<tr><td>NUM</td><td>numeral</td><td>*1, 2017, one, seventy-seven, IV, MMXIV*</td></tr>\n",
    "<tr><td>PART</td><td>particle</td><td>*'s, not,*</td></tr>\n",
    "<tr><td>PRON</td><td>pronoun</td><td>*I, you, he, she, myself, themselves, somebody*</td></tr>\n",
    "<tr><td>PROPN</td><td>proper noun</td><td>*Mary, John, London, NATO, HBO*</td></tr>\n",
    "<tr><td>PUNCT</td><td>punctuation</td><td>*., (, ), ?*</td></tr>\n",
    "<tr><td>SCONJ</td><td>subordinating conjunction</td><td>*if, while, that*</td></tr>\n",
    "<tr><td>SYM</td><td>symbol</td><td>*$, %, §, ©, +, −, ×, ÷, =, :), 😝*</td></tr>\n",
    "<tr><td>VERB</td><td>verb</td><td>*run, runs, running, eat, ate, eating*</td></tr>\n",
    "<tr><td>X</td><td>other</td><td>*sfpksdpsxmsa*</td></tr>\n",
    "<tr><td>SPACE</td><td>space</td></tr>"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Fine-grained Part-of-speech Tags\n",
    "Tokens are subsequently given a fine-grained tag as determined by morphology:\n",
    "<table>\n",
    "<tr><th>POS</th><th>Description</th><th>Fine-grained Tag</th><th>Description</th><th>Morphology</th></tr>\n",
    "<tr><td>ADJ</td><td>adjective</td><td>AFX</td><td>affix</td><td>Hyph=yes</td></tr>\n",
    "<tr><td>ADJ</td><td></td><td>JJ</td><td>adjective</td><td>Degree=pos</td></tr>\n",
    "<tr><td>ADJ</td><td></td><td>JJR</td><td>adjective, comparative</td><td>Degree=comp</td></tr>\n",
    "<tr><td>ADJ</td><td></td><td>JJS</td><td>adjective, superlative</td><td>Degree=sup</td></tr>\n",
    "<tr><td>ADJ</td><td></td><td>PDT</td><td>predeterminer</td><td>AdjType=pdt PronType=prn</td></tr>\n",
    "<tr><td>ADJ</td><td></td><td>PRP\\$</td><td>pronoun, possessive</td><td>PronType=prs Poss=yes</td></tr>\n",
    "<tr><td>ADJ</td><td></td><td>WDT</td><td>wh-determiner</td><td>PronType=int rel</td></tr>\n",
    "<tr><td>ADJ</td><td></td><td>WP\\$</td><td>wh-pronoun, possessive</td><td>Poss=yes PronType=int rel</td></tr>\n",
    "<tr><td>ADP</td><td>adposition</td><td>IN</td><td>conjunction, subordinating or preposition</td><td></td></tr>\n",
    "<tr><td>ADV</td><td>adverb</td><td>EX</td><td>existential there</td><td>AdvType=ex</td></tr>\n",
    "<tr><td>ADV</td><td></td><td>RB</td><td>adverb</td><td>Degree=pos</td></tr>\n",
    "<tr><td>ADV</td><td></td><td>RBR</td><td>adverb, comparative</td><td>Degree=comp</td></tr>\n",
    "<tr><td>ADV</td><td></td><td>RBS</td><td>adverb, superlative</td><td>Degree=sup</td></tr>\n",
    "<tr><td>ADV</td><td></td><td>WRB</td><td>wh-adverb</td><td>PronType=int rel</td></tr>\n",
    "<tr><td>CONJ</td><td>conjunction</td><td>CC</td><td>conjunction, coordinating</td><td>ConjType=coor</td></tr>\n",
    "<tr><td>DET</td><td>determiner</td><td>DT</td><td>determiner</td><td></td></tr>\n",
    "<tr><td>INTJ</td><td>interjection</td><td>UH</td><td>interjection</td><td></td></tr>\n",
    "<tr><td>NOUN</td><td>noun</td><td>NN</td><td>noun, singular or mass</td><td>Number=sing</td></tr>\n",
    "<tr><td>NOUN</td><td></td><td>NNS</td><td>noun, plural</td><td>Number=plur</td></tr>\n",
    "<tr><td>NOUN</td><td></td><td>WP</td><td>wh-pronoun, personal</td><td>PronType=int rel</td></tr>\n",
    "<tr><td>NUM</td><td>numeral</td><td>CD</td><td>cardinal number</td><td>NumType=card</td></tr>\n",
    "<tr><td>PART</td><td>particle</td><td>POS</td><td>possessive ending</td><td>Poss=yes</td></tr>\n",
    "<tr><td>PART</td><td></td><td>RP</td><td>adverb, particle</td><td></td></tr>\n",
    "<tr><td>PART</td><td></td><td>TO</td><td>infinitival to</td><td>PartType=inf VerbForm=inf</td></tr>\n",
    "<tr><td>PRON</td><td>pronoun</td><td>PRP</td><td>pronoun, personal</td><td>PronType=prs</td></tr>\n",
    "<tr><td>PROPN</td><td>proper noun</td><td>NNP</td><td>noun, proper singular</td><td>NounType=prop Number=sign</td></tr>\n",
    "<tr><td>PROPN</td><td></td><td>NNPS</td><td>noun, proper plural</td><td>NounType=prop Number=plur</td></tr>\n",
    "<tr><td>PUNCT</td><td>punctuation</td><td>-LRB-</td><td>left round bracket</td><td>PunctType=brck PunctSide=ini</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>-RRB-</td><td>right round bracket</td><td>PunctType=brck PunctSide=fin</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>,</td><td>punctuation mark, comma</td><td>PunctType=comm</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>:</td><td>punctuation mark, colon or ellipsis</td><td></td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>.</td><td>punctuation mark, sentence closer</td><td>PunctType=peri</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>''</td><td>closing quotation mark</td><td>PunctType=quot PunctSide=fin</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>\"\"</td><td>closing quotation mark</td><td>PunctType=quot PunctSide=fin</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>``</td><td>opening quotation mark</td><td>PunctType=quot PunctSide=ini</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>HYPH</td><td>punctuation mark, hyphen</td><td>PunctType=dash</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>LS</td><td>list item marker</td><td>NumType=ord</td></tr>\n",
    "<tr><td>PUNCT</td><td></td><td>NFP</td><td>superfluous punctuation</td><td></td></tr>\n",
    "<tr><td>SYM</td><td>symbol</td><td>#</td><td>symbol, number sign</td><td>SymType=numbersign</td></tr>\n",
    "<tr><td>SYM</td><td></td><td>\\$</td><td>symbol, currency</td><td>SymType=currency</td></tr>\n",
    "<tr><td>SYM</td><td></td><td>SYM</td><td>symbol</td><td></td></tr>\n",
    "<tr><td>VERB</td><td>verb</td><td>BES</td><td>auxiliary \"be\"</td><td></td></tr>\n",
    "<tr><td>VERB</td><td></td><td>HVS</td><td>forms of \"have\"</td><td></td></tr>\n",
    "<tr><td>VERB</td><td></td><td>MD</td><td>verb, modal auxiliary</td><td>VerbType=mod</td></tr>\n",
    "<tr><td>VERB</td><td></td><td>VB</td><td>verb, base form</td><td>VerbForm=inf</td></tr>\n",
    "<tr><td>VERB</td><td></td><td>VBD</td><td>verb, past tense</td><td>VerbForm=fin Tense=past</td></tr>\n",
    "<tr><td>VERB</td><td></td><td>VBG</td><td>verb, gerund or present participle</td><td>VerbForm=part Tense=pres Aspect=prog</td></tr>\n",
    "<tr><td>VERB</td><td></td><td>VBN</td><td>verb, past participle</td><td>VerbForm=part Tense=past Aspect=perf</td></tr>\n",
    "<tr><td>VERB</td><td></td><td>VBP</td><td>verb, non-3rd person singular present</td><td>VerbForm=fin Tense=pres</td></tr>\n",
    "<tr><td>VERB</td><td></td><td>VBZ</td><td>verb, 3rd person singular present</td><td>VerbForm=fin Tense=pres Number=sing Person=3</td></tr>\n",
    "<tr><td>X</td><td>other</td><td>ADD</td><td>email</td><td></td></tr>\n",
    "<tr><td>X</td><td></td><td>FW</td><td>foreign word</td><td>Foreign=yes</td></tr>\n",
    "<tr><td>X</td><td></td><td>GW</td><td>additional word in multi-word expression</td><td></td></tr>\n",
    "<tr><td>X</td><td></td><td>XX</td><td>unknown</td><td></td></tr>\n",
    "<tr><td>SPACE</td><td>space</td><td>_SP</td><td>space</td><td></td></tr>\n",
    "<tr><td></td><td></td><td>NIL</td><td>missing tag</td><td></td></tr>\n",
    "</table>"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}