word2vecTalk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro and setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates the use of word2vec in Python using the  [gensim libraries](https://github.com/RaRe-Technologies/gensim).  Information is available on the [gensim website](https://radimrehurek.com/gensim/index.html) along with tutorials and the [API](https://radimrehurek.com/gensim/apiref.html). \n",
    "\n",
    "You can install them to your local machine using the command:\n",
    "```\n",
    "pip install --upgrade gensim\n",
    "```\n",
    "Of course if you are using a jupyter notebook, that doesn't mean it is installed in the correct place.  Make sure the `sys.path` command yields a location that matches the location of where the `pip install` occured.  Because the `sys.path` includes the current working directory, I use a `-t` flag to place the libraries in a location I can make the current working directory.  You could also theroretically append to `sys.path`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word2Vec was first presented in the paper: Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). *Efficient estimation of word representations in vector space*. arXiv preprint arXiv:1301.3781.  Extensions were presented in the paper: Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). *Distributed representations of words and phrases and their compositionality*. In Advances in neural information processing systems (pp. 3111-3119).\n",
    "\n",
    "The original C code was published on Google Code, but was subsequently moved to https://github.com/tmikolov/word2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pretty printing has been turned OFF\n"
     ]
    }
   ],
   "source": [
    "# turn off pretty printing to get horizontal display - optional, but I'm saving space for display\n",
    "%pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Commands like `os.getcwd()` and `os.chdir()` can help you navigate to the proper location.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['word2vec.ipynb', '__pycache__', 'python_dateutil-2.7.3.dist-info', 'numpy', 'requests-2.19.1.dist-info', 'dateutil', 'idna', 'scipy-1.1.0.dist-info', 'jmespath-0.9.3.dist-info', 'boto', 'botocore', 'gensim-3.6.0.dist-info', 'file', 'idna-2.7.dist-info', 'jmespath', 'certifi-2018.8.24.dist-info', 'bz2file-0.98-py3.6.egg-info', 'chardet-3.0.4.dist-info', 'docutils-0.14.dist-info', 'bin', 'savedModelText8', 'urllib3-1.23.dist-info', 'smart_open', 'chardet', 'Tutorials.html', 'six-1.11.0.dist-info', 'requests', 'gensim', 'docutils', 'botocore-1.12.18.dist-info', 'boto3-1.9.18.dist-info', '.ipynb_checkpoints', 's3transfer-0.1.13.dist-info', 'bz2file.py', 'numpy-1.15.2.dist-info', '__init__.py', 'six.py', 'certifi', 'boto-2.49.0.dist-info', 'smart_open-1.7.1-py3.6.egg-info', 'scipy', 'boto3', 's3transfer', 'word2vecTalk.ipynb', 'urllib3']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# gensim library also requires numpy, scipy, requests, docutils\n",
    "os.listdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## import data\n",
    "There are lots of possible data sources.  Gensim includes specific loaders for the [Brown corpus](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.BrownCorpus) and Matt Mahoney's [text8 corpus](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Text8Corpus).  The Brown corpus is in the  widely used [Natural Language ToolKit](https://www.nltk.org/).  Text8 is a [cleaned text](http://mattmahoney.net/dc/textdata.html) 100MB selection of Wikipedia, getting rid of html tags, tables, spelling out numbers, etc.  It can be downloaded at http://mattmahoney.net/dc/text8.zip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/home/milton/data/wikidump/text8.txt', 'r') as f:\n",
    "    raw_data = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<class 'str'>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(raw_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100000000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# yah, it is 100 MB\n",
    "len(raw_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' anarchism originated as a term of abuse first used against early working class radicals including the diggers of the english revolution and the sans culottes of the french revolution whilst the term is still used in a pejorative way to describe any act that used violent means to destroy the organization of society it has also been taken up as a positive label by self defined anarchists the word anarchism is derived from the greek without archons ruler chief king anarchism as a political philosophy is the belief that rulers are unnecessary and should be abolished although there are differing interpretations of what this means anarchism also refers to related social movements that advocate the elimination of authoritarian institutions particularly the state the word anarchy as most anarchists use it does not imply chaos nihilism or anomie but rather a harmonious anti authoritarian society in place of what are regarded as authoritarian political structures and coercive economic instituti'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# what does it look like?\n",
    "raw_data[:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = raw_data.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17005207"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "253854"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many unique words?\n",
    "len(set(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tyrranion', 'overbending', 'padshah', 'seutter', 'xaverian', 'pharmacia', 'arplaninac', 'marque', 'homoplasy', 'paraneoplastic', 'ilocano', 'castagnoli', 'yamas', 'hexahedrites', 'aharonov', 'naiditsch', 'upn', 'choo', 'hyland', 'patna', 'harve', 'openable', 'elevate', 'anglophiliac', 'agraphia', 'bank', 'lawmeme', 'playback', 'prado', 'demodulator', 'fortean', 'makkedah', 'catv', 'engages', 'personam', 'diatessaron', 'modwenna', 'nowshak', 'hyperarid', 'khanjan', 'sram', 'erikson', 'khalav', 'infiltrated', 'hama', 'reliving', 'percepts', 'patriae', 'ectotherm', 'rafto', 'subtilatum', 'carpool', 'windle', 'salvific', 'dunk', 'biorobotics', 'dakha', 'rozemond', 'dephasing', 'unexposed', 'qus', 'bontade', 'roddy', 'nathuram', 'metacognition', 'qa', 'overcast', 'codimension', 'facesez', 'polyadenylyl', 'paracelsus', 'mpps', 'gordimer', 'rosie', 'greatly', 'sterilising', 'subbookkeeper', 'spels', 'blackwells', 'crist', 'caked', 'claridade', 'yamin', 'eschenburg', 'sherpa', 'hollyhock', 'freesite', 'ardilaun', 'devarim', 'oldversion', 'wargames', 'lische', 'marissa', 'elrond', 'towikipedia', 'preformationist', 'deathday', 'curtiz', 'omnimax', 'skintones']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the kinds of words you might expect from a Wikipedia corpus\n",
    "list(set(words))[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup data\n",
    "`gensim.models` takes a corpus broken into sentences.  I'm using the `Text8Corpus` iterator that comes as part of the `word2vec` class.  You can use any other data as long as you create an iterable to yield sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = gensim.models.word2vec.Text8Corpus('/home/milton/data/wikidump/text8.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## run model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-11-02 10:02:12,052 : INFO : collecting all words and their counts\n",
      "2018-11-02 10:02:12,056 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types\n",
      "2018-11-02 10:02:17,292 : INFO : collected 253854 word types from a corpus of 17005207 raw words and 1701 sentences\n",
      "2018-11-02 10:02:17,292 : INFO : Loading a fresh vocabulary\n",
      "2018-11-02 10:02:17,670 : INFO : effective_min_count=5 retains 71290 unique words (28% of original 253854, drops 182564)\n",
      "2018-11-02 10:02:17,670 : INFO : effective_min_count=5 leaves 16718844 word corpus (98% of original 17005207, drops 286363)\n",
      "2018-11-02 10:02:17,829 : INFO : deleting the raw counts dictionary of 253854 items\n",
      "2018-11-02 10:02:17,837 : INFO : sample=0.001 downsamples 38 most-common words\n",
      "2018-11-02 10:02:17,838 : INFO : downsampling leaves estimated 12506280 word corpus (74.8% of prior 16718844)\n",
      "2018-11-02 10:02:18,046 : INFO : estimated required memory for 71290 words and 100 dimensions: 92677000 bytes\n",
      "2018-11-02 10:02:18,046 : INFO : resetting layer weights\n",
      "2018-11-02 10:02:18,687 : INFO : training model with 3 workers on 71290 vocabulary and 100 features, using sg=0 hs=0 sample=0.001 negative=5 window=6\n",
      "2018-11-02 10:02:19,704 : INFO : EPOCH 1 - PROGRESS: at 7.35% examples, 898173 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:20,708 : INFO : EPOCH 1 - PROGRESS: at 14.70% examples, 904327 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:21,714 : INFO : EPOCH 1 - PROGRESS: at 21.99% examples, 905184 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:22,719 : INFO : EPOCH 1 - PROGRESS: at 29.22% examples, 906232 words/s, in_qsize 4, out_qsize 0\n",
      "2018-11-02 10:02:23,729 : INFO : EPOCH 1 - PROGRESS: at 36.33% examples, 903657 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:24,730 : INFO : EPOCH 1 - PROGRESS: at 43.03% examples, 892966 words/s, in_qsize 6, out_qsize 0\n",
      "2018-11-02 10:02:25,731 : INFO : EPOCH 1 - PROGRESS: at 49.21% examples, 876167 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:26,747 : INFO : EPOCH 1 - PROGRESS: at 55.61% examples, 865636 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:27,752 : INFO : EPOCH 1 - PROGRESS: at 62.79% examples, 868723 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:28,756 : INFO : EPOCH 1 - PROGRESS: at 69.78% examples, 869142 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:29,760 : INFO : EPOCH 1 - PROGRESS: at 77.13% examples, 872308 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:30,768 : INFO : EPOCH 1 - PROGRESS: at 84.30% examples, 873545 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:31,768 : INFO : EPOCH 1 - PROGRESS: at 91.59% examples, 876312 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:32,771 : INFO : EPOCH 1 - PROGRESS: at 98.41% examples, 873947 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:32,976 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2018-11-02 10:02:32,979 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2018-11-02 10:02:32,982 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2018-11-02 10:02:32,983 : INFO : EPOCH - 1 : training on 17005207 raw words (12504826 effective words) took 14.3s, 874799 effective words/s\n",
      "2018-11-02 10:02:33,990 : INFO : EPOCH 2 - PROGRESS: at 7.17% examples, 885877 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:34,990 : INFO : EPOCH 2 - PROGRESS: at 14.52% examples, 899978 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:35,992 : INFO : EPOCH 2 - PROGRESS: at 22.46% examples, 930507 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:36,995 : INFO : EPOCH 2 - PROGRESS: at 30.34% examples, 946442 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:38,000 : INFO : EPOCH 2 - PROGRESS: at 37.80% examples, 944802 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:39,001 : INFO : EPOCH 2 - PROGRESS: at 45.27% examples, 943232 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:40,013 : INFO : EPOCH 2 - PROGRESS: at 52.67% examples, 939737 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:41,014 : INFO : EPOCH 2 - PROGRESS: at 59.85% examples, 934918 words/s, in_qsize 6, out_qsize 0\n",
      "2018-11-02 10:02:42,016 : INFO : EPOCH 2 - PROGRESS: at 66.90% examples, 929091 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:43,028 : INFO : EPOCH 2 - PROGRESS: at 73.13% examples, 913487 words/s, in_qsize 4, out_qsize 1\n",
      "2018-11-02 10:02:44,030 : INFO : EPOCH 2 - PROGRESS: at 80.31% examples, 910222 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:45,032 : INFO : EPOCH 2 - PROGRESS: at 87.01% examples, 903917 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:46,043 : INFO : EPOCH 2 - PROGRESS: at 94.06% examples, 901307 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:46,858 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2018-11-02 10:02:46,861 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2018-11-02 10:02:46,867 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2018-11-02 10:02:46,868 : INFO : EPOCH - 2 : training on 17005207 raw words (12506115 effective words) took 13.9s, 900880 effective words/s\n",
      "2018-11-02 10:02:47,870 : INFO : EPOCH 3 - PROGRESS: at 7.05% examples, 874316 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:48,875 : INFO : EPOCH 3 - PROGRESS: at 14.34% examples, 888290 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:49,881 : INFO : EPOCH 3 - PROGRESS: at 21.63% examples, 894376 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:50,891 : INFO : EPOCH 3 - PROGRESS: at 28.92% examples, 898577 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:51,896 : INFO : EPOCH 3 - PROGRESS: at 36.10% examples, 899640 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:52,898 : INFO : EPOCH 3 - PROGRESS: at 43.39% examples, 901754 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:53,911 : INFO : EPOCH 3 - PROGRESS: at 50.85% examples, 905280 words/s, in_qsize 6, out_qsize 0\n",
      "2018-11-02 10:02:54,920 : INFO : EPOCH 3 - PROGRESS: at 57.67% examples, 898020 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:55,923 : INFO : EPOCH 3 - PROGRESS: at 64.61% examples, 894719 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:56,935 : INFO : EPOCH 3 - PROGRESS: at 70.61% examples, 879556 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:57,939 : INFO : EPOCH 3 - PROGRESS: at 76.95% examples, 870371 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:58,959 : INFO : EPOCH 3 - PROGRESS: at 83.30% examples, 862268 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:02:59,968 : INFO : EPOCH 3 - PROGRESS: at 90.59% examples, 865572 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:00,982 : INFO : EPOCH 3 - PROGRESS: at 96.00% examples, 851050 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:01,530 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2018-11-02 10:03:01,532 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2018-11-02 10:03:01,536 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2018-11-02 10:03:01,537 : INFO : EPOCH - 3 : training on 17005207 raw words (12504616 effective words) took 14.7s, 852526 effective words/s\n",
      "2018-11-02 10:03:02,552 : INFO : EPOCH 4 - PROGRESS: at 7.11% examples, 870978 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:03,559 : INFO : EPOCH 4 - PROGRESS: at 13.35% examples, 821119 words/s, in_qsize 6, out_qsize 0\n",
      "2018-11-02 10:03:04,564 : INFO : EPOCH 4 - PROGRESS: at 20.11% examples, 827344 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:05,567 : INFO : EPOCH 4 - PROGRESS: at 26.98% examples, 836744 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:06,576 : INFO : EPOCH 4 - PROGRESS: at 33.51% examples, 833307 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:07,585 : INFO : EPOCH 4 - PROGRESS: at 39.45% examples, 817789 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:08,588 : INFO : EPOCH 4 - PROGRESS: at 46.44% examples, 826024 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:09,604 : INFO : EPOCH 4 - PROGRESS: at 52.73% examples, 819864 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:10,607 : INFO : EPOCH 4 - PROGRESS: at 58.32% examples, 806627 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:11,620 : INFO : EPOCH 4 - PROGRESS: at 63.43% examples, 789249 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:12,631 : INFO : EPOCH 4 - PROGRESS: at 70.37% examples, 795704 words/s, in_qsize 5, out_qsize 1\n",
      "2018-11-02 10:03:13,632 : INFO : EPOCH 4 - PROGRESS: at 77.66% examples, 804108 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:14,638 : INFO : EPOCH 4 - PROGRESS: at 84.83% examples, 810555 words/s, in_qsize 5, out_qsize 1\n",
      "2018-11-02 10:03:15,643 : INFO : EPOCH 4 - PROGRESS: at 92.30% examples, 818876 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:16,593 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2018-11-02 10:03:16,600 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2018-11-02 10:03:16,601 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2018-11-02 10:03:16,602 : INFO : EPOCH - 4 : training on 17005207 raw words (12505819 effective words) took 15.1s, 830288 effective words/s\n",
      "2018-11-02 10:03:17,610 : INFO : EPOCH 5 - PROGRESS: at 7.41% examples, 916530 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:18,612 : INFO : EPOCH 5 - PROGRESS: at 14.29% examples, 885656 words/s, in_qsize 6, out_qsize 1\n",
      "2018-11-02 10:03:19,620 : INFO : EPOCH 5 - PROGRESS: at 20.93% examples, 864973 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:20,621 : INFO : EPOCH 5 - PROGRESS: at 25.46% examples, 791414 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:21,625 : INFO : EPOCH 5 - PROGRESS: at 31.69% examples, 790440 words/s, in_qsize 6, out_qsize 0\n",
      "2018-11-02 10:03:22,628 : INFO : EPOCH 5 - PROGRESS: at 38.21% examples, 795418 words/s, in_qsize 6, out_qsize 0\n",
      "2018-11-02 10:03:23,629 : INFO : EPOCH 5 - PROGRESS: at 45.44% examples, 811168 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:24,633 : INFO : EPOCH 5 - PROGRESS: at 52.56% examples, 820962 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:25,635 : INFO : EPOCH 5 - PROGRESS: at 58.44% examples, 811786 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:26,638 : INFO : EPOCH 5 - PROGRESS: at 65.14% examples, 814369 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:27,644 : INFO : EPOCH 5 - PROGRESS: at 71.96% examples, 817692 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:28,650 : INFO : EPOCH 5 - PROGRESS: at 78.84% examples, 819600 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:29,653 : INFO : EPOCH 5 - PROGRESS: at 84.42% examples, 809949 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:30,672 : INFO : EPOCH 5 - PROGRESS: at 90.06% examples, 801560 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:31,685 : INFO : EPOCH 5 - PROGRESS: at 95.18% examples, 789738 words/s, in_qsize 5, out_qsize 0\n",
      "2018-11-02 10:03:32,414 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
      "2018-11-02 10:03:32,421 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
      "2018-11-02 10:03:32,423 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
      "2018-11-02 10:03:32,423 : INFO : EPOCH - 5 : training on 17005207 raw words (12504916 effective words) took 15.8s, 790655 effective words/s\n",
      "2018-11-02 10:03:32,425 : INFO : training on a 85026035 raw words (62526292 effective words) took 73.7s, 847962 effective words/s\n"
     ]
    }
   ],
   "source": [
    "# run model\n",
    "model = gensim.models.Word2Vec(sentences, size = 100, window = 6, min_count=5, workers=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that due to the stochastic nature of network training, the weights in the models will be different.  Each trained model will yield different results below in [Distance from mean](#distance-from-mean) or [Similarity](#similarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-11-02 10:03:32,431 : INFO : saving Word2Vec object under savedModelText8, separately None\n",
      "2018-11-02 10:03:32,434 : INFO : not storing attribute vectors_norm\n",
      "2018-11-02 10:03:32,435 : INFO : not storing attribute cum_table\n",
      "2018-11-02 10:03:33,194 : INFO : saved savedModelText8\n"
     ]
    }
   ],
   "source": [
    "# save it as binary\n",
    "model.save('savedModelText8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word2Vec(vocab=71290, size=100, alpha=0.025)\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get list of word vectors\n",
    "words = list(model.wv.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get sorted list of word vectors\n",
    "words = list(model.wv.index2word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "71290"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'of', 'and', 'one', 'in', 'a', 'to', 'zero', 'nine', 'two', 'is', 'as', 'eight', 'for', 's', 'five', 'three', 'was', 'by', 'that', 'four', 'six', 'seven', 'with', 'on', 'are', 'it', 'from', 'or', 'his', 'an', 'be', 'this', 'which', 'at', 'he', 'also', 'not', 'have', 'were', 'has', 'but', 'other', 'their', 'its', 'first', 'they', 'some', 'had', 'all', 'more', 'most', 'can', 'been', 'such', 'many', 'who', 'new', 'used', 'there', 'after', 'when', 'into', 'american', 'time', 'these', 'only', 'see', 'may', 'than', 'world', 'i', 'b', 'would', 'd', 'no', 'however', 'between', 'about', 'over', 'years', 'states', 'people', 'war', 'during', 'united', 'known', 'if', 'called', 'use', 'th', 'system', 'often', 'state', 'so', 'history', 'will', 'up', 'while', 'where', 'city', 'being', 'english', 'then', 'any', 'both', 'under', 'out', 'made', 'well', 'her', 'e', 'number', 'government', 'them', 'm', 'later', 'since', 'him', 'part', 'name', 'c', 'century', 'through', 'because', 'x', 'university', 'early', 'life', 'british', 'year', 'like', 'same', 'including', 'became', 'example', 'day', 'each', 'even', 'work', 'language', 'although', 'several', 'form', 'john', 'u', 'national', 'very', 'much', 'g', 'french', 'before', 'general', 'what', 't', 'against', 'n', 'high', 'links', 'could', 'based', 'those', 'now', 'second', 'de', 'music', 'another', 'large', 'she', 'f', 'external', 'german', 'different', 'modern', 'great', 'do', 'common', 'set', 'list', 'south', 'series', 'major', 'game', 'power', 'long', 'country', 'king', 'law', 'group', 'film', 'still', 'until', 'north', 'international', 'term', 'we', 'end', 'book', 'found', 'own']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words[:200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the index for a word\n",
    "model.wv.vocab['one'].index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mType:\u001b[0m        Word2VecKeyedVectors\n",
       "\u001b[0;31mString form:\u001b[0m <gensim.models.keyedvectors.Word2VecKeyedVectors object at 0x7f6b52182ef0>\n",
       "\u001b[0;31mFile:\u001b[0m        ~/programming/python/gensim/gensim/models/keyedvectors.py\n",
       "\u001b[0;31mDocstring:\u001b[0m  \n",
       "Mapping between words and vectors for the :class:`~gensim.models.Word2Vec` model.\n",
       "Used to perform operations on the vectors such as vector lookup, distance, similarity etc.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.wv?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.24125664,  1.9268551 , -2.2396307 , -0.9135645 ,  0.30719054,\n",
       "       -0.3901619 , -0.72596925,  1.6858315 ,  1.9339089 , -1.8731937 ,\n",
       "        0.19566078, -0.5991121 , -1.479587  , -1.5020843 ,  0.18137535,\n",
       "       -0.23894407, -0.12432036, -0.05521582, -0.1348761 , -0.8019359 ,\n",
       "       -1.9279583 , -1.3177425 , -0.33780292,  1.0493244 ,  0.31354004,\n",
       "        1.8714385 , -0.8742712 , -0.17126894, -0.14679043, -4.7852097 ,\n",
       "       -0.3815103 ,  1.8579111 , -1.7332219 ,  0.7389472 ,  2.1735961 ,\n",
       "        1.7000883 ,  1.0617127 , -2.405899  , -1.6414772 , -0.63530964,\n",
       "       -1.3333199 ,  1.3860669 ,  0.42986315, -0.53261244,  0.11340322,\n",
       "       -1.9065561 , -1.080117  , -1.1631343 ,  0.42832732,  0.8277875 ,\n",
       "       -0.8820234 ,  1.0434159 ,  1.4340398 ,  0.28658646,  0.54299736,\n",
       "       -2.195669  , -1.9429504 ,  2.371765  ,  0.52610004,  0.31313244,\n",
       "        0.95485383,  1.1148375 ,  0.38447228, -0.74583143, -0.2827666 ,\n",
       "        0.9085859 , -0.1010727 ,  1.9584153 , -0.6831744 , -0.3444428 ,\n",
       "       -0.5328941 , -0.60484874, -0.5407519 ,  2.0439267 , -0.45526335,\n",
       "        1.2840799 ,  0.01451097,  0.20404617, -1.118855  , -1.1415514 ,\n",
       "        0.63656855, -1.3573003 ,  1.0828034 ,  1.7164483 ,  3.0423548 ,\n",
       "       -0.47130018, -0.5705444 ,  0.41316906,  0.7004269 ,  0.73310083,\n",
       "        0.26853767,  1.5918186 ,  0.9021663 , -0.83974534, -0.9439738 ,\n",
       "        1.904814  , -0.6161951 ,  0.5150735 , -0.5842606 ,  0.05099605],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.get_vector('one')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(model.wv.get_vector('one'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distance from mean\n",
    "<a id=\"distance-from-mean\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdoesnt_match\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Which word from the given list doesn't go with the others?\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "words : list of str\n",
       "    List of words.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "str\n",
       "    The word further away from the mean of all words.\n",
       "\u001b[0;31mFile:\u001b[0m      ~/programming/python/gensim/gensim/models/keyedvectors.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.wv.doesnt_match?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-11-02 10:03:33,670 : INFO : precomputing L2-norms of word weight vectors\n",
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'cereal'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find word in list that is farthest from the mean\n",
    "model.wv.doesnt_match(\"breakfast cereal dinner lunch\".split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'sport'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.doesnt_match(\"cook janitor pilot sport teacher\".split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'angst'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.doesnt_match(\"joy time peace angst\".split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'peace'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.doesnt_match(\"joy timely peace angst\".split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'peace'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.doesnt_match(\"joy peace angst\".split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similarity\n",
    "<a id=\"similarity\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cosine similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msimilarity\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mw1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mw2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Compute cosine similarity between two words.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "w1 : str\n",
       "    Input word.\n",
       "w2 : str\n",
       "    Input word.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "float\n",
       "    Cosine similarity between `w1` and `w2`.\n",
       "\u001b[0;31mFile:\u001b[0m      ~/programming/python/gensim/gensim/models/keyedvectors.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.wv.similarity?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.72925234"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.similarity('woman', 'man')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.29012758"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.similarity('woman', 'tree')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.40712953"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.similarity('tree', 'shrub')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "-0.13019277"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.similarity('tree', 'bush')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7098724246025085"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# distance is just the opposite of similarity\n",
    "model.wv.distance('woman', 'tree')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.distance('woman', 'tree') + model.wv.similarity('woman', 'tree')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('child', 0.8045989274978638), ('girl', 0.7382532358169556), ('man', 0.7292523384094238), ('herself', 0.7025970816612244), ('lady', 0.6832849979400635), ('lover', 0.6755771636962891), ('mother', 0.6658172607421875), ('wife', 0.6456671357154846), ('prostitute', 0.6433481574058533), ('person', 0.6396008729934692)]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# closest by cosine similarity\n",
    "model.wv.similar_by_word('woman', topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('he', 0.8081147074699402), ('herself', 0.7315353155136108), ('her', 0.7108720541000366), ('leto', 0.6110916137695312), ('nobody', 0.6102216243743896), ('faramir', 0.5981771349906921), ('odrade', 0.5976393222808838), ('him', 0.5975172519683838), ('rachel', 0.5924820899963379), ('baldrick', 0.5899745225906372)]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# closest by cosine similarity\n",
    "model.wv.similar_by_word('she', topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmost_similar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpositive\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnegative\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtopn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrestrict_vocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindexer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Find the top-N most similar words.\n",
       "Positive words contribute positively towards the similarity, negative words negatively.\n",
       "\n",
       "This method computes cosine similarity between a simple mean of the projection\n",
       "weight vectors of the given words and the vectors for each word in the model.\n",
       "The method corresponds to the `word-analogy` and `distance` scripts in the original\n",
       "word2vec implementation.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "positive : list of str, optional\n",
       "    List of words that contribute positively.\n",
       "negative : list of str, optional\n",
       "    List of words that contribute negatively.\n",
       "topn : int, optional\n",
       "    Number of top-N similar words to return.\n",
       "restrict_vocab : int, optional\n",
       "    Optional integer which limits the range of vectors which\n",
       "    are searched for most-similar values. For example, restrict_vocab=10000 would\n",
       "    only check the first 10000 word vectors in the vocabulary order. (This may be\n",
       "    meaningful if you've sorted the vocabulary by descending frequency.)\n",
       "\n",
       "Returns\n",
       "-------\n",
       "list of (str, float)\n",
       "    Sequence of (word, similarity).\n",
       "\u001b[0;31mFile:\u001b[0m      ~/programming/python/gensim/gensim/models/keyedvectors.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.wv.most_similar?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('child', 0.8045989274978638), ('girl', 0.7382532358169556), ('man', 0.7292523384094238), ('herself', 0.7025970816612244), ('lady', 0.6832849979400635), ('lover', 0.6755771636962891), ('mother', 0.6658172607421875), ('wife', 0.6456671357154846), ('prostitute', 0.6433481574058533), ('person', 0.6396008729934692)]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(positive=['woman'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('hippocampal', 0.43745601177215576), ('touman', 0.3993116021156311), ('automated', 0.3980580270290375), ('abm', 0.3962332010269165), ('districting', 0.38271263241767883), ('automation', 0.378354012966156), ('divestiture', 0.3775786757469177), ('operations', 0.37553471326828003), ('loran', 0.37270671129226685), ('samsung', 0.36578959226608276)]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(negative=['woman'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('queen', 0.73751300573349), ('princess', 0.7058136463165283), ('son', 0.691680908203125), ('man', 0.6852022409439087), ('daughter', 0.6846308708190918), ('lady', 0.6647908091545105), ('bride', 0.6599146723747253), ('prince', 0.657150387763977), ('lover', 0.6516727209091187), ('wife', 0.648215651512146)]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(positive=['woman', 'king'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('queen', 0.6976450085639954), ('daughter', 0.6202482581138611), ('empress', 0.6175616979598999), ('princess', 0.6116492748260498), ('matilda', 0.6097214221954346), ('prince', 0.6079800128936768), ('son', 0.5972870588302612), ('isabella', 0.5928862690925598), ('aquitaine', 0.5920809507369995), ('jadwiga', 0.5864522457122803)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(positive=['woman', 'king'], negative=['man'], topn=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiplicative combination"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmost_similar_cosmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpositive\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnegative\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtopn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Find the top-N most similar words, using the multiplicative combination objective,\n",
       "proposed by `Omer Levy and Yoav Goldberg \"Linguistic Regularities in Sparse and Explicit Word Representations\"\n",
       "<http://www.aclweb.org/anthology/W14-1618>`_. Positive words still contribute positively towards the similarity,\n",
       "negative words negatively, but with less susceptibility to one large distance dominating the calculation.\n",
       "In the common analogy-solving case, of two positive and one negative examples,\n",
       "this method is equivalent to the \"3CosMul\" objective (equation (4)) of Levy and Goldberg.\n",
       "\n",
       "Additional positive or negative examples contribute to the numerator or denominator,\n",
       "respectively - a potentially sensible but untested extension of the method.\n",
       "With a single positive example, rankings will be the same as in the default\n",
       ":meth:`~gensim.models.keyedvectors.WordEmbeddingsKeyedVectors.most_similar`.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "positive : list of str, optional\n",
       "    List of words that contribute positively.\n",
       "negative : list of str, optional\n",
       "    List of words that contribute negatively.\n",
       "topn : int, optional\n",
       "    Number of top-N similar words to return.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "list of (str, float)\n",
       "    Sequence of (word, similarity).\n",
       "\u001b[0;31mFile:\u001b[0m      ~/programming/python/gensim/gensim/models/keyedvectors.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.wv.most_similar_cosmul?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('queen', 0.9062806963920593), ('matilda', 0.8731217980384827), ('empress', 0.8670921921730042), ('daughter', 0.8613948822021484), ('princess', 0.8548787832260132), ('aquitaine', 0.8529645204544067), ('jadwiga', 0.8512430191040039), ('isabella', 0.8460915684700012), ('son', 0.8443405628204346), ('prince', 0.8387503623962402)]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar_cosmul(positive=['woman', 'king'], negative=['man'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('queen', 0.6976450085639954), ('daughter', 0.6202482581138611), ('empress', 0.6175616979598999), ('princess', 0.6116492748260498), ('matilda', 0.6097214221954346), ('prince', 0.6079800128936768), ('son', 0.5972870588302612), ('isabella', 0.5928862690925598), ('aquitaine', 0.5920809507369995), ('jadwiga', 0.5864522457122803)]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(positive=['woman', 'king'], negative=['man'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('queen', 0.6143661141395569), ('princess', 0.600183367729187), ('son', 0.5903830528259277), ('daughter', 0.5863444805145264), ('man', 0.5782984495162964), ('bride', 0.5700618028640747), ('lady', 0.5688951015472412), ('wife', 0.5610396265983582), ('lover', 0.560962438583374), ('prince', 0.5588306188583374)]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar_cosmul(positive=['woman', 'king'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/milton/programming/python/gensim/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('queen', 0.73751300573349), ('princess', 0.7058136463165283), ('son', 0.691680908203125), ('man', 0.6852022409439087), ('daughter', 0.6846308708190918), ('lady', 0.6647908091545105), ('bride', 0.6599146723747253), ('prince', 0.657150387763977), ('lover', 0.6516727209091187), ('wife', 0.648215651512146)]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(positive=['woman', 'king'], topn=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "[<img style=\"float: left;\" src=\"https://i.creativecommons.org/l/by-sa/4.0/88x31.png\">](http://creativecommons.org/licenses/by-sa/4.0/)  \n",
    "\n",
    "Licensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-sa/4.0/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}