2_intermediate/4.1-Sequences.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequences\n",
    "\n",
    "In this lab we will introduce data which has internal dependencies on previous relations in a sequence of data points, and how to model such data.\n",
    "\n",
    "Examples of data with a sequence dimension are stock prices, weather data, protein sequences, speech, text, and many more.\n",
    "In previous labs we mainly considered data $x \\in \\mathrm{R}^d$, where $d$ is the feature space.\n",
    "With time sequences our data can be represented as $x \\in \\mathrm{R}^{t \\, \\times \\, d}$, where $t$ is the sequence length. This emphasises sequence dependence and that the samples along the sequence are not independent and identically distributed (i.i.d.).\n",
    "\n",
    "In the following we will exemplify methods on text given the same challenges as presented in [learning when to skim and when to read](https://einstein.ai/research/learning-when-to-skim-and-when-to-read)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text classification: Sentiment analysis\n",
    "\n",
    "In our first work on sequences we will classify sequences of text.\n",
    "We will model functions as $\\mathrm{R}^{t \\, \\times \\, d} \\rightarrow \\mathrm{R}^c$, where $c$ is the amount of classes in the output.\n",
    "\n",
    "With text the challenge is how to represent a word as the feature $d$, as it is required to represent text with decimal numbers.\n",
    "Currently, two popular approaches exist; one-hot encoding and embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One-hot encoding over vocabulary\n",
    "\n",
    "One way to represent a fixed amount of words is by making a one-hot encoded vector, which consists of 0s in all cells with the exception of a single 1 in a cell used uniquely to identify each word.\n",
    "\n",
    "| vocabulary    | one-hot encoded vector   |\n",
    "| ------------- |--------------------------|\n",
    "| Copenhagen    | $= [0, 0, 1, \\ldots, 0]$ |\n",
    "| Paris         | $= [1, 0, 0, \\ldots, 0]$ |\n",
    "| Rome          | $= [0, 1, 0, \\ldots, 0]$ |\n",
    "\n",
    "Representing a large vocabulary with one-hot encodings often becomes inefficient because of the size of each sparse vector.\n",
    "To overcome this challenge it is common practice to truncate the vocabulary to contain the $k$ most used words and represent the rest with a special symbol, $\\mathtt{UNK}$, to define unknown/unimportant words.\n",
    "This often causes entities such as names to be represented with $\\mathtt{UNK}$.\n",
    "\n",
    "Consider the following text\n",
    "> I love the corny jokes in Spielberg's new movie.\n",
    "\n",
    "where an example result would be similar to\n",
    "> I love the corny jokes in $\\mathtt{UNK}$'s new movie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "Word embeddings tries to tackle the intractability of one-hot encoded vectors, as $k$ is often in the range of 50k to 100k elements.\n",
    "Furthermore, one-hot encoding of vectors assumes orthogonality between all words, which makes it inept to incorporate relationships between words, e.g. `ran` and `run` should be related, where e.g. `awkward` and `space` should be far apart in the vector space.\n",
    "\n",
    "An embedding is defined as $\\mathrm{R}^d \\rightarrow \\mathrm{R}^{d'}$, where $d' \\ll d$.\n",
    "In practice this is often achieved by having a lookup table with $d'$-dimensional embeddings, similar to the following matrix operation $\\mathrm{R}^d \\cdot \\mathrm{R}^{d \\, \\times \\, d'}$.\n",
    "\n",
    "For visualizations and more intuition check out [learning when to skim and when to read](https://einstein.ai/research/learning-when-to-skim-and-when-to-read)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of Words\n",
    "\n",
    "A simple way to model sequences of words is by averaging the word embeddings across the sequence dimension.\n",
    "This gives us a vector which has a little information of each word, although completely disregarding the order of the words. Even though this might seem like a lossy approach to condense information it works surprisingly well.\n",
    "\n",
    "A bag of words model is represented as $\\mathrm{R}^{t \\, \\times \\, d'} \\rightarrow \\mathrm{R}^{d'}$, afterwards the representation can be used to do classification $\\mathrm{R}^{d'} \\rightarrow \\mathrm{R}^{c}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stanford sentiment treebank\n",
    "\n",
    "A great public dataset for sentiment analysis is the Stanford sentiment treebank (SST).\n",
    "The SST provides not only the class (positive, negative) for a sentence, but also each of its grammatical subphrases.\n",
    "We will not utilize any tree information.\n",
    "The original SST constitutes five classes: *very positive*, *positive*, *neutral*, *negative* and *very negative*.\n",
    "We consider the simpler task of binary classification where *very positive* is combined with *positive*, *very negative* is combined with *negative* and all *neutrals* are removed.\n",
    "\n",
    "## positive examples\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "  <ul>\n",
    "    <li>The actors are fantastic</li>\n",
    "    <li>A smart, witty follow-up.</li>\n",
    "    <li>You'll probably love it.</li>\n",
    "  </ul>\n",
    "</div>\n",
    "\n",
    "## negative examples\n",
    "\n",
    "<div class=\"alert alert-danger\">\n",
    "  <ul>\n",
    "    <li>Unflinchingly bleak and desperate.</li>\n",
    "    <li>An absurdist spider web.</li>\n",
    "    <li>Who cares?</li>\n",
    "  </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import stuff\n",
    "from pprint import pprint\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from torchtext import data\n",
    "from torchtext import datasets\n",
    "from torchtext.vocab import Vectors\n",
    "\n",
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.nn import Linear\n",
    "from torch.nn.functional import softmax, relu\n",
    "\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "# we'll use the bokeh library to create beautiful plots\n",
    "# *_notebook functions are needed for correct use in jupyter\n",
    "from bokeh.plotting import figure, ColumnDataSource\n",
    "from bokeh.models import HoverTool\n",
    "from bokeh.io import output_notebook, show, push_notebook\n",
    "output_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "use_cuda = torch.cuda.is_available()\n",
    "\n",
    "def get_variable(x):\n",
    "    \"\"\" Converts tensors to cuda, if available. \"\"\"\n",
    "    if use_cuda:\n",
    "        return x.cuda()\n",
    "    return x\n",
    "\n",
    "def get_numpy(x):\n",
    "    \"\"\" Get numpy array for both cuda and not. \"\"\"\n",
    "    if use_cuda:\n",
    "        return x.cpu().data.numpy()\n",
    "    return x.data.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data loader - `torchtext`\n",
    "\n",
    "Creating data loaders for NLP is quite a hassle.\n",
    "[torchtext](https://github.com/pytorch/text/) is a convenient library with builtin functionality useful when working with text, e.g. building vocabularies and padding sequences to max length."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `torchtext` - Fields and Dataset\n",
    "\n",
    "Our dataset must have a predefined structure, e.g. similar to a database table.\n",
    "\n",
    "- `torchtext.data.Field()` defines a column in our dataset table\n",
    "- `torchtext.datasets.SST` is a data loader for the Stanford Sentiment Treebank (SST) dataset\n",
    "- `torchtext.datasets.SST.split()` is a function to create train/validation/test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we assume that all fields are sequential, i.e. there will be a sequence of data\n",
    "# however, the label field will not contain any sequence\n",
    "TEXT = data.Field(sequential=True)\n",
    "LABEL = data.Field(sequential=False)\n",
    "# create SST dataset splits\n",
    "# note, we remove samples with neutral labels\n",
    "train_set, validation_set, _ = datasets.SST.splits(TEXT,\n",
    "                                                   LABEL,\n",
    "                                                   fine_grained=False,\n",
    "                                                   train_subtrees=True,\n",
    "                                                   filter_pred=lambda ex: ex.label != 'neutral')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print('train_set.fields:', list(train_set.fields.keys()))\n",
    "print('validation_set.fields:', list(validation_set.fields.keys()))\n",
    "print()\n",
    "print('size of training set', len(train_set))\n",
    "print('size of validation set', len(validation_set))\n",
    "print()\n",
    "print('content of first training sample:')\n",
    "pprint(vars(train_set[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `torchtext` - Vocabulary\n",
    "\n",
    "For each `Field` we build a vocabulary to numericalize the symbols, e.g. `\"fun\" => 471`.\n",
    "When building a vocabulary we can attach embedding vectors, e.g. GloVe, FastText, etc.\n",
    "Many of these are already built into `torchtext`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build the vocabularies\n",
    "url = 'https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.simple.vec'\n",
    "TEXT.build_vocab(train_set, max_size=None, vectors=Vectors('wiki.simple.vec', url=url))\n",
    "LABEL.build_vocab(train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Text fields:')\n",
    "#print('keys of TEXT.vocab:', list(TEXT.vocab.__dict__.keys()))\n",
    "print(' size of vocabulary:', len(TEXT.vocab))\n",
    "print(\" vocabulary's embedding dimension:\", TEXT.vocab.vectors.size())\n",
    "print(' no. times the \"fun\" appear in the dataset:', TEXT.vocab.freqs['fun'])\n",
    "\n",
    "print('\\nLabel fields:')\n",
    "#print('keys of LABEL.vocab:', list(LABEL.vocab.__dict__.keys()))\n",
    "print(\" list of vocabulary (int-to-str):\", LABEL.vocab.itos)\n",
    "print(\" list of vocabulary (str-to-int):\", dict(LABEL.vocab.stoi))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `torchtext` - Iterator over datasets\n",
    "\n",
    "`torchtext.data.Iterator` is a class which can be used to create iterators.\n",
    "These iterators have various useful functionality, e.g. to shuffle at every epoch, or to generate data endlessly.\n",
    "It is useful to be able to generate endless batches of training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make iterator for splits\n",
    "# device gives a CUDA enabled device (-1 disables it)\n",
    "train_iter, val_iter, _ = data.BucketIterator.splits((train_set, validation_set, _),\n",
    "                                                     batch_size=128, \n",
    "                                                     device=0 if use_cuda else -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print batch information\n",
    "batch = next(iter(train_iter))\n",
    "print(\"dimension of batch's text:\", batch.text.size())\n",
    "print(\"first sequence in text:\", batch.text[:,0])\n",
    "print(\"correct label index:\", batch.label[0])\n",
    "print(\"the actual label:\", LABEL.vocab.itos[get_numpy(batch.label[0])[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# size of embeddings\n",
    "embedding_dim = TEXT.vocab.vectors.size()[1]\n",
    "num_embeddings = TEXT.vocab.vectors.size()[0]\n",
    "num_classes = len(LABEL.vocab.itos)\n",
    "\n",
    "class Net(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.embeddings = nn.Embedding(num_embeddings, embedding_dim)\n",
    "        # use pretrained embeddings\n",
    "        self.embeddings.weight.data.copy_(TEXT.vocab.vectors)\n",
    "        \n",
    "        # add hidden layers\n",
    "#         self.l_1 = Linear(in_features=embedding_dim,\n",
    "#                           out_features=30,\n",
    "#                           bias=True)\n",
    "#         self.l_2 = Linear(in_features=30,\n",
    "#                           out_features=30,\n",
    "#                           bias=True)\n",
    "        # output layer\n",
    "        self.l_out = Linear(in_features=embedding_dim,\n",
    "                            out_features=num_classes,\n",
    "                            bias=False)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        out = {}\n",
    "        # get embeddings\n",
    "        x = self.embeddings(x)\n",
    "        # mean embeddings, this is the bag of words trick\n",
    "        out['bow'] = x = torch.mean(x, dim=0)\n",
    "        # apply layer\n",
    "        #out['l1_activations'] = x = relu(self.l_1(x))\n",
    "        #out['l2_activations'] = x = relu(self.l_2(x))\n",
    "        # classify\n",
    "        out['out'] = softmax(self.l_out(x), dim=1)\n",
    "        return out\n",
    "\n",
    "net = Net()\n",
    "if use_cuda:\n",
    "    net.cuda()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(net.parameters(), lr=0.001)\n",
    "\n",
    "def accuracy(ys, ts):\n",
    "    # making a one-hot encoded vector of correct (1) and incorrect (0) predictions\n",
    "    correct_prediction = torch.eq(torch.max(ys, 1)[1], ts)\n",
    "    # averaging the one-hot encoded vector\n",
    "    return torch.mean(correct_prediction.float())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def construct_sentences(batch):\n",
    "    \"\"\"    \n",
    "    Parameters\n",
    "    ----------\n",
    "    batch: torchtext.data.batch.Batch\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    [str]\n",
    "    \"\"\"\n",
    "    return [\" \".join([TEXT.vocab.itos[elm] \n",
    "                      for elm in get_numpy(batch.text[:,i])])\n",
    "            for i in range(batch.text.size()[1])]\n",
    "\n",
    "def get_labels(batch):\n",
    "    \"\"\"    \n",
    "    Parameters\n",
    "    ----------\n",
    "    batch: torchtext.data.batch.Batch\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    [str]\n",
    "    \"\"\"\n",
    "    return [LABEL.vocab.itos[get_numpy(batch.label[i])[0]] for i in range(len(batch.label))]\n",
    "#construct_sentences(batch)\n",
    "#get_labels(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to project our hidden embeddings to a visualizable space\n",
    "tsne = TSNE(perplexity=10.0, learning_rate=5.0, n_iter=2000)\n",
    "\n",
    "# index for each label\n",
    "colormap = {1: 'DodgerBlue', 2: 'FireBrick'}\n",
    "# create a tmp source to be updated later\n",
    "validation_set_size = len(validation_set)\n",
    "source = ColumnDataSource(data={'x': np.random.randn(validation_set_size),\n",
    "                                'y': np.random.randn(validation_set_size),\n",
    "                                'colors': ['green']*validation_set_size,\n",
    "                                'sentences': [\"tmp\"]*validation_set_size,\n",
    "                                'labels': [\"unk\"]*validation_set_size})\n",
    "# instance to define hover logic in plot\n",
    "hover = HoverTool(tooltips=[(\"Sentence\", \"@sentences\"), (\"Label\", \"@labels\")])\n",
    "\n",
    "# set up the bokeh figure for later visualizations\n",
    "p = figure(tools=[hover])\n",
    "p.circle(x='x', y='y', fill_color='colors', size=5, line_color=None, source=source)\n",
    "\n",
    "def update_plot(meta, layer, handle):\n",
    "    \"\"\" Update existing plot\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    meta: dict\n",
    "    layer: str\n",
    "    \"\"\"\n",
    "    tsne_acts = tsne.fit_transform(meta[layer])\n",
    "    source.data['x'] = tsne_acts[:,0]\n",
    "    source.data['y'] = tsne_acts[:,1]\n",
    "    source.data['colors'] = [colormap[l] for l in meta['label_idx']]\n",
    "    \n",
    "    source.data['sentences'] = meta['sentences']\n",
    "    source.data['labels'] = meta['labels']\n",
    "    \n",
    "    # this updates the given plot\n",
    "    push_notebook(handle=handle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the bag of words model\n",
    "\n",
    "**Warning** this might take a while on CPU.\n",
    "Go get a cop of coffe, and enjoy the visualizations.\n",
    "\n",
    "Notice that each data point in the plot corresponds to an entire sentence in the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "max_iter = 3000\n",
    "eval_every = 1000\n",
    "log_every = 200\n",
    "\n",
    "# will be updated while iterating\n",
    "tsne_plot = show(p, notebook_handle=True)\n",
    "\n",
    "train_loss, train_accs = [], []\n",
    "\n",
    "net.train()\n",
    "for i, batch in enumerate(train_iter):\n",
    "    if i % eval_every == 0:\n",
    "        net.eval()\n",
    "        val_losses, val_accs, val_lengths = 0, 0, 0\n",
    "        val_meta = {'label_idx': [], 'sentences': [], 'labels': []}\n",
    "        for val_batch in val_iter:\n",
    "            output = net(val_batch.text)\n",
    "            # batches sizes might vary, which is why we cannot just mean the batch's loss\n",
    "            # we multiply the loss and accuracies with the batch's size,\n",
    "            # to later divide by the total size\n",
    "            val_losses += criterion(output['out'], val_batch.label) * val_batch.batch_size\n",
    "            val_accs += accuracy(output['out'], val_batch.label) * val_batch.batch_size\n",
    "            val_lengths += val_batch.batch_size\n",
    "            \n",
    "            for key, _val in output.items():\n",
    "                if key not in val_meta:\n",
    "                    val_meta[key] = []\n",
    "                val_meta[key].append(get_numpy(_val)) \n",
    "            val_meta['label_idx'].append(get_numpy(val_batch.label))\n",
    "            val_meta['sentences'].append(construct_sentences(val_batch))\n",
    "            val_meta['labels'].append(get_labels(val_batch))\n",
    "        \n",
    "        for key, _val in val_meta.items():\n",
    "            val_meta[key] = np.concatenate(_val)\n",
    "        \n",
    "        # divide by the total accumulated batch sizes\n",
    "        val_losses /= val_lengths\n",
    "        val_accs /= val_lengths\n",
    "        \n",
    "        print(\"it: {} loss: {:.2f} accs: {:.2f}\".format(i, get_numpy(val_losses)[0], get_numpy(val_accs)[0]))\n",
    "        update_plot(val_meta, 'bow', tsne_plot)\n",
    "        \n",
    "        net.train()\n",
    "    \n",
    "    output = net(batch.text)\n",
    "    batch_loss = criterion(output['out'], batch.label)\n",
    "    \n",
    "    train_loss.append(get_numpy(batch_loss))\n",
    "    train_accs.append(get_numpy(accuracy(output['out'], batch.label)))\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    batch_loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if i % log_every == 0:        \n",
    "        print(\"train, it: {} loss: {:.2f} accs: {:.2f}\".format(i, \n",
    "                                                               np.mean(train_loss), \n",
    "                                                               np.mean(train_accs)))\n",
    "        # reset\n",
    "        train_loss, train_accs = [], []\n",
    "        \n",
    "    if max_iter < i:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignments\n",
    "\n",
    "## Assignment 1 - add hidden layer\n",
    "\n",
    "- add one hidden layer to the bag of words (BoW) model\n",
    "  - plot the hidden layer's activations instead of the BoW representation\n",
    "- add a second hidden layer\n",
    "  - try and plot the activations of the second hidden layer\n",
    "\n",
    "Notice any difference in the plots?\n",
    "Describe what you see.\n",
    "Hover over the data points."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}