week3_Embeddings_colab_final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4GYv0hWy7DEt"
   },
   "source": [
    "# Find duplicate questions on StackOverflow by their embeddings\n",
    "\n",
    "In this assignment you will learn how to calculate a similarity for pieces of text. Using this approach you will know how to find duplicate questions from [StackOverflow](https://stackoverflow.com)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dgwB8R9j7DEx"
   },
   "source": [
    "### Libraries\n",
    "\n",
    "In this task you will you will need the following libraries:\n",
    "- [StarSpace](https://github.com/facebookresearch/StarSpace) — a general-purpose model for efficient learning of entity embeddings from Facebook\n",
    "- [Gensim](https://radimrehurek.com/gensim/) — a tool for solving various NLP-related tasks (topic modeling, text representation, ...)\n",
    "- [Numpy](http://www.numpy.org) — a package for scientific computing.\n",
    "- [scikit-learn](http://scikit-learn.org/stable/index.html) — a tool for data mining and data analysis.\n",
    "- [Nltk](http://www.nltk.org) — a platform to work with human language data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "id": "1U1eCAHM7DE3",
    "outputId": "6e48230a-90ab-4b2c-d217-7c680582c448"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'natural-language-processing'...\n",
      "remote: Enumerating objects: 3, done.\u001b[K\n",
      "remote: Counting objects: 100% (3/3), done.\u001b[K\n",
      "remote: Compressing objects: 100% (3/3), done.\u001b[K\n",
      "remote: Total 650 (delta 0), reused 1 (delta 0), pack-reused 647\n",
      "Receiving objects: 100% (650/650), 228.73 KiB | 438.00 KiB/s, done.\n",
      "Resolving deltas: 100% (371/371), done.\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!git clone https://github.com/hse-aml/natural-language-processing\n",
    "sys.path.append(\"/content/natural-language-processing\")\n",
    "sys.path.append(\"/content/natural-language-processing/week3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 279,
     "referenced_widgets": [
      "1110c036cc4a4cd595a4a844b1a41883",
      "0672487f0c9a41388889449eb1783365",
      "19fb87ca329647fc9a126f520e818d4c",
      "d91135a8bad44ef8bc5ca456465a545f",
      "83164a11df3a439ba793207c043e7c3f",
      "42ec9a54fd864a74b9b7e9c59f58a2ef",
      "8d963d100aa84e9eab346822533ba948",
      "5eaf251296e049d68500612b60ae73e3",
      "aca691b408ef4e74b59a932be8d3b9b1",
      "7201ad2a82054603ba30d3ba649aa873",
      "bbafc7af5afe450b8db6b40ed4497ae8",
      "460b0f6d77484794bc63e9311eb5f38d",
      "9dfa47b67e114079b0f31db867180c2d",
      "7bb3d98550ff40d58ca8b281c935f7e0",
      "db97e55fc65f4f84b899c33630c31211",
      "ec06b708ff0944d093d0fa8ae264e5ad",
      "731830e3cd8d428aa6f060324887e1d8",
      "6c884661dc9e4277a59cf20e3f4308db",
      "f199c968aca44eb580872f18a75d1b6a",
      "c287c385af9342b889a3a5ff5bb8d541",
      "75683a90194444a1a00fffafd6e2cb94",
      "4f07c2f10cfd43cd83a1d4fec5d63723",
      "9d5e66da0fdb4d648d9435e3b03a321f",
      "63572138a97940e0b844ee2b3ecdd37e",
      "3cfa31d9e32a4c9e8d2377df0341bb8f",
      "e343370fff704459983677226e89c4e4",
      "7af713c5ad5c46069a818af2d185519e",
      "8fc7851ee1e648a8a8e95f3d2de125e9",
      "7e24556aab8941b4817c3aec4f7a9c53",
      "2fd4ab2863e649368eca57d04324e366",
      "03453a45770a4cdd9ee86668a548175f",
      "963ac1a5aa344bc99f353d2e8ac759a8",
      "5cf8df791ebd43e1b10b9a746e43b731",
      "7af1ea82bf9d4cd681f479b899611cfd",
      "da3e40b27bc14fdd99921adde06eaaad",
      "c3e2733bc15642c6a3270c1e8a908558",
      "e72d84d4e4f44b2fa944b4f6fcc06b14",
      "9a8563ae67cb4e519568b70e504a58f4",
      "b0a9ea14160344a5a06019885be5be6b",
      "29cf43ae622546ba8c4955ae418f1de6"
     ]
    },
    "colab_type": "code",
    "id": "v5DifgZxm3hb",
    "outputId": "980bba23-25e4-45fb-b330-2a78d379f0c9"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1110c036cc4a4cd595a4a844b1a41883",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=119127793), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aca691b408ef4e74b59a932be8d3b9b1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=535543630), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "731830e3cd8d428aa6f060324887e1d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=46408910), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3cfa31d9e32a4c9e8d2377df0341bb8f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=5333), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Downloading GoogleNews-vectors-negative300.bin.gz (1.5G) for you, it will take a while...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5cf8df791ebd43e1b10b9a746e43b731",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=1647046227), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from common.download_utils import download_week3_resources\n",
    "\n",
    "download_week3_resources()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BuUSN2397DFD"
   },
   "source": [
    "### Grading\n",
    "We will create a grader instace below and use it to collect your answers. Note that these outputs will be stored locally inside grader and will be uploaded to platform only after running submiting function in the last part of this assignment. If you want to make partial submission, you can run that cell any time you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s8aiK3mo7DFF"
   },
   "outputs": [],
   "source": [
    "from grader import Grader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s1VBo0zs7DFM"
   },
   "outputs": [],
   "source": [
    "grader = Grader()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tuGZylEP7DFT"
   },
   "source": [
    "## Word embedding\n",
    "\n",
    "To solve the problem, you will use two different models of embeddings:\n",
    "\n",
    " - [Pre-trained word vectors](https://code.google.com/archive/p/word2vec/) from Google which were trained on a part of Google News dataset (about 100 billion words). The model contains 300-dimensional vectors for 3 million words and phrases. `GoogleNews-vectors-negative300.bin.gz` will be downloaded in `download_week3_resources()`.\n",
    " - Representations using StarSpace on StackOverflow data sample. You will need to train them from scratch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6OGbwiAl7DFW"
   },
   "source": [
    "It's always easier to start with pre-trained embeddings. Unpack the pre-trained Goggle's vectors and upload them using the function [KeyedVectors.load_word2vec_format](https://radimrehurek.com/gensim/models/keyedvectors.html) from gensim library with the parameter *binary=True*. If the size of the embeddings is larger than the avaliable memory, you could load only a part of the embeddings by defining the parameter *limit* (recommended: 500000)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-RmzMDxo7DFY"
   },
   "outputs": [],
   "source": [
    "\n",
    "file='/content/GoogleNews-vectors-negative300.bin'\n",
    "import gzip\n",
    "f_gz = gzip.open(file+'.gz', 'rb')\n",
    "f_ugz = open(file,'wb')\n",
    "f_ugz.write(f_gz.read())\n",
    "f_ugz.close()\n",
    "f_gz.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IWMCLW-z7DE0"
   },
   "source": [
    "### Data\n",
    "\n",
    "The following cell will download all data required for this assignment into the folder `week3/data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YfmS6KIhDQbc"
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "from gensim.test.utils import datapath\n",
    "from gensim.models import KeyedVectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "SHhKUNwX8bIF",
    "outputId": "268fd896-e46b-4359-b38f-4125cbf6fcb3"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
     ]
    }
   ],
   "source": [
    "#wv_embeddings = ######### YOUR CODE HERE #############\n",
    "wv_embeddings = KeyedVectors.load_word2vec_format(datapath(file),binary=True,limit=500000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "TMSLX-bHIY1W",
    "outputId": "dd1bf91f-be37-47d1-aba7-71aff3dfbd0d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check if wv_embeddings contains 'word'\n",
    "'word' in wv_embeddings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "4KP7khPrJOKo",
    "outputId": "5d94f264-0142-496a-caae-d6188ecf2d40"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300,)"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wv_embeddings['word'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5n5dZm9Z7DFn"
   },
   "source": [
    "### How to work with Google's word2vec embeddings?\n",
    "\n",
    "Once you have loaded the representations, make sure you can access them. First, you can check if the loaded embeddings contain a word:\n",
    "    \n",
    "    'word' in wv_embeddings\n",
    "    \n",
    "Second, to get the corresponding embedding you can use the square brackets:\n",
    "\n",
    "    wv_embeddings['word']\n",
    " \n",
    "### Checking that the embeddings are correct \n",
    " \n",
    "To prevent any errors during the first stage, we can check that the loaded embeddings are correct. You can call the function *check_embeddings*, implemented below, which runs 3 tests:\n",
    "1. Find the most similar word for provided \"positive\" and \"negative\" words.\n",
    "2. Find which word from the given list doesn’t go with the others.\n",
    "3. Find the most similar word for the provided one.\n",
    "\n",
    "In the right case the function will return the string *These embeddings look good*. Othervise, you need to validate the previous steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rImn15sN7DFp"
   },
   "outputs": [],
   "source": [
    "def check_embeddings(embeddings):\n",
    "    error_text = \"Something wrong with your embeddings ('%s test isn't correct).\"\n",
    "    most_similar = embeddings.most_similar(positive=['woman', 'king'], negative=['man'])\n",
    "    if len(most_similar) < 1 or most_similar[0][0] != 'queen':\n",
    "        return error_text % \"Most similar\"\n",
    "\n",
    "    doesnt_match = embeddings.doesnt_match(['breakfast', 'cereal', 'dinner', 'lunch'])\n",
    "    if doesnt_match != 'cereal':\n",
    "        return error_text % \"Doesn't match\"\n",
    "    \n",
    "    most_similar_to_given = embeddings.most_similar_to_given('music', ['water', 'sound', 'backpack', 'mouse'])\n",
    "    if most_similar_to_given != 'sound':\n",
    "        return error_text % \"Most similar to given\"\n",
    "    \n",
    "    return \"These embeddings look good.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "id": "H6D0Daou7DFw",
    "outputId": "8abdca9e-0817-456e-f316-d920021511cb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These embeddings look good.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n",
      "/usr/local/lib/python3.6/dist-packages/gensim/models/keyedvectors.py:895: FutureWarning: arrays to stack must be passed as a \"sequence\" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.\n",
      "  vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)\n"
     ]
    }
   ],
   "source": [
    "print(check_embeddings(wv_embeddings))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p741iZG67DF4"
   },
   "source": [
    "## From word to text embeddings\n",
    "\n",
    "**Task 1 (Question2Vec).** Usually, we have word-based embeddings, but for the task we need to create a representation for the whole question. It could be done in different ways. In our case we will use a **mean** of all word vectors in the question. Now you need to implement the function *question_to_vec*, which calculates the question representation described above. This function should work with the input text as is without any preprocessing.\n",
    "\n",
    "Note that there could be words without the corresponding embeddings. In this case, you can just skip these words and don't take them into account during calculating the result. If the question doesn't contain any known word with embedding, the function should return a zero vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wCXhMMZj7DF7"
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jh01H9lQ7DGE"
   },
   "outputs": [],
   "source": [
    "def question_to_vec(question, embeddings, dim=300):\n",
    "    \"\"\"\n",
    "        question: a string\n",
    "        embeddings: dict where the key is a word and a value is its' embedding\n",
    "        dim: size of the representation\n",
    "\n",
    "        result: vector representation for the question\n",
    "    \"\"\"\n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    word_list=question.split()\n",
    "    \n",
    "    kword_list=[wv for wv in word_list if wv in embeddings]\n",
    "\n",
    "    output_vector=np.zeros(dim)\n",
    "    if kword_list:\n",
    "               output_vector= np.mean(np.vstack([embeddings[wv] for wv in kword_list]),axis=0)\n",
    "    return output_vector\n",
    "    ######################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LxrTunxR7DGL"
   },
   "source": [
    "To check the basic correctness of your implementation, run the function *question_to_vec_tests*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jvqiOHbp7DGN"
   },
   "outputs": [],
   "source": [
    "def question_to_vec_tests():\n",
    "    if (np.zeros(300) != question_to_vec('', wv_embeddings)).any():\n",
    "        return \"You need to return zero vector for empty question.\"\n",
    "    if (np.zeros(300) != question_to_vec('thereisnosuchword', wv_embeddings)).any():\n",
    "        return \"You need to return zero vector for the question, which consists only unknown words.\"\n",
    "    if (wv_embeddings['word'] != question_to_vec('word', wv_embeddings)).any():\n",
    "        return \"You need to check the corectness of your function.\"\n",
    "    if ((wv_embeddings['I'] + wv_embeddings['am']) / 2 != question_to_vec('I am', wv_embeddings)).any():\n",
    "        return \"Your function should calculate a mean of word vectors.\"\n",
    "    if (wv_embeddings['word'] != question_to_vec('thereisnosuchword word', wv_embeddings)).any():\n",
    "        return \"You should not consider words which embeddings are unknown.\"\n",
    "    return \"Basic tests are passed.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "M8GtjCdl7DGU",
    "outputId": "68edbde8-3ece-4227-c4cb-935992e4fa07"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Basic tests are passed.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print(question_to_vec_tests())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jp2FC6TE7DGe"
   },
   "source": [
    "You can submit embeddings for the questions from the file *test_embeddings.tsv* to earn the points. In this task you don't need to transform the text of a question somehow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "1cXKfq9K7DGh",
    "outputId": "32921497-9102-4084-8174-8148dc5bf3a6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "from util import array_to_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "93whEWZJ8HgL"
   },
   "outputs": [],
   "source": [
    "from grader import Grader\n",
    "grader=Grader()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "pxSbgNw17DGn",
    "outputId": "399d2321-5416-473d-f9fc-d9f0b435f71e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current answer for task Question2Vec is: 0.01929389126598835\n",
      "-0.02872721292078495\n",
      "0.0460561104118824\n",
      "0.0852593332529068\n",
      "0.0243055559694767\n",
      "-0...\n"
     ]
    }
   ],
   "source": [
    "question2vec_result = []\n",
    "for question in open('data/test_embeddings.tsv'):\n",
    "    question = question.strip()\n",
    "    answer = question_to_vec(question, wv_embeddings)\n",
    "    question2vec_result = np.append(question2vec_result, answer)\n",
    "\n",
    "grader.submit_tag('Question2Vec', array_to_string(question2vec_result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V0eKX2ey7DGu"
   },
   "source": [
    "Now we have a method to create a representation of any sentence and we are ready for the first evaluation. So, let's check how well our solution (Google's vectors + *question_to_vec*) will work.\n",
    "\n",
    "## Evaluation of text similarity\n",
    "\n",
    "We can imagine that if we use good embeddings, the cosine similarity between the duplicate sentences should be less than for the random ones. Overall, for each pair of duplicate sentences we can generate *R* random negative examples and find out the position of the correct duplicate.  \n",
    "\n",
    "For example, we have the question *\"Exceptions What really happens\"* and we are sure that another question *\"How does the catch keyword determine the type of exception that was thrown\"* is a duplicate. But our model doesn't know it and tries to find out the best option also among questions like *\"How Can I Make These Links Rotate in PHP\"*, *\"NSLog array description not memory address\"* and *\"PECL_HTTP not recognised php ubuntu\"*. The goal of the model is to rank all these 4 questions (1 *positive* and *R* = 3 *negative*) in the way that the correct one is in the first place.\n",
    "\n",
    "However, it is unnatural to count on that the best candidate will be always in the first place. So let us consider the place of the best candidate in the sorted list of candidates and formulate a metric based on it. We can fix some *K* — a reasonalble number of top-ranked elements and *N* — a number of queries (size of the sample).\n",
    "\n",
    "### Hits@K\n",
    "\n",
    "The first simple metric will be a number of correct hits for some *K*:\n",
    "$$ \\text{Hits@K} = \\frac{1}{N}\\sum_{i=1}^N \\, [dup_i \\in topK(q_i)]$$\n",
    "\n",
    "where $q_i$ is the i-th query, $dup_i$ is its duplicate, $topK(q_i)$ is the top K elements of the ranked sentences provided by our model and the operation $[dup_i \\in topK(q_i)]$ equals 1 if the condition is true and 0 otherwise (more details about this operation could be found [here](https://en.wikipedia.org/wiki/Iverson_bracket)).\n",
    "\n",
    "\n",
    "### DCG@K\n",
    "The second one is a simplified [DCG metric](https://en.wikipedia.org/wiki/Discounted_cumulative_gain):\n",
    "\n",
    "$$ \\text{DCG@K} = \\frac{1}{N} \\sum_{i=1}^N\\frac{1}{\\log_2(1+rank_{dup_i})}\\cdot[rank_{dup_i} \\le K] $$\n",
    "\n",
    "where $rank_{dup_i}$ is a position of the duplicate in the sorted list of the nearest sentences for the query $q_i$. According to this metric, the model gets a higher reward for a higher position of the correct answer. If the answer does not appear in topK at all, the reward is zero. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rCe2sK-d7DGw"
   },
   "source": [
    "### Evaluation examples\n",
    "\n",
    "Let's calculate the described metrics for the toy example introduced above. In this case $N$ = 1 and the correct candidate for $q_1$ is *\"How does the catch keyword determine the type of exception that was thrown\"*. Consider the following ranking of the candidates:\n",
    "1. *\"How Can I Make These Links Rotate in PHP\"*\n",
    "2. *\"How does the catch keyword determine the type of exception that was thrown\"*\n",
    "3. *\"NSLog array description not memory address\"*\n",
    "4. *\"PECL_HTTP not recognised php ubuntu\"*\n",
    "\n",
    "Using the ranking above, calculate *Hits@K* metric for *K = 1, 2, 4*: \n",
    " \n",
    "- [K = 1] $\\text{Hits@1} = \\frac{1}{1}\\sum_{i=1}^1 \\, [dup_i \\in top1(q_i)] = [dup_1 \\in top1(q_1)] = 0$ because the correct answer doesn't appear in the *top1* list.\n",
    "- [K = 2] $\\text{Hits@2} = \\frac{1}{1}\\sum_{i=1}^1 \\, [dup_i \\in top2(q_i)] = [dup_1 \\in top2(q_1)] = 1$ because $rank_{dup_1} = 2$.\n",
    "- [K = 4] $\\text{Hits@4} = \\frac{1}{1}\\sum_{i=1}^1 \\, [dup_i \\in top4(q_i)] = [dup_1 \\in top4(q_1)] = 1$\n",
    "\n",
    "Using the ranking above, calculate *DCG@K* metric for *K = 1, 2, 4*:\n",
    "\n",
    "- [K = 1] $\\text{DCG@1} = \\frac{1}{1} \\sum_{i=1}^1\\frac{1}{\\log_2(1+rank_{dup_i})}\\cdot[rank_{dup_i} \\le 1] = \\frac{1}{\\log_2(1+rank_{dup_i})}\\cdot[rank_{dup_i} \\le 1] = 0$ because the correct answer doesn't appear in the top1 list.\n",
    "- [K = 2] $\\text{DCG@2} = \\frac{1}{1} \\sum_{i=1}^1\\frac{1}{\\log_2(1+rank_{dup_i})}\\cdot[rank_{dup_i} \\le 2] = \\frac{1}{\\log_2{3}}$, because $rank_{dup_1} = 2$.\n",
    "- [K = 4] $\\text{DCG@4} = \\frac{1}{1} \\sum_{i=1}^1\\frac{1}{\\log_2(1+rank_{dup_i})}\\cdot[rank_{dup_i} \\le 4] = \\frac{1}{\\log_2{3}}$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x8d8JXc_7DGz"
   },
   "source": [
    "**Tasks 2 and 3 (HitsCount and DCGScore).** Implement the functions *hits_count* and *dcg_score* as described above. Each function has two arguments: *dup_ranks* and *k*. *dup_ranks* is a list which contains *values of ranks* of duplicates. For example, *dup_ranks* is *[2]* for the example provided above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WXLYoKZd7DG1"
   },
   "outputs": [],
   "source": [
    "def hits_count(dup_ranks, k):\n",
    "    \"\"\"\n",
    "        dup_ranks: list of duplicates' ranks; one rank per question; \n",
    "                   length is a number of questions which we are looking for duplicates; \n",
    "                   rank is a number from 1 to len(candidates of the question); \n",
    "                   e.g. [2, 3] means that the first duplicate has the rank 2, the second one — 3.\n",
    "        k: number of top-ranked elements (k in Hits@k metric)\n",
    "\n",
    "        result: return Hits@k value for current ranking\n",
    "    \"\"\"\n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    Hits_k=0\n",
    "    N_answ=len(dup_ranks)\n",
    "    for ii in range(N_answ):\n",
    "       Hits_k+= (1./N_answ)*(dup_ranks[ii]<=k)\n",
    "    return Hits_k\n",
    "    ######################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YtpQqIXO7DG-"
   },
   "source": [
    "Test your code on the tiny examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0P5cd2N17DHB"
   },
   "outputs": [],
   "source": [
    "def test_hits():\n",
    "    # *Evaluation example*\n",
    "    # answers — dup_i\n",
    "    answers = [\"How does the catch keyword determine the type of exception that was thrown\"]\n",
    "    \n",
    "    # candidates_ranking — the ranked sentences provided by our model\n",
    "    candidates_ranking = [[\"How Can I Make These Links Rotate in PHP\", \n",
    "                           \"How does the catch keyword determine the type of exception that was thrown\",\n",
    "                           \"NSLog array description not memory address\",\n",
    "                           \"PECL_HTTP not recognised php ubuntu\"]]\n",
    "    # dup_ranks — position of the dup_i in the list of ranks +1\n",
    "    dup_ranks = [candidates_ranking[i].index(answers[i]) + 1 for i in range(len(answers))]\n",
    "    \n",
    "    # correct_answers — the expected values of the result for each k from 1 to 4\n",
    "    correct_answers = [0, 1, 1, 1]\n",
    "    for k, correct in enumerate(correct_answers, 1):\n",
    "        if not np.isclose(hits_count(dup_ranks, k), correct):\n",
    "            return \"Check the function.\"\n",
    "    \n",
    "    # Other tests\n",
    "    answers = [\"How does the catch keyword determine the type of exception that was thrown\", \n",
    "               \"Convert Google results object (pure js) to Python object\"]\n",
    "    \n",
    "    # The first test: both duplicates on the first position in ranked list\n",
    "    candidates_ranking = [[\"How does the catch keyword determine the type of exception that was thrown\",\n",
    "                           \"How Can I Make These Links Rotate in PHP\"], \n",
    "                          [\"Convert Google results object (pure js) to Python object\",\n",
    "                           \"WPF- How to update the changes in list item of a list\"]]\n",
    "    dup_ranks = [candidates_ranking[i].index(answers[i]) + 1 for i in range(len(answers))]\n",
    "    correct_answers = [1, 1]\n",
    "    for k, correct in enumerate(correct_answers, 1):\n",
    "        if not np.isclose(hits_count(dup_ranks, k), correct):\n",
    "            return \"Check the function (test: both duplicates on the first position in ranked list).\"\n",
    "        \n",
    "    # The second test: one candidate on the first position, another — on the second\n",
    "    candidates_ranking = [[\"How Can I Make These Links Rotate in PHP\", \n",
    "                           \"How does the catch keyword determine the type of exception that was thrown\"], \n",
    "                          [\"Convert Google results object (pure js) to Python object\",\n",
    "                           \"WPF- How to update the changes in list item of a list\"]]\n",
    "    dup_ranks = [candidates_ranking[i].index(answers[i]) + 1 for i in range(len(answers))]\n",
    "    correct_answers = [0.5, 1]\n",
    "    for k, correct in enumerate(correct_answers, 1):\n",
    "        if not np.isclose(hits_count(dup_ranks, k), correct):\n",
    "            return \"Check the function (test: one candidate on the first position, another — on the second).\"\n",
    "\n",
    "    # The third test: both candidates on the second position\n",
    "    candidates_ranking = [[\"How Can I Make These Links Rotate in PHP\", \n",
    "                           \"How does the catch keyword determine the type of exception that was thrown\"], \n",
    "                          [\"WPF- How to update the changes in list item of a list\",\n",
    "                           \"Convert Google results object (pure js) to Python object\"]]\n",
    "    dup_ranks = [candidates_ranking[i].index(answers[i]) + 1 for i in range(len(answers))]\n",
    "    correct_answers = [0, 1]\n",
    "    for k, correct in enumerate(correct_answers, 1):\n",
    "        if not np.isclose(hits_count(dup_ranks, k), correct):\n",
    "            return \"Check the function (test: both candidates on the second position).\"\n",
    "\n",
    "    return \"Basic test are passed.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "fdqqIbgD7DHK",
    "outputId": "e5418ea2-30b8-4842-b347-a216d53ec3ea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Basic test are passed.\n"
     ]
    }
   ],
   "source": [
    "print(test_hits())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nyTXcm-T7DHY"
   },
   "outputs": [],
   "source": [
    "def dcg_score(dup_ranks, k):\n",
    "    \"\"\"\n",
    "        dup_ranks: list of duplicates' ranks; one rank per question; \n",
    "                   length is a number of questions which we are looking for duplicates; \n",
    "                   rank is a number from 1 to len(candidates of the question); \n",
    "                   e.g. [2, 3] means that the first duplicate has the rank 2, the second one — 3.\n",
    "        k: number of top-ranked elements (k in DCG@k metric)\n",
    "\n",
    "        result: return DCG@k value for current ranking\n",
    "    \"\"\"\n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    DCG_k=0\n",
    "    N_answ=len(dup_ranks)\n",
    "    for ii in range(N_answ):\n",
    "       DCG_k+= (1./N_answ)*(1./np.log2(1+dup_ranks[ii]))*(dup_ranks[ii]<=k)\n",
    "    return DCG_k\n",
    "    ######################################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Q3Hs3J0Q7DHh"
   },
   "outputs": [],
   "source": [
    "def test_dcg():\n",
    "    # *Evaluation example*\n",
    "    # answers — dup_i\n",
    "    answers = [\"How does the catch keyword determine the type of exception that was thrown\"]\n",
    "    \n",
    "    # candidates_ranking — the ranked sentences provided by our model\n",
    "    candidates_ranking = [[\"How Can I Make These Links Rotate in PHP\", \n",
    "                           \"How does the catch keyword determine the type of exception that was thrown\",\n",
    "                           \"NSLog array description not memory address\",\n",
    "                           \"PECL_HTTP not recognised php ubuntu\"]]\n",
    "    # dup_ranks — position of the dup_i in the list of ranks +1\n",
    "    dup_ranks = [candidates_ranking[i].index(answers[i]) + 1 for i in range(len(answers))]\n",
    "    \n",
    "    # correct_answers — the expected values of the result for each k from 1 to 4\n",
    "    correct_answers = [0, 1 / (np.log2(3)), 1 / (np.log2(3)), 1 / (np.log2(3))]\n",
    "    for k, correct in enumerate(correct_answers, 1):\n",
    "        if not np.isclose(dcg_score(dup_ranks, k), correct):\n",
    "            return \"Check the function.\"\n",
    "    \n",
    "    # Other tests\n",
    "    answers = [\"How does the catch keyword determine the type of exception that was thrown\", \n",
    "               \"Convert Google results object (pure js) to Python object\"]\n",
    "\n",
    "    # The first test: both duplicates on the first position in ranked list\n",
    "    candidates_ranking = [[\"How does the catch keyword determine the type of exception that was thrown\",\n",
    "                           \"How Can I Make These Links Rotate in PHP\"], \n",
    "                          [\"Convert Google results object (pure js) to Python object\",\n",
    "                           \"WPF- How to update the changes in list item of a list\"]]\n",
    "    dup_ranks = [candidates_ranking[i].index(answers[i]) + 1 for i in range(len(answers))]\n",
    "    correct_answers = [1, 1]\n",
    "    for k, correct in enumerate(correct_answers, 1):\n",
    "        if not np.isclose(dcg_score(dup_ranks, k), correct):\n",
    "            return \"Check the function (test: both duplicates on the first position in ranked list).\"\n",
    "        \n",
    "    # The second test: one candidate on the first position, another — on the second\n",
    "    candidates_ranking = [[\"How Can I Make These Links Rotate in PHP\", \n",
    "                           \"How does the catch keyword determine the type of exception that was thrown\"], \n",
    "                          [\"Convert Google results object (pure js) to Python object\",\n",
    "                           \"WPF- How to update the changes in list item of a list\"]]\n",
    "    dup_ranks = [candidates_ranking[i].index(answers[i]) + 1 for i in range(len(answers))]\n",
    "    correct_answers = [0.5, (1 + (1 / (np.log2(3)))) / 2]\n",
    "    for k, correct in enumerate(correct_answers, 1):\n",
    "        if not np.isclose(dcg_score(dup_ranks, k), correct):\n",
    "            return \"Check the function (test: one candidate on the first position, another — on the second).\"\n",
    "        \n",
    "    # The third test: both candidates on the second position\n",
    "    candidates_ranking = [[\"How Can I Make These Links Rotate in PHP\",\n",
    "                           \"How does the catch keyword determine the type of exception that was thrown\"], \n",
    "                          [\"WPF- How to update the changes in list item of a list\",\n",
    "                           \"Convert Google results object (pure js) to Python object\"]]\n",
    "    dup_ranks = [candidates_ranking[i].index(answers[i]) + 1 for i in range(len(answers))]\n",
    "    correct_answers = [0, 1 / (np.log2(3))]\n",
    "    for k, correct in enumerate(correct_answers, 1):\n",
    "        if not np.isclose(dcg_score(dup_ranks, k), correct):\n",
    "            return \"Check the function (test: both candidates on the second position).\"\n",
    "\n",
    "    return \"Basic test are passed.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "CCoeAVet7DHo",
    "outputId": "66d410e4-6674-44cd-cde8-7858aad4226c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Basic test are passed.\n"
     ]
    }
   ],
   "source": [
    "print(test_dcg())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bxwfzXC17DHw"
   },
   "source": [
    "Submit results of the functions *hits_count* and *dcg_score* for the following examples to earn the points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QJuwvZO27DHy"
   },
   "outputs": [],
   "source": [
    "test_examples = [\n",
    "    [1],\n",
    "    [1, 2],\n",
    "    [2, 1],\n",
    "    [1, 2, 3],\n",
    "    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],\n",
    "    [9, 5, 4, 2, 8, 10, 7, 6, 1, 3],\n",
    "    [4, 3, 5, 1, 9, 10, 7, 8, 2, 6],\n",
    "    [5, 1, 7, 6, 2, 3, 8, 9, 10, 4],\n",
    "    [6, 3, 1, 4, 7, 2, 9, 8, 10, 5],\n",
    "    [10, 9, 8, 7, 6, 5, 4, 3, 2, 1],\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "id": "a5rYL0x17DH6",
    "outputId": "0138af22-1c3d-4021-ab66-d12f39a7030e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current answer for task HitsCount is: 1.0\n",
      "0.5\n",
      "1.0\n",
      "0.5\n",
      "1.0\n",
      "0.3333333333333333\n",
      "0.6666666666666666\n",
      "1.0\n",
      "0.1\n",
      "0.2\n",
      "0.30000000000000004\n",
      "0.4\n",
      "0.5\n",
      "0....\n"
     ]
    }
   ],
   "source": [
    "hits_results = []\n",
    "for example in test_examples:\n",
    "    for k in range(len(example)):\n",
    "        hits_results.append(hits_count(example, k + 1))\n",
    "grader.submit_tag('HitsCount', array_to_string(hits_results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "mgRwqgmQ7DIC",
    "outputId": "b2eb9f01-6747-4c37-df5a-f850ed98b550"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current answer for task DCGScore is: 1.0\n",
      "0.5\n",
      "0.8154648767857288\n",
      "0.5\n",
      "0.8154648767857288\n",
      "0.3333333333333333\n",
      "0.5436432511904858\n",
      "0.7103099178...\n"
     ]
    }
   ],
   "source": [
    "dcg_results = []\n",
    "for example in test_examples:\n",
    "    for k in range(len(example)):\n",
    "        dcg_results.append(dcg_score(example, k + 1))\n",
    "grader.submit_tag('DCGScore', array_to_string(dcg_results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "85uepJzK7DIL"
   },
   "source": [
    "##  First solution: pre-trained embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Tltk0YQl7DIN"
   },
   "source": [
    "We will work with predefined train, validation and test corpora. All the files are tab-separated, but have a different format:\n",
    " - *train* corpus contains similar sentences at the same row.\n",
    " - *validation* corpus contains the following columns: *question*, *similar question*, *negative example 1*, *negative example 2*, ... \n",
    " - *test* corpus contains the following columns: *question*, *example 1*, *example 2*, ...\n",
    "\n",
    "Validation corpus will be used for the intermediate validation of models. The test data will be necessary for submitting the quality of your model in the system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VK5-Sesc7DIP"
   },
   "source": [
    "Now you should read *validation* corpus, located at `data/validation.tsv`. You will use it later to evaluate current solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6i3cQmad7DIS"
   },
   "outputs": [],
   "source": [
    "def read_corpus(filename):\n",
    "    data = []\n",
    "    for line in open(filename, encoding='utf-8'):\n",
    "        data.append(line.strip().split('\\t'))\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PWhDx30I7DIk"
   },
   "outputs": [],
   "source": [
    "#validation = ######### YOUR CODE HERE #############\n",
    "validation=read_corpus('data/validation.tsv')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WsR5-kVu7DIt"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bv0aO1XJ7DI3"
   },
   "source": [
    "We will use cosine distance to rank candidate questions which you need to implement in the function *rank_candidates*. The function should return a sorted list of pairs *(initial position in candidates list, candidate)*. Index of some pair corresponds to its rank (the first is the best). For example, if the list of candidates was *[a, b, c]* and the most similar is *c*, then *a* and *b*, the function should return a list *[(2, c), (0, a), (1, b)]*.\n",
    "\n",
    "Pay attention, if you use the function *cosine_similarity* from *sklearn.metrics.pairwise* to calculate similarity because it works in a different way: most similar objects has greatest similarity. It's preferable to use a vectorized version of *cosine_similarity* function. Try to compute similarity at once and not use list comprehension. It should speed up your computations significantly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xa-OJ9Ie7DI6"
   },
   "outputs": [],
   "source": [
    "def rank_candidates(question, candidates, embeddings, dim=300):\n",
    "    \"\"\"\n",
    "        question: a string\n",
    "        candidates: a list of strings (candidates) which we want to rank\n",
    "        embeddings: some embeddings\n",
    "        dim: dimension of the current embeddings\n",
    "        \n",
    "        result: a list of pairs (initial position in the list, question)\n",
    "    \"\"\"\n",
    "    \n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    q2vec=question_to_vec(question,embeddings,dim)\n",
    "    q2vec=np.array([q2vec])\n",
    "    NN=len(candidates)\n",
    "    c2vec=[]\n",
    "    for q_candidates in candidates:\n",
    "        q_candidates = q_candidates.strip()\n",
    "        answer = question_to_vec(q_candidates, embeddings,dim)\n",
    "        c2vec= np.append(c2vec, answer)\n",
    "    c2vec=np.asarray(c2vec).reshape(len(candidates),dim)\n",
    "    cos_sim=cosine_similarity(q2vec,c2vec)\n",
    "    ranks_list=sorted([(k,v) for k,v in zip(cos_sim[0],candidates)],reverse=True)\n",
    "    item=[ranks_list[ii][1] for ii in range(NN)]  \n",
    "    #pp=[(loc,item[i]) for i in range(NN) for loc in [candidates.index(item[i])] for candidates[loc] in [\"None\"]]\n",
    "    return [(loc,item[i]) for i in range(NN) for loc in [candidates.index(item[i])] for candidates[loc] in [\"None\"]]\n",
    "    #[(candidates.index(ranks_list[i][1]),ranks_list[i][1]) for i in range(len(candidates))] \n",
    "    ######################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8-9SWBO97DJC"
   },
   "source": [
    "Test your code on the tiny examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sQM-jUf_7DJE"
   },
   "outputs": [],
   "source": [
    "def test_rank_candidates():\n",
    "    questions = ['converting string to list', 'Sending array via Ajax fails']\n",
    "    candidates = [['Convert Google results object (pure js) to Python object', \n",
    "                   'C# create cookie from string and send it',\n",
    "                   'How to use jQuery AJAX for an outside domain?'], \n",
    "                  ['Getting all list items of an unordered list in PHP', \n",
    "                   'WPF- How to update the changes in list item of a list', \n",
    "                   'select2 not displaying search results']]\n",
    "    results = [[(1, 'C# create cookie from string and send it'), \n",
    "                (0, 'Convert Google results object (pure js) to Python object'), \n",
    "                (2, 'How to use jQuery AJAX for an outside domain?')],\n",
    "               [(0, 'Getting all list items of an unordered list in PHP'), \n",
    "                (2, 'select2 not displaying search results'), \n",
    "                (1, 'WPF- How to update the changes in list item of a list')]]\n",
    "    for question, q_candidates, result in zip(questions, candidates, results):\n",
    "        ranks = rank_candidates(question, q_candidates, wv_embeddings, 300)\n",
    "        if not np.all(ranks == result):\n",
    "            return \"Check the function.\"\n",
    "    return \"Basic tests are passed.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "7_5B2MsE7DJK",
    "outputId": "2337b1fe-e634-42bd-a674-3d85c99bdeb8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Basic tests are passed.\n"
     ]
    }
   ],
   "source": [
    "print(test_rank_candidates())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cp6F0c8t7DJQ"
   },
   "source": [
    "Now we can test the quality of the current approach. Run the next two cells to get the results. Pay attention that calculation of similarity between vectors takes time and this calculation is computed approximately in 10 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qgWdXFFvT30W"
   },
   "outputs": [],
   "source": [
    "wv_ranking = []\n",
    "for line in validation:\n",
    "    q, *ex = line\n",
    "    ranks = rank_candidates(q, ex, wv_embeddings)\n",
    "    wv_ranking.append([r[0] for r in ranks].index(0) + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "v1vob25w7DJb",
    "outputId": "d045f03d-ac30-4b18-95e7-8889b104f440"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DCG@   1: 0.208 | Hits@   1: 0.208\n",
      "DCG@   5: 0.263 | Hits@   5: 0.311\n",
      "DCG@  10: 0.279 | Hits@  10: 0.360\n",
      "DCG@ 100: 0.316 | Hits@ 100: 0.549\n",
      "DCG@ 500: 0.349 | Hits@ 500: 0.810\n",
      "DCG@1000: 0.369 | Hits@1000: 1.000\n"
     ]
    }
   ],
   "source": [
    "for k in [1, 5, 10, 100, 500, 1000]:\n",
    "    print(\"DCG@%4d: %.3f | Hits@%4d: %.3f\" % (k, dcg_score(wv_ranking, k), k, hits_count(wv_ranking, k)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q1IpaCM57DJi"
   },
   "source": [
    "If you did all the steps correctly, you should be frustrated by the received results. Let's try to understand why the quality is so low. First of all, when you work with some data it is necessary to have an idea how the data looks like. Print several questions from the data:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UvMAIH6K7DJp"
   },
   "source": [
    "As you can see, we deal with the raw data. It means that we have many punctuation marks, special characters and unlowercased letters. In our case, it could lead to the situation where we can't find some embeddings, e.g. for the word \"grid?\". \n",
    "\n",
    "To solve this problem you should use the functions *text_prepare* from the previous assignments to prepare the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XKcm5tKM7DJr"
   },
   "outputs": [],
   "source": [
    "from util import text_prepare\n",
    "import re\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RMp2PASEgh5t"
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "REPLACE_BY_SPACE_RE = re.compile('[/(){}\\[\\]\\|@,;]')\n",
    "BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')\n",
    "STOPWORDS = set(stopwords.words('english'))\n",
    "\n",
    "def text_prepare(text):\n",
    "    \"\"\"\n",
    "        text: a string\n",
    "        \n",
    "        return: modified initial string\n",
    "    \"\"\"\n",
    "    text = text.lower()\n",
    "    text = REPLACE_BY_SPACE_RE.sub('  ',text)\n",
    "    text = BAD_SYMBOLS_RE.sub('',text)\n",
    "    text =' '.join ([word for word in text.split() if word not in collections.Counter(STOPWORDS)])\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "noTIzJBj7DJx"
   },
   "source": [
    "Now transform all the questions from the validation set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aLgQvVcZ7DJz"
   },
   "outputs": [],
   "source": [
    "######### YOUR CODE HERE #############\n",
    "prepared_validation = []\n",
    "file=\"/content/data/prepared_validation_data.tsv\"\n",
    "out =open(file,'w')\n",
    "for line in validation:\n",
    "      val_pre=[text_prepare(sentences.strip()) for sentences in line]\n",
    "      print(*val_pre, sep='\\t', file=out)\n",
    "      prepared_validation.append(val_pre)\n",
    "out.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "k1M5ib_T7DJ7"
   },
   "source": [
    "Let's evaluate the approach again after the preparation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yR_y1Nd_7DJ9"
   },
   "outputs": [],
   "source": [
    "wv_prepared_ranking = []\n",
    "for line in prepared_validation:\n",
    "    q, *ex = line\n",
    "    ranks = rank_candidates(q, ex, wv_embeddings)\n",
    "    wv_prepared_ranking.append([r[0] for r in ranks].index(0) + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "ZVGSaWpy7DKH",
    "outputId": "335864d2-431b-4455-ddec-c0b275598cc5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DCG@   1: 0.304 | Hits@   1: 0.304\n",
      "DCG@   5: 0.375 | Hits@   5: 0.438\n",
      "DCG@  10: 0.391 | Hits@  10: 0.489\n",
      "DCG@ 100: 0.425 | Hits@ 100: 0.657\n",
      "DCG@ 500: 0.448 | Hits@ 500: 0.833\n",
      "DCG@1000: 0.465 | Hits@1000: 1.000\n"
     ]
    }
   ],
   "source": [
    "for k in [1, 5, 10, 100, 500, 1000]:\n",
    "    print(\"DCG@%4d: %.3f | Hits@%4d: %.3f\" % (k, dcg_score(wv_prepared_ranking, k), \n",
    "                                              k, hits_count(wv_prepared_ranking, k)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YDDibqpQ7DKT"
   },
   "source": [
    "Now, prepare also train and test data, because you will need it in the future:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZEhytS647DKV"
   },
   "outputs": [],
   "source": [
    "def prepare_file(in_, out_):\n",
    "    out = open(out_, 'w')\n",
    "    for line in open(in_, encoding='utf8'):\n",
    "        line = line.strip().split('\\t')\n",
    "        new_line = [text_prepare(q) for q in line]\n",
    "        print(*new_line, sep='\\t', file=out)\n",
    "    out.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "IqfDtnA7JXI6",
    "outputId": "aa8e64fc-82e9-42ef-c9fb-4da83ad422b1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['train.tsv',\n",
       " 'test_embeddings.tsv',\n",
       " 'test.tsv',\n",
       " 'prepared_validation_data.tsv',\n",
       " 'validation.tsv']"
      ]
     },
     "execution_count": 42,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.listdir('data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JdcLKkkh7DKa"
   },
   "outputs": [],
   "source": [
    "######################################\n",
    "######### YOUR CODE HERE #############\n",
    "prepare_file('data/train.tsv','data/prepared_train_data.tsv')\n",
    "######################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4FHuHzW27DKf"
   },
   "source": [
    "**Task 4 (W2VTokenizedRanks).** For each question from prepared *test.tsv* submit the ranks of the candidates to earn the points. The calculations should take about 3-5 minutes. Pay attention that the function *rank_candidates* returns a ranking, while in this case you should find a position in this ranking. Ranks should start with 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mdTvcRNx7DKg"
   },
   "outputs": [],
   "source": [
    "from util import matrix_to_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7QhFFiBpOeIe"
   },
   "outputs": [],
   "source": [
    "#prepared_test_data = ######### YOUR CODE HERE #############\n",
    "prepare_file('data/test.tsv','data/prepared_test_data.tsv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "nv1KS6HF7DKm",
    "outputId": "24df019f-44b3-47c8-f6c0-96d83e0b78a7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current answer for task W2VTokenizedRanks is: 95\t94\t7\t9\t64\t36\t31\t93\t23\t100\t99\t20\t60\t6\t97\t48\t70\t37\t41\t96\t29\t56\t2\t65\t68\t44\t27\t25\t57\t62\t11\t87\t50\t66\t7...\n"
     ]
    }
   ],
   "source": [
    "w2v_ranks_results = []\n",
    "for line in open('data/prepared_test_data.tsv'):\n",
    "    q, *ex = line.strip().split('\\t')\n",
    "    ranks = rank_candidates(q, ex, wv_embeddings, 300)\n",
    "    ranked_candidates = [r[0] for r in ranks]\n",
    "    w2v_ranks_results.append([ranked_candidates.index(i) + 1 for i in range(len(ranked_candidates))])\n",
    "    \n",
    "grader.submit_tag('W2VTokenizedRanks', matrix_to_string(w2v_ranks_results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "id": "GN0XsgMFEzKh",
    "outputId": "2d131744-5cb1-4d7a-837f-71658da20c1b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'Starspace'...\n",
      "remote: Enumerating objects: 5, done.\u001b[K\n",
      "remote: Counting objects:  20% (1/5)\u001b[K\r",
      "remote: Counting objects:  40% (2/5)\u001b[K\r",
      "remote: Counting objects:  60% (3/5)\u001b[K\r",
      "remote: Counting objects:  80% (4/5)\u001b[K\r",
      "remote: Counting objects: 100% (5/5)\u001b[K\r",
      "remote: Counting objects: 100% (5/5), done.\u001b[K\n",
      "remote: Compressing objects: 100% (5/5), done.\u001b[K\n",
      "remote: Total 873 (delta 0), reused 0 (delta 0), pack-reused 868\n",
      "Receiving objects: 100% (873/873), 3.05 MiB | 2.39 MiB/s, done.\n",
      "Resolving deltas: 100% (567/567), done.\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!git clone https://github.com/facebookresearch/Starspace.git\n",
    "sys.path.append('/content/Starspace')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "colab_type": "code",
    "id": "1L3DRuwyjyae",
    "outputId": "c039f8a3-bf1c-4cdf-b84e-b92d8123c443"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -g -c src/utils/normalize.cpp\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/dict.cpp\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -g -c src/utils/args.cpp\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/proj.cpp\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/parser.cpp -o parser.o\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/data.cpp -o data.o\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/model.cpp\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/starspace.cpp\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/doc_parser.cpp -o doc_parser.o\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/doc_data.cpp -o doc_data.o\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops -I/usr/local/bin/boost_1_63_0/ -g -c src/utils/utils.cpp -o utils.o\n",
      "g++ -pthread -std=gnu++11 -O3 -funroll-loops normalize.o dict.o args.o proj.o parser.o data.o model.o starspace.o doc_parser.o doc_data.o utils.o -I/usr/local/bin/boost_1_63_0/ -g src/main.cpp -o starspace\n"
     ]
    }
   ],
   "source": [
    "\n",
    "os.path.dirname('/content/Starspace')\n",
    "os.chdir('/content/Starspace')\n",
    "!make"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Nh_3ulkA1q9t"
   },
   "outputs": [],
   "source": [
    "! sudo link /content/Starspace/./starspace /usr/local/bin/starspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "INRS1pcK7DKs"
   },
   "source": [
    "## Advanced solution: StarSpace embeddings\n",
    "\n",
    "Now you are ready to train your own word embeddings! In particular, you need to train embeddings specially for our task of duplicates detection. Unfortunately, StarSpace cannot be run on Windows and we recommend to use provided\n",
    "[docker container](https://github.com/hse-aml/natural-language-processing/blob/master/Docker-tutorial.md) or other alternatives. Don't delete results of this task because you will need it in the final project.\n",
    "\n",
    "### How it works and what's the main difference with word2vec?\n",
    "The main point in this section is that StarSpace can be trained specifically for some tasks. In contrast to word2vec model, which tries to train similar embeddings for words in similar contexts, StarSpace uses embeddings for the whole sentence (just as a sum of embeddings of words and phrases). Despite the fact that in both cases we get word embeddings as a result of the training, StarSpace embeddings are trained using some supervised data, e.g. a set of similar sentence pairs, and thus they can better suit the task.\n",
    "\n",
    "In our case, StarSpace should use two types of sentence pairs for training: \"positive\" and \"negative\". \"Positive\" examples are extracted from the train sample (duplicates, high similarity) and the \"negative\" examples are generated randomly (low similarity assumed). \n",
    "\n",
    "### How to choose the best params for the model?\n",
    "Normally, you would start with some default choice and then run extensive experiments to compare different strategies. However, we have some recommendations ready for you to save your time:\n",
    "- Be careful with choosing the suitable training mode. In this task we want to explore texts similarity which corresponds to *trainMode = 3*.\n",
    "- Use adagrad optimization (parameter *adagrad = true*).\n",
    "- Set the length of phrase equal to 1 (parameter *ngrams*), because we need embeddings only for words.\n",
    "- Don't use a large number of *epochs* (we think that 5 should be enough).\n",
    "- Try dimension *dim* equal to 100.\n",
    "- To compare embeddings usually *cosine* *similarity* is used.\n",
    "- Set *minCount* greater than 1 (for example, 2) if you don't want to get embeddings for extremely rare words.\n",
    "- Parameter *verbose = true* could show you the progress of the training process.\n",
    "- Set parameter *fileFormat* equals *labelDoc*.\n",
    "- Parameter *negSearchLimit* is responsible for a number of negative examples which is used during the training. We think that 10 will be enought for this task.\n",
    "- To increase a speed of training we recommend to set *learning rate* to 0.05."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3Cxr4AFV7DKt"
   },
   "source": [
    "Train StarSpace embeddings for unigrams on the train dataset. You don't need to change the format of the input data. Just don't forget to use prepared version of the training data. \n",
    "\n",
    "If you follow the instruction, the training process will take about 1 hour. The size of the embeddings' dictionary should be approximately 100 000 (number of lines in the result file). If you got significantly more than this number, try to check all the instructions above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-E0bqXyo7JmC"
   },
   "outputs": [],
   "source": [
    "#!starspace train -trainFile /content/data/prepared_train_data.tsv -model /content/startspace_stackoverflow_model -trainMode 3 -lr 0.01 -dim 100 -epoch 5 -negSearchLimit 10 -minCount 2  -adagrad 1 -fileFormat 'labelDoc' -validationFile /content/data/prepared_validation_data.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pexB9hln7DKv"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SecR4CnypWEa"
   },
   "outputs": [],
   "source": [
    "#!starspace train -trainFile /content/data/prepared_train_data.tsv -model /content/startspace_stackoverflow_model_01 -trainMode 3 -lr 0.01 -dim 100 -epoch 5 -negSearchLimit 10 -minCount 2 -batchSize 10 -adagrad 1 -fileFormat 'labelDoc' -validationFile /content/data/prepared_validation_data.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-SQ9LIDXsE89"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AsVbzGWwsGRM"
   },
   "outputs": [],
   "source": [
    "#!starspace train -trainFile /content/data/prepared_train_data.tsv -model /content/startspace_stackoverflow_model_02 -trainMode 3 -lr 0.05 -dim 100 -epoch 10 -negSearchLimit 10 -minCount 2  -adagrad 1 -fileFormat 'labelDoc' -validationFile /content/data/prepared_validation_data.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 935
    },
    "colab_type": "code",
    "id": "bN2D3CD4v8MS",
    "outputId": "2e3dfaa8-fffe-4d84-ba9e-90220334c057"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Arguments: \n",
      "lr: 0.05\n",
      "dim: 100\n",
      "epoch: 5\n",
      "maxTrainTime: 8640000\n",
      "validationPatience: 10\n",
      "saveEveryEpoch: 0\n",
      "loss: hinge\n",
      "margin: 0.05\n",
      "similarity: cosine\n",
      "maxNegSamples: 10\n",
      "negSearchLimit: 10\n",
      "batchSize: 5\n",
      "thread: 10\n",
      "minCount: 2\n",
      "minCountLabel: 1\n",
      "label: __label__\n",
      "label: __label__\n",
      "ngrams: 1\n",
      "bucket: 2000000\n",
      "adagrad: 1\n",
      "trainMode: 3\n",
      "fileFormat: labelDoc\n",
      "normalizeText: 0\n",
      "dropoutLHS: 0\n",
      "dropoutRHS: 0\n",
      "useWeight: 0\n",
      "weightSep: :\n",
      "Start to initialize starspace model.\n",
      "Build dict from input file : /content/data/prepared_train_data.tsv\n",
      "Read 12M words\n",
      "Number of words in dictionary:  95058\n",
      "Number of labels in dictionary: 0\n",
      "Loading data from file : /content/data/prepared_train_data.tsv\n",
      "Total number of examples loaded : 999740\n",
      "Initialized model weights. Model size :\n",
      "matrix : 95058 100\n",
      "Training epoch 0: 0.05 0.01\n",
      "Epoch: 100.0%  lr: 0.040010  loss: 0.043385  eta: 0h5m  tot: 0h1m22s  (20.0%)\n",
      " ---+++                Epoch    0 Train error : 0.04398545 +++--- ☃\n",
      "Training epoch 1: 0.04 0.01\n",
      "Epoch: 100.0%  lr: 0.030090  loss: 0.013090  eta: 0h3m  tot: 0h2m37s  (40.0%)\n",
      " ---+++                Epoch    1 Train error : 0.01329004 +++--- ☃\n",
      "Training epoch 2: 0.03 0.01\n",
      "Epoch: 100.0%  lr: 0.020040  loss: 0.009414  eta: 0h2m  tot: 0h3m51s  (60.0%)\n",
      " ---+++                Epoch    2 Train error : 0.00944294 +++--- ☃\n",
      "Training epoch 3: 0.02 0.01\n",
      "Epoch: 100.0%  lr: 0.010130  loss: 0.007588  eta: 0h1m  tot: 0h5m5s  (80.0%)\n",
      " ---+++                Epoch    3 Train error : 0.00765867 +++--- ☃\n",
      "Training epoch 4: 0.01 0.01\n",
      "Epoch: 100.0%  lr: 0.000170  loss: 0.006908  eta: <1min   tot: 0h6m19s  (100.0%)\n",
      " ---+++                Epoch    4 Train error : 0.00689065 +++--- ☃\n",
      "Saving model to file : /content/startspace_stackoverflow_model_03\n",
      "Saving model in tsv format : /content/startspace_stackoverflow_model_03.tsv\n"
     ]
    }
   ],
   "source": [
    "!starspace train -trainFile /content/data/prepared_train_data.tsv -model /content/startspace_stackoverflow_model_03 -trainMode 3 -lr 0.05 -dim 100 -epoch 5 -negSearchLimit 10 -minCount 2 -adagrad true -fileFormat 'labelDoc' -verbose 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "1D6ddNOn7DK7"
   },
   "source": [
    "And now we can compare the new embeddings with the previous ones. You can find trained word vectors in the file *[model_file_name].tsv*. Upload the embeddings from StarSpace into a dict. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "X3B3V4cACS3T",
    "outputId": "e7fe7ac9-915d-4b01-ccc6-566688003495"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.config',\n",
       " 'Starspace',\n",
       " 'GoogleNews-vectors-negative300.bin',\n",
       " 'data',\n",
       " 'startspace_stackoverflow_model_03',\n",
       " 'GoogleNews-vectors-negative300.bin.gz',\n",
       " 'natural-language-processing',\n",
       " 'startspace_stackoverflow_model_03.tsv',\n",
       " 'sample_data']"
      ]
     },
     "execution_count": 51,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir('/content/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "jTnoeyEHA-5M",
    "outputId": "880b2f9b-7492-4846-a060-bcbbce59fa6a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/content/data/starspace_stackoverflow_model_03'"
      ]
     },
     "execution_count": 52,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import shutil\n",
    "shutil.copy(\"/content/startspace_stackoverflow_model_03.tsv\",\"/content/data/starspace_stackoverflow_model_03.tsv\")\n",
    "shutil.copy(\"/content/startspace_stackoverflow_model_03\",\"/content/data/starspace_stackoverflow_model_03\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CaV27LEgAjHA"
   },
   "outputs": [],
   "source": [
    "in_='/content/data/starspace_stackoverflow_model_03.tsv'\n",
    "out_='/content/data/starspace_stackoverflow_model_03.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 88
    },
    "colab_type": "code",
    "id": "ejrASwxhH6wI",
    "outputId": "c664a228-aaac-4d7c-e97f-664b24829c49"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['using', '-0.00239059', '0.00951964', '-0.0103395', '-0.00153699', '-0.000757079', '0.0155529', '-0.00681664', '-0.0128238', '0.00139905', '0.00436572', '-0.0111223', '0.00836997', '-0.00677271', '-0.00361113', '-0.00496252', '-0.0133998', '-0.000844995', '0.0180437', '0.00820865', '0.0257946', '0.00437668', '-0.0141056', '0.0053542', '0.006689', '0.00922228', '-0.0136161', '0.0242094', '7.83168e-05', '-0.0397751', '-0.0106131', '0.00374198', '-0.0308243', '-0.00386576', '-0.0104921', '-0.00839556', '-0.0136116', '-0.00732436', '-0.0232465', '-0.00474069', '0.00446419', '-0.0190227', '-0.00945454', '-0.00177378', '-0.0075927', '-0.00688368', '0.00768384', '-0.010331', '-0.0278613', '0.00726723', '-0.0109303', '0.0126516', '0.0165699', '0.00821907', '-0.0016834', '0.0111982', '0.0258603', '0.00901068', '-0.0245025', '0.0311753', '0.00517124', '-0.00708995', '0.0120408', '0.00992495', '0.0111854', '0.00768099', '-0.00980045', '0.00380898', '-0.00249913', '0.000807122', '-0.0224614', '-0.00114944', '0.00638051', '0.00200107', '0.0197434', '-0.0106372', '0.00429295', '-0.020343', '-0.00587995', '0.00294934', '0.0135362', '-0.014199', '0.0100038', '0.00817768', '-0.0127639', '-0.00347189', '0.00888593', '0.0210385', '0.0165162', '-0.00946857', '-0.0134543', '-0.000810565', '0.0140659', '-0.00312497', '0.0160845', '-0.00801434', '0.00708949', '-0.0117696', '-0.00279048', '-0.0147452', '-0.0109606']\n",
      "using -0.00239059 0.00951964 -0.0103395 -0.00153699 -0.000757079 0.0155529 -0.00681664 -0.0128238 0.00139905 0.00436572 -0.0111223 0.00836997 -0.00677271 -0.00361113 -0.00496252 -0.0133998 -0.000844995 0.0180437 0.00820865 0.0257946 0.00437668 -0.0141056 0.0053542 0.006689 0.00922228 -0.0136161 0.0242094 7.83168e-05 -0.0397751 -0.0106131 0.00374198 -0.0308243 -0.00386576 -0.0104921 -0.00839556 -0.0136116 -0.00732436 -0.0232465 -0.00474069 0.00446419 -0.0190227 -0.00945454 -0.00177378 -0.0075927 -0.00688368 0.00768384 -0.010331 -0.0278613 0.00726723 -0.0109303 0.0126516 0.0165699 0.00821907 -0.0016834 0.0111982 0.0258603 0.00901068 -0.0245025 0.0311753 0.00517124 -0.00708995 0.0120408 0.00992495 0.0111854 0.00768099 -0.00980045 0.00380898 -0.00249913 0.000807122 -0.0224614 -0.00114944 0.00638051 0.00200107 0.0197434 -0.0106372 0.00429295 -0.020343 -0.00587995 0.00294934 0.0135362 -0.014199 0.0100038 0.00817768 -0.0127639 -0.00347189 0.00888593 0.0210385 0.0165162 -0.00946857 -0.0134543 -0.000810565 0.0140659 -0.00312497 0.0160845 -0.00801434 0.00708949 -0.0117696 -0.00279048 -0.0147452 -0.0109606\n",
      "line_count= 95058\n"
     ]
    }
   ],
   "source": [
    "import csv\n",
    "line_count=0\n",
    "with open(in_) as fd:\n",
    "    rd = csv.reader(fd, delimiter=\"\\t\")\n",
    "    for row in rd:\n",
    "        line_count+=1\n",
    "        if line_count==1:\n",
    "           print(row)\n",
    "           print(' '.join(words for words in row))\n",
    "print(\"line_count=\",line_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c0p2Gc5h-foz"
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "def transform_tsv_to_txt(in_, out_):\n",
    "    out = open(out_, 'w')\n",
    "    line_count='95058' # line count of tsv file (as string)\n",
    "    dimensions='100' # vector size (as string)\n",
    "    out.write(' '.join(words for words in [line_count, dimensions])+'\\n')\n",
    "    with open(in_) as fd:\n",
    "         rd= csv.reader(fd, delimiter=\"\\t\")\n",
    "         for line in rd:\n",
    "            out.write(' '.join(words for words in line) + '\\n')\n",
    "  \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qGgcR-ORK7lP"
   },
   "outputs": [],
   "source": [
    "transform_tsv_to_txt(in_,out_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 173
    },
    "colab_type": "code",
    "id": "e3a6CP4yQi0Q",
    "outputId": "6c6e3fe3-e074-4663-bf94-5b0011815813"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "95058 100\n",
      "\n",
      "using -0.00239059 0.00951964 -0.0103395 -0.00153699 -0.000757079 0.0155529 -0.00681664 -0.0128238 0.00139905 0.00436572 -0.0111223 0.00836997 -0.00677271 -0.00361113 -0.00496252 -0.0133998 -0.000844995 0.0180437 0.00820865 0.0257946 0.00437668 -0.0141056 0.0053542 0.006689 0.00922228 -0.0136161 0.0242094 7.83168e-05 -0.0397751 -0.0106131 0.00374198 -0.0308243 -0.00386576 -0.0104921 -0.00839556 -0.0136116 -0.00732436 -0.0232465 -0.00474069 0.00446419 -0.0190227 -0.00945454 -0.00177378 -0.0075927 -0.00688368 0.00768384 -0.010331 -0.0278613 0.00726723 -0.0109303 0.0126516 0.0165699 0.00821907 -0.0016834 0.0111982 0.0258603 0.00901068 -0.0245025 0.0311753 0.00517124 -0.00708995 0.0120408 0.00992495 0.0111854 0.00768099 -0.00980045 0.00380898 -0.00249913 0.000807122 -0.0224614 -0.00114944 0.00638051 0.00200107 0.0197434 -0.0106372 0.00429295 -0.020343 -0.00587995 0.00294934 0.0135362 -0.014199 0.0100038 0.00817768 -0.0127639 -0.00347189 0.00888593 0.0210385 0.0165162 -0.00946857 -0.0134543 -0.000810565 0.0140659 -0.00312497 0.0160845 -0.00801434 0.00708949 -0.0117696 -0.00279048 -0.0147452 -0.0109606\n",
      "\n",
      "android -0.00391709 -0.0472689 0.0133037 0.0618566 0.0257844 -0.031652 0.00328812 -0.0782126 0.0424345 0.00588315 0.0370231 0.0214065 0.0325652 0.00885347 0.0253749 0.00428678 0.0131615 0.0423935 -0.00818683 -0.0440409 -0.0300703 -0.0488727 0.0727313 0.0191879 -0.0428828 0.0462249 -0.0483753 -0.000875021 0.027569 0.0670776 -0.018759 0.0460397 -0.128767 -0.0405324 -0.0198019 0.0789792 -0.0647647 -0.0296245 0.0250241 0.0849931 -0.0329854 -0.0754499 0.00222394 -0.0812417 0.0245354 -0.00979253 0.00765503 0.0326502 -0.0642324 -0.00673893 0.00957823 0.100517 0.0384735 0.013793 -0.0177012 -0.0121744 -0.0737033 -0.0102158 -0.0590195 0.0038996 0.0843493 0.04064 0.00322103 -0.109285 0.0730654 0.0648247 0.0173853 0.0114193 -0.0427276 0.0431394 -0.00826057 0.0386325 -0.0452733 0.0212755 0.0415398 -0.0450823 -0.0525925 -0.0188808 0.056512 0.0294815 -0.0205851 -0.0340656 0.0330956 0.0610109 0.0541638 -0.0113773 0.00472733 -0.0119766 -0.0162558 0.00715095 0.0696681 -0.0190133 -0.00955941 0.00415494 -0.00597593 0.002607 0.0178769 0.0273852 -0.0444083 0.0450371\n",
      "\n",
      "file 0.0274776 -0.0132635 0.0398729 0.00515028 -0.0242121 0.0190134 -0.0289662 -0.0168738 -0.0509574\n",
      " -0.00666667 -0.00510427 -0.0177556 0.0745567 0.0351407 -0.0191535 -0.0453947 -0.031979 0.0459261 0.\n"
     ]
    }
   ],
   "source": [
    "file=open(out_,'r')\n",
    "print(file.readline(20))\n",
    "print(file.readline(50000))\n",
    "print(file.readline(10000))\n",
    "print(file.readline(100))\n",
    "print(file.readline(100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "8fFQujIDbUBk",
    "outputId": "e5212b33-5155-4818-a1f4-9387c59f8aab"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['train.tsv',\n",
       " 'starspace_stackoverflow_model_03',\n",
       " 'test_embeddings.tsv',\n",
       " 'prepared_train_data.tsv',\n",
       " 'starspace_stackoverflow_model_03.txt',\n",
       " 'test.tsv',\n",
       " 'starspace_stackoverflow_model_03.tsv',\n",
       " 'prepared_validation_data.tsv',\n",
       " 'validation.tsv',\n",
       " 'prepared_test_data.tsv']"
      ]
     },
     "execution_count": 58,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.listdir('/content/data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WSvUTHR-Glrk"
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "LTy-W0VM7DK-",
    "outputId": "c45359ef-23d7-4cc8-f1bc-97d9e9de7034"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
     ]
    }
   ],
   "source": [
    "#starspace_embeddings = ######### YOUR CODE HERE #############\n",
    "#file=\"/content/data/starspace_stackoverflow_model_03\"\n",
    "file=out_\n",
    "#starspace_embeddings = KeyedVectors.load_word2vec_format(datapath(file))\n",
    "\n",
    "starspace_embeddings = KeyedVectors.load_word2vec_format(datapath(file),binary=False,limit=500000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "deMzuA707DLE"
   },
   "outputs": [],
   "source": [
    "ss_prepared_ranking = []\n",
    "for line in prepared_validation:\n",
    "    q, *ex = line\n",
    "    ranks = rank_candidates(q, ex, starspace_embeddings, 100)\n",
    "    ss_prepared_ranking.append([r[0] for r in ranks].index(0) + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "mJENPwRh7DLJ",
    "outputId": "f9f6b2c7-d71c-4de8-b443-5c791887e5cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DCG@   1: 0.521 | Hits@   1: 0.521\n",
      "DCG@   5: 0.618 | Hits@   5: 0.699\n",
      "DCG@  10: 0.636 | Hits@  10: 0.754\n",
      "DCG@ 100: 0.667 | Hits@ 100: 0.903\n",
      "DCG@ 500: 0.677 | Hits@ 500: 0.980\n",
      "DCG@1000: 0.679 | Hits@1000: 1.000\n"
     ]
    }
   ],
   "source": [
    "for k in [1, 5, 10, 100, 500, 1000]:\n",
    "    print(\"DCG@%4d: %.3f | Hits@%4d: %.3f\" % (k, dcg_score(ss_prepared_ranking, k), \n",
    "                                               k, hits_count(ss_prepared_ranking, k)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D1Gtsijm7DLO"
   },
   "source": [
    "Due to training for the particular task with the supervised data, you should expect to obtain a higher quality than for the previous approach. In additiion, despite the fact that StarSpace's trained vectors have a smaller dimension than word2vec's, it provides better results in this task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y6EpSqFX7DLQ"
   },
   "source": [
    "**Task 5 (StarSpaceRanks).** For each question from prepared *test.tsv* submit the ranks of the candidates for trained representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "wffVAvBg7DLU",
    "outputId": "26801c43-573d-4396-c0e3-1458a86b3107"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current answer for task StarSpaceRanks is: 78\t99\t30\t8\t43\t80\t76\t38\t47\t77\t14\t7\t18\t27\t91\t52\t9\t89\t31\t21\t86\t39\t6\t97\t71\t4\t55\t100\t57\t1\t56\t60\t81\t88\t70\t...\n"
     ]
    }
   ],
   "source": [
    "starspace_ranks_results = []\n",
    "#prepared_test_data = ######### YOUR CODE HERE #############\n",
    "\n",
    "for line in open('/content/data/prepared_test_data.tsv'):\n",
    "    q, *ex = line.strip().split('\\t')\n",
    "    ranks = rank_candidates(q, ex, starspace_embeddings, 100)\n",
    "    ranked_candidates = [r[0] for r in ranks]\n",
    "    starspace_ranks_results.append([ranked_candidates.index(i) + 1 for i in range(len(ranked_candidates))])\n",
    "    \n",
    "grader.submit_tag('StarSpaceRanks', matrix_to_string(starspace_ranks_results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0QF4WPdr7DLZ"
   },
   "source": [
    "Please, **don't remove** the file with these embeddings because you will need them in the final project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WdTVtfc87DLa"
   },
   "source": [
    "### Authorization & Submission\n",
    "To submit assignment parts to Cousera platform, please, enter your e-mail and token into variables below. You can generate token on this programming assignment page. <b>Note:</b> Token expires 30 minutes after generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Gv_mC4dD7DLc"
   },
   "outputs": [],
   "source": [
    "STUDENT_EMAIL = \"ruperto.bonet@gmail.com\"\n",
    "STUDENT_TOKEN = # TOKEN \n",
    "grader.status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bldxF2RF7DLg"
   },
   "source": [
    "If you want to submit these answers, run cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OQtSEEaq7DLh"
   },
   "outputs": [],
   "source": [
    "grader.submit(STUDENT_EMAIL, STUDENT_TOKEN)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "week3-Embeddings-colab.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "03453a45770a4cdd9ee86668a548175f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0672487f0c9a41388889449eb1783365": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1110c036cc4a4cd595a4a844b1a41883": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_19fb87ca329647fc9a126f520e818d4c",
       "IPY_MODEL_d91135a8bad44ef8bc5ca456465a545f"
      ],
      "layout": "IPY_MODEL_0672487f0c9a41388889449eb1783365"
     }
    },
    "19fb87ca329647fc9a126f520e818d4c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "IntProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "IntProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "train.tsv",
      "description_tooltip": null,
      "layout": "IPY_MODEL_42ec9a54fd864a74b9b7e9c59f58a2ef",
      "max": 119127793,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_83164a11df3a439ba793207c043e7c3f",
      "value": 119127793
     }
    },
    "29cf43ae622546ba8c4955ae418f1de6": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2fd4ab2863e649368eca57d04324e366": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3cfa31d9e32a4c9e8d2377df0341bb8f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7af713c5ad5c46069a818af2d185519e",
       "IPY_MODEL_8fc7851ee1e648a8a8e95f3d2de125e9"
      ],
      "layout": "IPY_MODEL_e343370fff704459983677226e89c4e4"
     }
    },
    "42ec9a54fd864a74b9b7e9c59f58a2ef": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "460b0f6d77484794bc63e9311eb5f38d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ec06b708ff0944d093d0fa8ae264e5ad",
      "placeholder": "​",
      "style": "IPY_MODEL_db97e55fc65f4f84b899c33630c31211",
      "value": "100% 536M/536M [00:32&lt;00:00, 16.6MB/s]"
     }
    },
    "4f07c2f10cfd43cd83a1d4fec5d63723": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5cf8df791ebd43e1b10b9a746e43b731": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_da3e40b27bc14fdd99921adde06eaaad",
       "IPY_MODEL_c3e2733bc15642c6a3270c1e8a908558"
      ],
      "layout": "IPY_MODEL_7af1ea82bf9d4cd681f479b899611cfd"
     }
    },
    "5eaf251296e049d68500612b60ae73e3": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "63572138a97940e0b844ee2b3ecdd37e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c884661dc9e4277a59cf20e3f4308db": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7201ad2a82054603ba30d3ba649aa873": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "731830e3cd8d428aa6f060324887e1d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f199c968aca44eb580872f18a75d1b6a",
       "IPY_MODEL_c287c385af9342b889a3a5ff5bb8d541"
      ],
      "layout": "IPY_MODEL_6c884661dc9e4277a59cf20e3f4308db"
     }
    },
    "75683a90194444a1a00fffafd6e2cb94": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "7af1ea82bf9d4cd681f479b899611cfd": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7af713c5ad5c46069a818af2d185519e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "IntProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "IntProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "test_embeddings.tsv",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2fd4ab2863e649368eca57d04324e366",
      "max": 5333,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_7e24556aab8941b4817c3aec4f7a9c53",
      "value": 5333
     }
    },
    "7bb3d98550ff40d58ca8b281c935f7e0": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7e24556aab8941b4817c3aec4f7a9c53": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "83164a11df3a439ba793207c043e7c3f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "8d963d100aa84e9eab346822533ba948": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8fc7851ee1e648a8a8e95f3d2de125e9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_963ac1a5aa344bc99f353d2e8ac759a8",
      "placeholder": "​",
      "style": "IPY_MODEL_03453a45770a4cdd9ee86668a548175f",
      "value": "100% 5.33k/5.33k [00:00&lt;00:00, 252kB/s]"
     }
    },
    "963ac1a5aa344bc99f353d2e8ac759a8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9a8563ae67cb4e519568b70e504a58f4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9d5e66da0fdb4d648d9435e3b03a321f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9dfa47b67e114079b0f31db867180c2d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "aca691b408ef4e74b59a932be8d3b9b1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_bbafc7af5afe450b8db6b40ed4497ae8",
       "IPY_MODEL_460b0f6d77484794bc63e9311eb5f38d"
      ],
      "layout": "IPY_MODEL_7201ad2a82054603ba30d3ba649aa873"
     }
    },
    "b0a9ea14160344a5a06019885be5be6b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "bbafc7af5afe450b8db6b40ed4497ae8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "IntProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "IntProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "validation.tsv",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7bb3d98550ff40d58ca8b281c935f7e0",
      "max": 535543630,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9dfa47b67e114079b0f31db867180c2d",
      "value": 535543630
     }
    },
    "c287c385af9342b889a3a5ff5bb8d541": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_63572138a97940e0b844ee2b3ecdd37e",
      "placeholder": "​",
      "style": "IPY_MODEL_9d5e66da0fdb4d648d9435e3b03a321f",
      "value": "100% 46.4M/46.4M [00:03&lt;00:00, 11.7MB/s]"
     }
    },
    "c3e2733bc15642c6a3270c1e8a908558": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_29cf43ae622546ba8c4955ae418f1de6",
      "placeholder": "​",
      "style": "IPY_MODEL_b0a9ea14160344a5a06019885be5be6b",
      "value": "100% 1.65G/1.65G [01:36&lt;00:00, 17.0MB/s]"
     }
    },
    "d91135a8bad44ef8bc5ca456465a545f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5eaf251296e049d68500612b60ae73e3",
      "placeholder": "​",
      "style": "IPY_MODEL_8d963d100aa84e9eab346822533ba948",
      "value": "100% 119M/119M [00:08&lt;00:00, 14.4MB/s]"
     }
    },
    "da3e40b27bc14fdd99921adde06eaaad": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "IntProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "IntProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "GoogleNews-vectors-negative300.bin.gz",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9a8563ae67cb4e519568b70e504a58f4",
      "max": 1647046227,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e72d84d4e4f44b2fa944b4f6fcc06b14",
      "value": 1647046227
     }
    },
    "db97e55fc65f4f84b899c33630c31211": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e343370fff704459983677226e89c4e4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e72d84d4e4f44b2fa944b4f6fcc06b14": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "ec06b708ff0944d093d0fa8ae264e5ad": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f199c968aca44eb580872f18a75d1b6a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "IntProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "IntProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "test.tsv",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4f07c2f10cfd43cd83a1d4fec5d63723",
      "max": 46408910,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_75683a90194444a1a00fffafd6e2cb94",
      "value": 46408910
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}