semeval-2022/Semeval_2022_Task_4_Demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Semeval 2022 - Task 4 - Demo",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "HpRLLRzkTwdL"
      },
      "source": [
        "import random\n",
        "import os\n",
        "from urllib import request"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p3srm96X7Jo9"
      },
      "source": [
        "# Resources\n",
        "\n",
        "In order to run this notebook, you first need the following: "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8h8sihAs6vYN"
      },
      "source": [
        "## The _Don't Patronize Me!_ dataset\n",
        "\n",
        "Files contained in the _Don't Patronize Me!_ data file, which you can access from [here](https://github.com/Perez-AlmendrosC/dontpatronizeme) (you need to fill a short form and it will be sent to you)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXW58vOI6wUi"
      },
      "source": [
        "## The `dont_patronize_me.py` module\n",
        "\n",
        "You can get this file by running the cell below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6MWC6jV16zt6",
        "outputId": "5ab6cd89-6020-4620-bdf4-f4149dfa3dd3"
      },
      "source": [
        "module_url = f\"https://raw.githubusercontent.com/Perez-AlmendrosC/dontpatronizeme/master/semeval-2022/dont_patronize_me.py\"\n",
        "module_name = module_url.split('/')[-1]\n",
        "print(f'Fetching {module_url}')\n",
        "#with open(\"file_1.txt\") as f1, open(\"file_2.txt\") as f2\n",
        "with request.urlopen(module_url) as f, open(module_name,'w') as outf:\n",
        "  a = f.read()\n",
        "  outf.write(a.decode('utf-8'))"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fetching https://raw.githubusercontent.com/Perez-AlmendrosC/dontpatronizeme/master/semeval-2022/dont_patronize_me.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pw4hx5VU6x0f"
      },
      "source": [
        "## The official scorer\n",
        "\n",
        "The script `evaluation.py` can also be obtained by running the cell below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VrDVHr6SAv0I",
        "outputId": "71e4b7ed-d2a0-4d0d-ab2e-baea899f0915"
      },
      "source": [
        "module_url = f\"https://raw.githubusercontent.com/Perez-AlmendrosC/dontpatronizeme/master/semeval-2022/evaluation.py\"\n",
        "module_name = module_url.split('/')[-1]\n",
        "print(f'Fetching {module_url}')\n",
        "#with open(\"file_1.txt\") as f1, open(\"file_2.txt\") as f2\n",
        "with request.urlopen(module_url) as f, open(module_name,'w') as outf:\n",
        "  a = f.read()\n",
        "  outf.write(a.decode('utf-8'))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fetching https://raw.githubusercontent.com/Perez-AlmendrosC/dontpatronizeme/master/semeval-2022/evaluation.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fXfGGCr2B2NY"
      },
      "source": [
        "At this point, your left pane on Google Colab should look like this:\n",
        "\n",
        "![Screenshot from 2021-09-27 17-13-06.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUcAAAERCAYAAADonmWBAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AAAAmdEVYdENyZWF0aW9uIFRpbWUAbHVuIDI3IHNlcCAyMDIxIDE3OjEzOjA53WBhjQAAIABJREFUeJzs3XtYVNX++PE3w3CJWzGjgIoXBDUFZAB1uB1F1CElVNLMskPn6fJomtVXk7JON083LfVYR8s62VGzvHdMCwEBD4po3oZAJRE1kJtyjYuIgL8/eGb/HBi5yQjaej2PzyN79t5r7b1nPrP2Wmv2x8TPz+8GgiAIgh5ZV1dAEAShOxLBURAEwQARHAVBEAwQwVEQBMEAERwFQRAMEMFREATBABEcBUEQDBDBURAEwQARHAVBEAwQwVEQBMEAERwFQRAMEMFREATBABEcBUEQDBDBURAEwQARHAVBEAwQwVEQBMEAeWfsxNXVlcDAwDatW1lZyc6dOzujWEEQBKO57eAYGhpKVFRUu7Zxc3Nj2bJlt1u0IAiC0dz2bfVTTz3V7m06ElAFQRDupNtuOTo6OgKwYcMG1q9f3+K6K1aswMvLC2gMkEC3bEG++uqr+Pv7Y2tre8t1KioqWLhwIVlZWXewZp3nz3CMgnA7unRApju2IAMDA9FoNC0GDQBbW1uWL1+Oq6vrHapZ5/kzHKMg3K5OGZC5Hd2tBenm5tbmdXXBoz0DTLGxsRQUFHSkap3mz3CMgnC7ujw4QvcLkO1ha2vbrn7XRx55hDlz5txVwePPcIyC0FS3mefYHW+xjcHW1rbN0566m8LCQlJTU6V/t+qLvJuPURB0ukXLUSc0NLRbtR7HjRvXqfuLj48HwMbGplP3e6esX7+emJgY6W8bGxs2bdpk8Hju1mMUBJ07GhxXr15tcBAgNDQUjUZzJ6sitFNqairTpk3TC46VlZVs2LCBuXPndmHNBME47mhwvNVtmG56j9B9rV69Gjc3N6ZNm8aOHTuk5Tt27GDatGnSlC5BuFfcsT5HcZuF1Fd3tw1UxMbGkpWVRUxMDF5eXs2u5erVq7uoZoJgPHckOL766qssX778Tx8gFyxYwIIFC/RuTbu7yspKveC3c+dOIiMj9dZJTk4mNTX1TldNEIzK6MHx1VdfRaPR4Obmxrvvvmvs4rqNwMBAIiMjiYqKIjIykoCAAKDx55bx8fEd+tllV9iwYQOVlZXS31qtFjc3N5ycnPTWE61H4V5j1D5HXWDUUalUREVFdasR6c7m5OTEokWLUKlUzV7TarVcvny5C2rVMYWFhXr9i9B4fF5eXjz//PO8/fbb0vKsrCxiY2PFwJpwzzBacGwaGHXu5gnfrbGxseGLL77A1taW1NRUYmJiKCwsxNHRkWnTphkMmN2ZoWukm4saFBSESqVCq9VKr61evZqAgIA/ffeJcG8wSnC8OTBWVlZKHxbd/+/VALlkyRJsbW2JjY1l6dKl0vLQ0FAOHjyIg4NDq79n7i6Sk5P1Ah80tvxtbGyk/sXIyMhm/Y+FhYUiOAr3BKMER61Wi0ajISsri/Xr17NkyRKgMRg+9dRTuLq63nMd+LrbzcLCQr3ACI3B8W6brpScnNxs2dy5c1mwYIFeH6QhusnugnA3M0pwjImJobKyktTUVL2HHFRVVbFgwQK8vLwMfvjuZrrgZ2gk2tDk9+4+nScyMlLvWHTzG1sLjHfLQJMgtMZofY63Cn6VlZX3XGAEWgwad+PzEJ2cnAgNDSUmJgYbGxsCAwNZsGBBq9s0vc0WhLtVt/ptdXfT1g/6zp07qaqqAmg2xaWp0NBQHB0d74rHekVGRlJQUEBoaCharbbVrgERGIV7idGDY0VFhdS/WFFRYezibtvNAautt4gmJibs2LGDwsJCNBoN69evNxj4dFOZCgsL2bBhQ6fVub3aGpSdnJxYsWJFh8o4d+5ch7YThO7C6JPAs7KypF+G3A23lzExMcTGxrbat3aziIgIAClNxBdffNHs6dmBgYG88847QNdPmO7IMbbHjh077smuE+HPxcTPz+/G7exg165dnTZ1o7CwkCeeeKJT9tVV5s6dy7Rp04DG1pNu+pJuYKotuXYEQeh6ps7Ozu/czg5KS0s77cGma9asuStaly05evQoWVlZmJubo1KpcHJyQqFQkJqaypo1a9i9e3dXV1EQhDa47ZYjgKurK0FBQbe1j4MHD971gbEpXYux6WRqQRC6v04JjoIgCPeabpNDRhAEoTsRwVEQBMEAERwFQRAMEMFREATBABEcBUEQDBDBURAEwQARHAVBEAwQwVEQBMEAERwFQRAMEMFREATBABEcBUEQDJAXFhZ2dR0EQRC6Hbmjo2NX10EQBKHbEbfVgiAIBojgKAiCYIAIjoIgCAaI4CgIgmCACI6CIAgGiOAoCIJggAiOgiAIBojgKAiCYIAIjoIgCAaI4CgIgmCACI6CIAgGiOAoCIJggAiOgiAIBojgKAiCYIAIjoIgCAbIjbFTGxsbFi1ahI2NjcHXKysr+fjjj6msrDRG8YIgCLfNKMExKiqKwMDAFtdxcnJi4cKFIkAKgtAtmfj5+d3o7J3Gx8e3ab2CggIKCgpaXKewsJA1a9aIICoIwh1llJZjWzk5OeHk5NTqejY2Nrz11lt3oEaCIAiN7uiATGVlJRs2bGj3dsOHDzdCbQRBEG7tjrUcKysrWbBgAVlZWRQUFBAVFdXmbW1tbY1YM0EQhObuSMvx5sAIEBMTw7Jly+5E0YIgCB1i9ODYNDDqiAApCEJ3ZtTgeKvAqCMCpCAI3ZVRg6NWqyUwMJDIyMhmo9IqlYrIyEgcHR05d+6cMashCILQbkYdkAkKCiIoKAiAX3/9VW9Oo5eXF5GRkZ1a3v33388777xDfHw8e/bsAWDatGmo1WreffddqqqqOrU8QRDuXffUb6tlMhkymQxTU1NpmVwuRy6XI5PdU4cqCIKRdekk8M5WWlrKSy+9pLdsy5YtbNmypYtqJAjC3eqOBceBAwdy48b//6Wio6PjnSpaEASh3e5YcJw3b96dKkoQBOG2GaUjLjk5uVvvTxAEoTVGeSqPjY0NUVFRt3yeY3tUVlaybNky8VQeQRDuKKMER0EQhLudmN8iCIJggAiOgiAIBojgKAiCYIAIjoIgCAaI4CgIgmCACI6CIAgGiOAoCIJggAiOgiAIBojgKAiCYIAIjoIgCAaI4NiFQkJC+Prrr+nVq5fRy5o4cSJff/019vb2Ri9LEO4FIjgKLVIoFEyZMoU+ffp0dVUE4Y4yyvMcbWxsWLRo0S2fylNZWcnHH38snrRzF1AoFEyePJn8/Hxyc3O7ujqCcMcYJThGRUURGBjY4jpOTk4sXLhQBEhBELolowTH1gIjgJubG2vXrtXLSGhIYWEha9asaXMQ7dmzJ9OnT2fw4MHIZDIuXLjAzp07yc7OltaxtbUlIiIClUqFpaUlBQUF7NmzhxMnTkjrvP3225SUlJCWloZGo0GhUFBYWMiWLVu4ePEijz32GCqVChMTE86cOcOmTZv4448/gMbMii+++CLr1q3D398fV1dXamtrOXLkCNu2beP69estHoNGoyE4OFgq86effuKXX35p0/EDWFlZMXPmTHx9famvr0er1VJeXt5svdbOw6pVq6TW/+zZs5k9ezbz58+nuroamUzGxIkTCQoK4oEHHqCsrIyDBw+yd+9e6uvr21xXQeiuTJ2dnd/p7J0+9dRTbVrPxsYGJyenFv+5ubnRr18/EhMTW92fpaUlf//731EoFMTHx3Pu3DmGDx/OmDFjOHbsGNXV1cjlct544w1cXFxITEzk5MmT9O7dm9DQUM6ePUtRUREAwcHBuLi44OTkREpKCtnZ2Tz44IOo1Wq8vLywsLAgOTmZyspKRo0aRa9evaQA5uTkhFqtZujQoSQlJZGQkEB9fT3jxo3D3t6ekydPAuDi4sLw4cNJSEiQgv+sWbOYNGkSx48f5+jRoyiVSkJDQ8nNzSU/P79N53X+/Pn4+vpy/Phxjh07Rr9+/VCr1chkMmJjY6mpqWnTecjJySE3N5dhw4YRHR1NdHQ0+fn53Lhxg8jISDQaDb/88gsHDx7E1NSUCRMmUF9fz9mzZ9tUT0Hozu6K7IPDhw9v03pDhgxBqVTyz3/+k7S0NADS0tJ49913GTFiBNHR0bi5uWFhYcG6detITU0FICUlhZUrV6JWq8nIyJD2V1NTwwcffMDVq1cByMnJYfbs2dTX1/Pxxx9LCcNMTU1Rq9XI5XLq6uqk7bds2UJSUhIA6enpWFhYEBAQwK5duyguLm5W//79+xMSEsJ3331HfHw8APv27eOtt94iPDyc48ePt3oO3NzccHd35+eff2bHjh0A7N27l/nz56NSqfTWa+08nDp1imvXrgGQnZ0trWdubs7QoUP5+eef+eGHHwA4cOAASqUSf39/du/e3Wo9BaG7k0VFRXV1HVpla2vbpvV0H+Rhw4ZJuasvXbrEnDlziI2NBSAjI4OoqCjpgw6NQbCkpIQePXro7a+wsFAKjAAXL14EIDU1VS+TYk5ODjKZDIVCobd9SUmJ3t/JycmYmJgwZMgQg/X38fGhoaGBtLQ07OzssLOzw9bWloyMDPr27YulpWWr58DDwwNACq46586d0/u7PeehqdraWl577TUpMOrk5eW1uq0g3C3koaGhACxbtqyLq3L7MjIyiI+PZ8KECQQFBXH69GlOnjzJ0aNH9frBevbsSVhYGEOGDEGhUCCXNzagmwazphoaGgCa9RnqlusC8q1cuXIFgAceeMDg6w4ODshkMj788EODr9vY2FBTU9NiGUqlkmvXrlFWVtbietDx8wCNXRgajYaRI0eiUCikwC36G4V7hRzgXgqQultSb29vPD09eeaZZwgLC+OTTz6hvLwchULBm2++SUlJCbt376awsJDa2lpeeOEFo9fNzMwMQO/W+2ZyuZyamho+//xzg6/rBnxaK6O1AR/gts6DTCZj4cKFODg4sGfPHrKzs6murubhhx/G29u71e0F4W4g9TmGhoaSmppKTExMV9bnttjb22NmZkZhYSF79+5l7969uLm5sXjxYiZMmMD27dvx9/fH2tqaJUuWSIMv0Lw1aAyOjo4ABvsbAUpLS7G0tOT333+noqKiQ2UUFxdjY2ODhYWF1M1gyO2ch4EDBzJw4EDWrVunlzZXTMsS7iXSL2RiY2Pv6sAIjT+Re//99/V+IpednU1dXR3m5uZAY6sH/n8rDqBPnz4olUpMTEw6tT7Ozs56f48dO5br16+TmZlpcH1d/9+UKVP0lj/44IP4+fm1qUzdQJS/v7/ecicnJ72/23oedLfJN3cZ6LbVnVNovM0eNGhQp59DQegqcmgMjEuXLu3quty2xMREAgMDWbRoEQcOHODatWuo1WpMTEykaTZarZbw8HBeeukljhw5gq2trbSOlZVVp9YnIiICpVJJXl4eXl5eeHp6Eh8ff8vb41OnTnHs2DHGjh2Lvb09p06dQqFQEBISQnp6OocPH261zIyMDNLT05k5cyZWVlbk5OTg6enZbO5pW8/D5cuXqaurY8KECdjb25OYmMjFixcpLi5m+vTpODk5cf36dUaNGoWdnR0ymQxLS8tW+0YFobuT3SuBESA/P5+lS5dy+fJlwsLCeOSRR7hx4warVq2SRmtzcnJYvXo1V69eRaPRMHjwYDZu3EhSUhINDQ2tDqq0x9atW+nZsyePPvooAwcOJD4+nq1bt7a4zdq1a/nvf/9Lnz59eOyxxxg1ahQxMTGsXbsWmUwmjWIb+qcbFPn888/55ZdfCAsLY86cOTzwwAPs2rVLr5y2noeqqiq2b99Ojx49GD9+PHK5nNraWlauXMm5c+cICgoiICCAlJQUvv32W7Kzs3FwcOi0cygIXcXEz8/vRuurtU/TaSSdYdy4cZ2+T2PR/UJm5cqVpKend9p+Bw4cyBtvvHHL15OSkli/fn2nlScIf2Z3xSRwoVF+fj4rV6685ettmYIjCELbiOB4F7l69WqntkQFQbg1ozzP8ebpHd1xf4IgCK0xSp+jjY0NUVFRt3yeY3tUVlaybNkyMYdOEIQ7yijBURAE4W4n0iQIgiAYIIKjIAiCASI4CoIgGCCCoyAIggEiOAqCIBgggqMgCIIBIjgKgiAYIIKjIAiCASI4CoIgGCCCoyAIggFGfSqPjY0NixYtIigoqMX1KioqWLhwIVlZWUapx8SJE5k+fTqvvPIKpaWlRilDEIR7i1FbjlFRUa0GxsrKSmxtbVm+fDmurq7GrI7RKBQKpkyZQp8+ff6UdXjyySf5xz/+IfLHCPcUowbHtjyVZ+fOncTGxt7VAVKhUDB58uQuD45dVYerV69SVVXFjRviGSbCvaNbPOxWl8NGo9GwfPlyo95i/9mZmppKGQU7y44dOzp1f4LQHXSL4AidFyCtrKyYOXMmvr6+1NfXo9VqKS8vN7heREQEPj4+WFlZcenSJXbv3s2vv/4qrTNjxgyCg4N5//33efzxx3F1daW8vJyEhARiY2MBWLVqldRCnj17NrNnz2b+/PlUV1e3WE9dnpl169bh7++Pq6srtbW1HDlyhG3btunlj7a1tSUiIgKVSoWlpSUFBQXs2bOHEydOtFiH+vp61qxZQ3x8PEqlEg8PD7Zs2UJCQoJ0rseMGYNSqaS0tJT9+/frpefV1XHFihWo1Wq8vb2pq6tDq9Xy/fffU1tbC8Dzzz/PoEGDWLBgQbP6NPXSSy9Jz+YcPnw44eHhODs7U1FRQXJyMrt376ahoaHFc6fz9ttvU1JSQlpaGhqNBoVCQWFhIVu2bOHixYs89thjqFQqTExMOHPmDJs2bdLL/Hjfffcxbdo0fH19sbCw4Pz582zbto3ff//9lmXOmDGDwMBANm3axMMPP4yjoyNlZWXs3buXxMREAJ577jlGjRrFwoUL9cpbvHgxSqWSV155pU3HJ3QtU2dn53eMtfPQ0NBm+ZKbSk1NlfI1Jycn06tXL4YNG0ZwcDBHjx5t9wDK/Pnz8fX15fjx4xw7dox+/fqhVquRyWTExsZSU1ODmZkZixcvxt3dnUOHDpGamkrfvn3RaDTk5eWRl5cHgLu7O4MHD8bX15esrCxOnDiBg4MDgYGBZGVlceXKFXJycsjNzWXYsGFER0cTHR1Nfn5+q7eYTk5OqNVqhg4dSlJSEgkJCdTX1zNu3Djs7e05efIkAHK5nDfeeAMXFxcSExM5efIkvXv3JjQ0lLNnz1JUVHTLOshkMsLCwnBxcaGsrIwDBw6QmZlJWVkZkZGRhIWFcebMGQ4dOoS5uTnjx4/HxsZGyn2tq6OHhwfV1dUcPnyYGzdu4OfnJwUcgJEjR6JUKqXAWlZWRnp6unRtz507x4MPPkheXh779u3jxo0bjBw5knnz5pGbm0tSUhLXr19n/PjxWFtbS+W3Jjg4GBcXF5ycnEhJSSE7O5sHH3wQtVqNl5cXFhYWJCcnU1lZyahRo+jVq5eUotfU1JTXXnuNQYMGkZiYyOnTpxk2bBghISH88ssvt/xyc3d3Z9iwYfTt25effvqJw4cP4+DgwLhx46RrcePGDdRqNQUFBWRnZwNgbW3N448/TkpKSpuPT+ha3ablqHNzC3LevHlSa6Qt3NzccHd35+eff5Zu9fbu3cv8+fNRqVTSeuPGjaNv3758+umnUmBOSEhg8eLFzJo1C61WS11dHQAmJiZs27aNlJQUoDGAL1++HJVKxalTpzh16hTXrl0DIDs7W9pfW23ZsoWkpCQA0tPTsbCwICAggF27dlFcXIybmxsWFhasW7dO2ndKSgorV65ErVaTkZFxyzrI5Y2XNyMjg08++UQqc8CAAYwZM4aYmBgpVWxsbCx//etfGTduHMnJyXqtJ61Wy3/+8x/p77fffhtvb2927txp8Jia5teeM2cODQ0N/Pvf/6a+vh5TU1OeeOIJTpw4wZo1a6T1ampqCA4OZs+ePbfM7d1UTU0NH3zwAVevXgUaU87Onj2b+vp6Pv74Y+lLytTUFLVajVwup66ujpCQEJydnXnvvfekAHbkyBE++OADJkyYwPfff99iuatWraKwsBBo/IJ/7733CA8P59ChQ6Snp1NTU4NKpeLAgQMAeHp6IpPJpNa+0P11+TzH4cOHExkZqfcvPz+fioqKdu/Lw8MDaJ4aVpezWsfX15e8vDy9QFZbW0tMTAx2dnYMHjxYb/1jx45J/6+oqKCsrAyFQtHu+hnSNGNgcnIyJiYmDBkyBGgMbFFRUXp1rampoaSkhB49erSpjAsXLuj9PWLECACio6P1lu/ZswcAHx8fveU3Hz80BqC2Hr9arWbkyJH8+OOPUhAaMmQIdnZ2nDhxQi/vdkZGBqampri4uLRp3wCFhYVSYAS4ePEi0Biwbm695+TkIJPJpHr7+vqSm5tLWVmZVH59fT05OTm4ubm1Wu7N162hoUFqQSoUCq5fv45Wq2XYsGGYm5sDje/ziooKzp492+ZjE7pWl7ccVSqVXqvudiiVSq5du0ZZWVmL6zk6OpKRkdFsue52umfPni1uX1dXJyW972xXrlwB4IEHHpCW9ezZk7CwMIYMGYJCoZBahB1Nxerg4EBVVVWzL6DS0lKuXr3a6vHrWn+tsbe3Z9asWZw7d04vEDs4OACNfXOG3E7uIV1/5c19tjcv19XbwcGB+++/32Cq2+Li4naXW1RUBDQec0lJCceOHcPPzw8PDw+0Wi3u7u6cOHGizf2pQtfr8uDYmczMzJp9KAwxMTExOGKr++B05RvYzMwMQLqtVygUvPnmm5SUlLB7924KCwupra3lhRde6HAZMplM2r+h1zprSs7TTz+NqakpX3/9td451QX3b7/9VvoyuFlubm6nlN8SuVxOZmam1Fq+2a3OTWv7u3nbtLQ0ampq8PHxoaKiAhsbG44fP357lRbuqHsqOBYXF2NjY4OFhYXUB2fIlStX6N27d7PlvXr1kvbTVRwdHfXq4O/vj7W1NUuWLJFaJ9C8ZdQely9fxtvbG2tra6qqqqTlSqUSCwsLvXI6aty4cQwbNowNGzZw+fJlvdd0Ld7Kysouy8NdWlqKtbV1p5Xf9LrpRvU9PT2pqKigurpaGsAS7g5d3ufYmXSjgP7+/nrLm46YnzhxAmdnZzw9PaVlcrmc8ePHU1FRwW+//daucnWt0I7cajs7O+v9PXbsWK5fv05mZibQ2JKD/9+iBOjTpw9KpVLvFyntqYNuJHzixIl6y0NDQwFue9CgV69eTJ8+nbS0NP73v/81ez0jI4Nr164xceJEveOyt7cnPDxcOmZjSk1NpXfv3owcOVJv+dixY+nXr1+r29882f6+++7D39+f7OxsvRTCR48excbGhuDgYFJTUzt9fqlgXPdUyzEjI4P09HRmzpyJlZUVOTk5eHp6EhgYqLdeXFwcarWauXPnkpiYSFlZGaNGjaJfv358+eWX7X4TX758mbq6OiZMmIC9vT2JiYl6gwQtiYiIQKlUkpeXh5eXF56ensTHx0ujtVqtlvDwcF566SWOHDmCra0tarUaExMTrKysWqzDrboHMjMzSUlJYeLEiSgUCrKyshg0aBAjR44kMTGxxXl+bfHss89ibm7O2bNnm5373377jaKiInbs2METTzzB66+/zuHDh5HJZIwdO5aGhgbi4+NbnSd6u/bu3cuIESN47rnnGDx4MHl5ebi5ueHn58d3331HdnY2999/P++88w7x8fHNbr9feuklEhMTqampISgoiPvvv59NmzbprZOens7Vq1e57777xC31XeieCo4An3/+OU888QRhYWEAnDp1il27djF16lRpnWvXrrFs2TIeeeQRAgICMDc3Jycnh1WrVnVoDlpVVRXbt28nPDyc8ePHk5ycrNciako3eRpg69atUgCvq6sjPj5eml4DjaOsq1evZurUqWg0GoqLi9m4cSMDBw5k4MCB0i9emtbhwIEDeuU09c0331BQUEBQUBA+Pj4UFRWxefNm4uLi2n38TQ0YMACAadOmNXtt7dq1FBUVSV8AEydOJCIigpqaGtLS0ti2bRvV1dXY2tre8rfaDQ0Nei20jqiurubDDz9k2rRpjBgxgvvuu4/8/Hz+/e9/S9O2ZDIZMpnMYGt8w4YNhIeH06dPH8rLy9m4cWOzAFhXV0dqaqo07Uu4u5j4+fkZ7QexK1aswMvLq8Pbp6amtmueY3cxcOBA3njjjVu+npSUhFar5cUXX2TlypVd1u/WnX3xxRe3/IIpLCzk9ddfv8M1ajRjxgxCQ0OZM2dOm/p9FyxYwNWrV/n888/vQO2EznTPtRy7g/z8fINTRHRKSkpanS7zZ/fpp5/esu+xpcG27sTBwYGhQ4eydu3arq6K0AEiOBrB1atXW20NiuDYstOnT3d1FTqsZ8+eBAUF4e/vz5UrV6QBMOHuck+NVgtCd6BQKNBoNJSXl/PZZ5+JUeq7lOhzFARBMMCoLcfbHVG83e0FQRA6yqjBcdmyZSQnJ3do2+TkZJYtW9bJNRIEQWgbo95WC4Ig3K3EgIwgCIIBIjgKgiAYIIKjIAiCASI4CoIgGCCCoyAIggEiOAqCIBgggqMgCIIBIjgKgiAYYNSn8tjY2LBo0SKCgoJaXK+iooKFCxeSlZVllHpMnDiR6dOn88orr1BaWmqUMgRBuLcYteUYFRXVamCsrKzE1taW5cuX4+rqaszqGI1CoWDKlCl6eUXuxjJuZdSoUXz22WdSEinBsICAAEJCQrq6Grf05JNP8o9//OOWT1jvbOHh4bf14JmuZtTg2Jb8wzt37iQ2NvauDpAKhYLJkycbPTgau4xbuX79OlVVVSLnciv8/f27dXC8evUqVVVVnZZ6tzXh4eEMHz78jpRlDN3iYbdLly4FQKPRsHz5cqPeYt/rdDllOtPJkyfFA1vvYjKZjIaGBnbs2NHVVbmrdPnzHDds2MD69esBePXVV9FoNLfVB2llZcXMmTPx9fWlvr4erVZLeXk5kyZN0utztLKyIiIiAh8fH6ysrLh06RK7d+/m119/lfY1Y8YMgoODef/993n88cdxdXWlvLychIQEYmNcoPQhAAAgAElEQVRjAVi1alWzFvL8+fOpr69nzZo1/Pjjjzg4OODh4YG5uTmnT59m06ZNUu5mAFtbWyIiIlCpVFhaWlJQUMCePXukFKmtlREfH49SqcTDw4MtW7aQkJAANH7ZjBkzBqVSSWlpKfv37ycmJkbah5eXFy+++CIrVqxArVbj7e0t5Vv+/vvvpQRdISEhzJo1i7///e/k5+fz/PPPM2LECIPnf82aNVKiKV2K1iFDhtDQ0EBqaipbt26loqICgJkzZ/KXv/yFVatW8eijj+Ls7ExlZSVxcXHs27ePSZMmMXr0aKytrbl06RKbN2/mwoULeuVpNBqCg4NRKBQUFhby008/8csvv7TlrSLx9/cnNDQUR0dHSktLSU5OJjo6Wq+l7Ovry0MPPYSzszNVVVWkpaWxY8cOKisrpfNzs7i4ODZv3tym8wBgbW3NzJkz8fb25saNG5w8eZILFy7w5JNP8uqrr0q5xNv6vtVoNHzzzTdMnToVMzMzXn75ZZ5//nkGDRqk94zU++67j2nTpuHr64uFhQXnz59n27ZtehkoLSwsmDJlCiNGjMDa2lo6z7fKqGjo/fGvf/1L+oINDg5m3LhxKJVKysrK9M73jBkzGDduHP/3f/+nl4Hy6aefxtfXl//7v/9rMXFcZzJ1dnZ+x1g7Dw0NbZYzuqnU1FRSU1OBxseU9erVi2HDhhEcHMzRo0fbPYAyf/58fH19OX78OMeOHaNfv36o1WpkMhmxsbHU1NRgZmbG4sWLcXd359ChQ6SmptK3b180Gg15eXnk5eUB4O7uzuDBg/H19SUrK4sTJ07g4OBAYGAgWVlZXLlyhZycHHJzcxk2bBjR0dFER0eTn5+PTCYjLCyMwYMHk5mZyU8//URubi6jR49GpVKRlJREQ0MDcrmcN954AxcXFxITEzl58iS9e/cmNDSUs2fPUlRU1GoZLi4ulJWVceDAATIzMykrKyMyMpKwsDDOnDnDoUOHMDc3Z/z48djY2EgZFp2cnFCr1Xh4eFBdXc3hw4e5ceMGfn5+mJiYSEnoXVxcGD58OAkJCVRWVlJVVUVmZqZ07VJTU3FxccHExIRdu3ZRXV2No6Mjr7/+OqampsTHx5OXl0dAQACenp4cPHiQGzdu4OHhwaBBg/D09OTkyZOcOXOGXr164efnx5AhQxg8eDAHDx6UUuyOGDGCxMREqWU8a9YsJk2axPHjxzl69ChKpZLQ0FByc3PJz89v0/tl/PjxPPXUU+Tk5JCUlMS1a9fQaDQ88MAD0vvSz8+POXPmcOnSJfbt20dJSQmjR4/G1dWVQ4cOUVFRIdW9vr6er776itOnT1NZWdmm8wDw4osv4uPjw4kTJzh+/Dj9+vXjL3/5C6ampuzbt4/q6up2vW/d3NwYNGgQhw8fRqvVcuHCBUaOHIlSqZS+IE1NTXnttdcYNGgQiYmJnD59mmHDhhESEsIvv/wiBac5c+agVqtJTk7m5MmTODo68tBDD3Hp0iWD5/nKlSukpqYycuRI0tPT2bJlC1lZWdTW1jJ+/HhmzZrFqVOnOHToECYmJkyYMAFra2vS0tK4evUqwcHB5ObmcunSJQBMTEyIjIzk1KlT7f7iux3d4rb6ZjffYs+bN69dTwJ3c3PD3d2dn3/+WbqF2Lt3L/Pnz0elUknrjRs3jr59+/Lpp59KH4CEhAQWL17MrFmz0Gq11NXVAY0XZtu2bVK6zuTkZJYvXy6l2zx16pSU8Ck7O1van1zeeGqPHDki5TM+deoUVVVVPP3004waNYpDhw7h5uaGhYUF69atk7ZNSUlh5cqVqNVqMjIyWi0jIyODTz75RDq+AQMGMGbMGGJiYqQ0r7Gxsfz1r39l3LhxJCcn67UMtFot//nPf6S/3377bby9vdm5c6fB83zmzBkpcAKMHTuW+++/n2+//ZYrV64Aja2Xmpoa3n//fanuFy9eZO7cuXh5eUmtYplMxurVqzl//rx0fpctW0b//v1ZvHixlL/78uXL/O1vf8PNzY3Tp0/Tv39/QkJC+O6774iPjwdg3759vPXWW4SHh7cpT7SlpSWPPPIIWq2Wzz77TFpeVlbG5MmT2b17N6WlpYwcOZLTp0/zz3/+U1qnoaGBhx9+GHt7e4qKiigqKmL8+PGYmZlJ16et58HNzY1hw4bpvW+jo6OZO3cuvr6+0r7a876FxjTFGRkZtzz+kJAQnJ2dee+998jOzgYa368ffPABEyZM4Pvvv8fS0hJvb2/27t3L9u3bAdi/fz8fffQRo0ePlq7jzX7//Xd+//13bty4QXFxsd75CAgI4Pz583z11VcAxMfHY2ZmRmBgIJs3b+b8+fMUFxfj7e3N4cOHAXB1dcXOzo6jR4+2eD07W5fPcxw+fDiRkZF6//Lz8/VuOdrKw8MDQPqw6Jw7d07vb19fX/Ly8vQuWm1tLTExMdjZ2TF48GC99Y8dOyb9v6KigrKyMhQKRZvqdPPtM8Dhw4epq6tjyJAhQGNgi4qK0qtLTU0NJSUl9OjRo01lNL3V1N3SREdH6y3XJab38fHRW37z8UFjruy2Hp+DgwPTp08nPT2dxMREAMzNzfHw8CA1NRULCwvs7Oyws7Pj/Pnz1NfX4+bmprePmwN1aWkp5eXlnD9/XgqMujoB0jnx8fGhoaGBtLQ0af+2trZkZGTQt29fLC0tW637sGHDsLCwkOqtk5SUxI8//ih9+Xz22WcsX75cb53c3FwAlErlLfff1vPg7u4OIHWH6DTtVmrv+7a1bilfX19yc3MpKyuT6lZfX09OTo5Ut4aGBq5fv87AgQOxtrYGGvNxv/rqq3pfKG1VU1NDjx496N27t7Rs7dq1vPzyy1I3xvHjx/Hw8JBS83p5eVFTU3PHUxh3ectRpVLptepuh1Kp5Nq1a5SVlbW4nqOjo8FvVN1tSWuZAevq6gwmem+L+vp6SktLeeCBB6RlPXv2JCwsjCFDhqBQKKQPZdPA2lYODg5UVVU1+4IpLS3l6tWrrR5ffX19m45PJpPx7LPPUldXp9fyVCqVyOVyQkJCDI7etjaLQfeBbFonXZnQeIwymYwPP/zQ4D5sbGyoqalpsRxdYNO1dnVKS0vZtWuX9LeJiQkBAQEEBwfj4ODQplkYuv235TwolUpqa2tb7UK63fdtUw4ODtx///0G0wgXFxcDjcF3/fr1REZGsmLFCjIzM/n11185dOhQh9KYfP/998yfP59//OMf5OTkkJaWxqFDh/Ruz3/55Rc0Gg3Dhg0jNTUVlUpFWlraHetr1Ony4NiZzMzM2pRo3cTExOCIri4gGHvKipmZmXT7o1AoePPNNykpKWH37t0UFhZSW1vLCy+80OH9y2Qyvdurpq911lSOSZMm4erqytq1a/U+2LrzmJiYiFarbbZda19ebSGXy6mpqeHzzz83+PrNrc7WtHY+ZsyYwfjx44mLi+OHH36gqqqKoUOH8uijj7a4XVvPQ319fZvec539vpXL5WRmZkp3FDe7+f1z+PBh0tLS8PX1xcPDgylTphAeHs6//vUvfvvtt3aVmZOTw+LFi1GpVHh4eBAYGMhDDz3E9u3bpb7QCxcuUFxcjEqlIi8vj969e+t9Wd0p91RwLC4uxsbGBgsLixYTv1+5ckWvWa/Tq1cvaT/GYmlpia2trVSGv78/1tbWLFmyRBqRBNoU5G/l8uXLeHt7Y21tTVVVlbRcqVRiYWGhV05H9e/fn/DwcI4cOdKsk/zm4GesW6HS0lIsLS35/fffO9QFA/+/Zd6zZ08uX74sLbewsKBv377k5+dLAwQpKSlS/y3QpgnxbT0PxcXF0vuipWPp7PdtaWkp1tbWLdbNysqK+++/n8uXL5OUlERSUhJ2dna8+eabTJ8+nffff7/N5cnlcnr27El5eTnHjx/n+PHjyOVy5s2bx7Rp00hKSuLq1atAY1ePv78/+fn5XLt2TW80/k7p8j7HzqQbhfX399db3nTE/MSJEzg7O+Pp6Sktk8vljB8/noqKinZ/G+q+zQ3dijadtB0cHIypqak0oKG7TdT1r+i2USqVer9kaKmMpnRTJiZOnKi3PDQ0FMBgJ3p7mJub8+yzz1JRUcG3337b7PXKykrOnTtHQECA3q2eTCZj6tSpUt/V7dD1u02ZMkVv+YMPPoifn1+b9pGRkUFtbS1jxozRWz5mzBgWL16MnZ0dJiYmyGQyzM3N9dbR9W83vUY3X5+2ngfd+3bChAl6ZTTtQ+zs921qaiq9e/dm5MiResvHjh1Lv379ABg6dCjvvfee3sDQH3/8QVFRUbNz0lTT83HfffexZMkSpk+fLi2rq6vj4sWLmJqa6q179OhR7OzsmDhxYpfcUsM91nLMyMggPT2dmTNnYmVlJU0BCQwM1FsvLi4OtVrN3LlzSUxMpKysjFGjRtGvXz++/PLLdk+ivnz5MnV1dUyYMAF7e3sSExOlWxxPT09efvllUlNT6dOnD2PGjOHSpUtSANNqtYSHh/PSSy9x5MgRbG1tUavVmJiYYGVl1aYymsrMzCQlJYWJEyeiUCjIyspi0KBBjBw5ksTERL0BkI6YOnUqvXv3Jjk5GW9vb73XCgoKyMrKYvPmzURFRfH666+zf/9+KioqUKvVDBgwAK1Wq9ei7YhTp05x7Ngxxo4di729PadOnUKhUBASEkJ6ero00tmSqqoqdu3axaOPPsrLL79MWloajo6OjBkzRq8fTKvVMmLECJ577jkuX76Mp6cnzs7OAHrXKC8vD09PT6ZPny5NdWrLefj999/59ddfCQsLo2fPnuTk5ODu7s6gQYP06tvZ79u9e/dKxzV48GDy8vJwc3PDz8+P7777juzsbLRaLXl5eURGRtK3b1+Ki4sZNGgQgwcPlkavAV577TVqamr0RvTz8/NRqVQ8/PDDaLVaLl26xMGDBxk9ejSmpqZcuHCBHj16MG7cONLS0vT6MC9cuEBRURE9evRoNmB4p9xTwREapy888cQThIWFAY0fol27djF16lRpnWvXrrFs2TIeeeQRAgICMDc3Jycnh1WrVknf4u1RVVXF9u3bCQ8PZ/z48Rw4cED6pktMTEQulxMREYGZmRm//vqrXmsrJyeH1atXM3XqVDQaDcXFxWzcuJGBAwcycOBA6RcvLZVhyDfffENBQQFBQUH4+PhQVFTE5s2biYuLa/fxNTVgwAAAAgMDm33x7N+/n6ysLC5cuMBHH31EREQEGo0GgPPnz/PRRx9x8eLF264DNI5yXrp0icDAQDw8PCgvLycmJoY9e/Ygk8laHDipra2lpqaGvXv3UlVVhUajYcaMGZSXl7N7925+/vlnad3169dTXV0tDR5qtVq2bdvGzJkzsbOzk9aLi4vD1dWVkJAQabJ3W8/DV199Jf14QTfvc+/evYSFhUl9op39vq2urubDDz9k2rRpjBgxgvvuu4/8/Hz+/e9/S1PX6uvr+fjjj4mIiCAwMJD77ruPoqIivSlUoN+PrrN161b+9re/8dBDD5GTk8OlS5fYuHEjxcXFBAQE4OfnR3l5Ofv37+fHH39sVr9jx44REhKiNzp/J3X5L2Rakpqa2q55jt2JhYUFa9as0Zu7Jtw5AwcO5I033rjl60lJSdIvs7qrGTNmEBoayvz58/V+LSLcGfdcy1EQoPGWztAUFZ2OTpMyloiICGJjY6XuBjMzM3x9fSkoKBCBsYuI4Cjck65evXrHJw13lKWlJYGBgfj5+ZGYmEhdXR0BAQH06NGDtWvXdnX1/rREcBSELlZTU8N7773H1KlTCQ0NxcrKitzcXL2HeAh3ngiORnLt2jWeeeaZrq6GcJcoKyvT+5WR0PWMOs+xIz8v6sztBUEQOsqowXHZsmUkJyd3aFvd01kEQRC6glGn8giCINyt7qmfDwqCIHQWERwFQRAMEMFREATBABEcBUEQDBDBURAEwQARHAVBEAwQwVEQBMEAERwFQRAMMOpvq21sbFi0aBFBQUEtrldRUcHChQtbTSXZURMnTmT69Om88sorrWZ4E4SOWrVqFadPnxZP0rlHGLXlGBUV1WpgrKysxNbWluXLl+Pq6mrM6hiNQqFgypQpzfLF3G1l3MqoUaP47LPP2pRUSmhZQECAwTStQvdj1ODYlvy+O3fuJDY29q4OkAqFgsmTJxs9OBq7jFu5fv06VVVVRk9Z+2fg7+8vguNdolv0OS5duvSuD5DdRVuyE7bXyZMnee2117hy5Uqn71sQuqtu8zzHpUuXAqDRaFi+fHmH+yCtrKykREX19fVotVrKy8sNrhcREYGPjw9WVlZcunSJ3bt36+XHnTFjBsHBwbz//vs8/vjjuLq6Ul5eTkJCArGxsUBjP5OuhTx79mxmz57N/Pnzqa+vZ82aNfz44484ODjg4eGBubk5p0+fZtOmTXqP6be1tSUiIgKVSoWlpSUFBQXs2bNHSqHaWhnx8fEolUo8PDzYsmULCQkJ0rkcM2YMSqWS0tJS9u/fLyVOB/Dy8uLFF19kxYoVqNVqvL29qaurQ6vV8v3330sJvEJCQpg1axZ///vfyc/P5/nnn2fEiBEGz//ND2jt1asX06dPZ8iQIVLCqa1bt0q5mWfOnMlf/vIXVq1axaOPPoqzszOVlZXExcWxb98+Jk2axOjRo7G2tubSpUts3ryZCxcu6JWn0WgIDg5GoVBQWFjITz/91CyP9q3o8vy05RqZmpoSHh5OQEAANjY2FBQU8NNPP7X5YbS6c6jz9ddfExcXx+bNm4HGlLJTp06lb9++0lPMd+7cyR9//CFdp1WrVum9PwMCAnjmmWf46KOPyMzMbFM9hLYzdXZ2fsdYOw8NDW2WM7qp1NRUKbtYcnIyvXr1YtiwYQQHB3P06NF2D6DMnz8fX19fjh8/zrFjx+jXrx9qtRqZTEZsbCw1NTWYmZmxePFi3N3dOXToEKmpqfTt2xeNRkNeXh55eXkAuLu7M3jwYHx9fcnKyuLEiRM4ODgQGBhIVlYWV65cIScnh9zcXIYNG0Z0dDTR0dHk5+cjk8kICwtj8ODBZGZm8tNPP5Gbm8vo0aNRqVQkJSXR0NCAXC7njTfewMXFhcTERE6ePEnv3r0JDQ3l7NmzFBUVtVqGi4sLZWVlHDhwgMzMTMrKyoiMjCQsLIwzZ85w6NAhzM3NGT9+PDY2NlKmOicnJ9RqNR4eHlRXV3P48GFu3LiBn58fJiYmUm5tFxcXhg8fTkJCApWVlVRVVUmpR3X/XFxcMDExYdeuXVRXV+Po6Mjrr7+Oqakp8fHx5OXlERAQgKenJwcPHuTGjRt4eHgwaNAgKdvemTNn6NWrF35+fgwZMoTBgwdz8OBBKcXuiBEjSExMlFKQzpo1i0mTJnH8+HGOHj2KUqkkNDSU3NxcKa1qS+RyeZuuEcDzzz/P6NGjpbLs7e0JCwujoKCA3NxcoHHg78qVKwYDZkVFhXR89fX1fPXVV5w+fZrKykoGDBjAokWL+OOPP4iPj6eoqIigoCA8PDw4cOAARUVFjBs3jhs3buhl4gsPD8fKykoKsELn6jYtR52bW5Dz5s1rV/ZBNzc33N3d9TL+7d27l/nz56NSqaT1xo0bR9++ffn000+lN1tCQgKLFy9m1qxZaLVaKc2kiYkJ27Ztk1JVJicns3z5clQqFadOneLUqVNcu3YNgOzsbGl/cnnjqT1y5AibNm0CGtPEVlVV8fTTTzNq1CgOHTqEm5sbFhYWrFu3Tto2JSWFlStXolarycjIaLWMjIwMPvnkE+n4BgwYwJgxY4iJiWHr1q0AxMbG8te//pVx48aRnJysl7taq9XqPYX67bffxtvbm507dxo8z2fOnJECJzQmgb///vv59ttvpVvvGTNmUFNTw/vvvy/V/eLFi8ydOxcvLy+pVSyTyVi9ejXnz5+Xzu+yZcvo378/ixcv5o8//gAa83b/7W9/w83NjdOnT9O/f39CQkL0UoTu27ePt956i/Dw8HalF2jtGg0dOhRfX1+2bNki3THExcURFRVFREREm1qqRUVFFBUVMX78eMzMzPSCnO7Le8WKFdTU1ABQWlrKE088wYABAzh//jxarRaVSsWGDRu4ceMGpqamuLu7k5ycLKVuFTpXl/c5Dh8+nMjISL1/+fn50q1Xe3h4eADo5dMFOHfunN7fvr6+5OXl6b1Ba2triYmJwc7OjsGDB+utf3NS8YqKCsrKylAoFG2qU9Msd4cPH6auro4hQ4YAjYEtKipKry41NTWUlJTQo0ePNpXR9FZTd8sbHR2tt3zPnj0A+Pj46C1vmjQ9Jyenzcfn4ODA9OnTSU9PJzExEQBzc3M8PDxITU3FwsICOzs77OzsOH/+PPX19bi5uent4+ZAXVpaSnl5OefPn5cCo65OgHROfHx8aGhoIC0tTdq/ra0tGRkZ9O3bF0tLyzbVH1q/RiqVioaGBvbv36+33g8//EBKSgoy2e19jGpqapDJZLi7u0vLEhISmDNnjnRtjx07hp2dndQfP3ToUCwtLbss4f2fQZe3HHWJ0juDUqnk2rVrlJWVtbieo6MjGRkZzZbrbqd79uzZ4vZ1dXUdHvior6+ntLSUBx54QFrWs2dPwsLCGDJkCAqFQmoRdjR9qIODA1VVVc2+YEpLS7l69Wqrx1dfX9+m45PJZDz77LPU1dXptTyVSiVyuZyQkBCDI7OtzWJoaGjg+vXrzeqkKxMaj1Emk/Hhhx8a3IeNjY3UCmuvptdIqVTyxx9/SH2wOpmZmZ3S1xcXF8eQIUOYO3cuJSUlpKWlcfToUb3WeXp6OtXV1fj4+HDu3Dm8vLwoLy8XfY1G1OXBsTOZmZk1+1AZYmJiIn3YbqYLCMaesmJmZibdtisUCt58801KSkrYvXs3hYWF1NbW8sILL3R4/zKZTNq/odc66zZs0qRJuLq6snbtWr2+Yd15TExMRKvVNtuutS+vtpDL5dTU1PD5558bfP3mVmdH3HyNTExMjHrrWl1dzbJlyxg0aBBeXl4MHz6cMWPGcPz4cb744gsaGhqkwUWVSsXWrVvx8vLi+PHj4pbaiO6p4FhcXIyNjQ0WFhZSP5chV65coXfv3s2W9+rVS9qPsVhaWmJrayuV4e/vj7W1NUuWLKGoqEhary1B/lYuX76Mt7c31tbWUpJ4aGwBWVhY6JXTUf379yc8PJwjR44063O7OfgZK3d0aWkplpaW/P777x3qgmlJ02tUXFzM8OHDMTc312s92tvbo1QqOX/+/G19oTo4OFBbWyu1RLdv387DDz9MREQE7u7u0gDasWPHCAgIwN/fH6VSKdK2GlmX9zl2Jt2byN/fX2950xHzEydO4OzsjKenp7RMLpczfvx4Kioq+O2339pVrq4VauhWtOmk7eDgYExNTaVbJt1topmZmd42SqUSExOTNpXR1MmTJ4HG0dObhYaGAkiDIR1lbm7Os88+S0VFBd9++22z1ysrKzl37hwBAQF6t/AymYypU6dibW19W+UDUh/tlClT9JY/+OCD+Pn5tWtfrV2jtLQ0ZDIZo0eP1lvvySefZO7cue0KjIa6LObNm8fChQv1+i51/eQ3vy90t9YzZsygvLy83e9ToX3uqZZjRkYG6enpzJw5EysrK2kKSGBgoN56cXFxqNVq5s6dS2JiImVlZYwaNYp+/frx5ZdfGrzlbsnly5epq6tjwoQJ2Nvbk5iYKH1gPD09efnll0lNTaVPnz6MGTOGS5cuSQFMq9USHh7OSy+9xJEjR7C1tUWtVmNiYoKVlVWbymgqMzOTlJQUJk6ciEKhICsri0GDBjFy5EgSExP1BkA6YurUqfTu3Zvk5GS8vb31XisoKCArK4vNmzcTFRXF66+/zv79+6moqECtVjNgwAC0Wq1ei7YjTp06xbFjxxg7diz29vacOnUKhUJBSEgI6enpHD58uM37au0apaWlkZaWxmOPPYaTkxO5ubl4eHigUqn45ptvbrnfadOmoVareffdd6XjzcvLw9PTk+nTp0vToaKjo3nuuedYsGABJ06cwMLCgrFjx1JaWqoXAOvr6zl58iSBgYEkJiaKW2oju6eCI8Dnn3/OE088QVhYGND4Idq1axdTp06V1rl27RrLli3jkUceISAgAHNzc3Jycli1apXU+myPqqoqtm/fTnh4OOPHj+fAgQPS7VdiYiJyuZyIiAjMzMz49ddf9VpbOTk5rF69mqlTp6LRaCguLmbjxo0MHDiQgQMHYmpqSn19fYtlGPLNN99QUFBAUFAQPj4+FBUVsXnzZuLi4tp9fE0NGDAAgMDAwGZfPPv37ycrK4sLFy7w0UcfERERgUajAeD8+fN89NFHXLx48bbrALB27VouXbpEYGAgHh4elJeXExMTw549e5DJZC0O/NTW1krBpbVrBI2T2ydPnoyfnx9BQUHk5+fzxRdfcPTo0VuWIZfLkcvlei3CuLg4XF1dCQkJkSbGHz58mPr6ekJDQ3n00Uepqanh7Nmz7Ny5s9mXyLFjxwgMDBSj1HeAUVOzrlixAi8vrw5vn5qa2q55jt2J7tcXN8+5FO6cgQMH8sYbb9zy9aSkJDZv3iyukXBL91zLURAA8vPzWbly5S1f7+g0KeHPQwRH4Z6k+31ySywsLO5QbYS70T01Wi0IgtBZRMvRSK5du8YzzzzT1dUQWiCukdASo7YcKysru3R7QRCEjjJqcFy2bBnJyckd2lb3dBZBEISuYNSpPIIgCHcrMSAjCIJggAiOgiAIBojgKAiCYIAIjoIgCAaI4CgIgmCACI6CIAgGiOAoCIJggAiOgiAIBhj1t9U2NjYsWrSIoKCgFterqKhg4cKFZGVlGbM6napPnz4sWbKEjRs3NkvZKQjC3c+oLceoqKhWA2NlZSW2trYsX75cyskrNGYlnDJlSrP8JkuWLCEyMrKLaiUIfx5GDY6t5ScG2LlzJ7GxsSJANqFQKJg8eXKz4FhVVdViZvL9jEsAAAlhSURBVEVBEDpHt+hzXLp0qQiQbbR06VK2bNnS1dUQhHtet3me49KlSwHQaDQsX778tvogNRoNwcHBKBQKCgsL+emnn6Tcym+99RYNDQ289957ett89NFHFBUV8cknnwDg6+vLQw89hLOzM1VVVaSlpbFjx45bPkbNzs6OlStX8uOPP7Jr1y5p+ejRo3nqqad45513yMnJAcDW1paIiAhUKhWWlpYUFBSwZ88eKWXqqlWrpFb37NmzmT17NvPnz6e6uppVq1Zx+vRp1q5dK5XRq1cvpk2bxuDBgzE1NSUjI4Pt27eTn58vrfPmm2/yxx9/cPDgQSZPnoyDgwMFBQVs27aN06dP3/Jczpgxg8DAQDZt2sTDDz+Mo6MjZWVl7N27l8TERAA+/vhjqqurefvtt6XtLCws+PTTTzl27BhfffVVK1dMELqfbtFy1OmMFuSsWbOYPn06Z86cYceOHVRVVTF79mx8fX0BOHr0KC4uLtjb20vbODs707NnTymjm5+fH3PnzqWqqorNmzeTkpKCn58fc+bMue1jlMvlREVF4evry/79+9m2bRvXr19n7ty5PPjggwB8+eWXbN++HYDo6Gg+/fTTW95K9+rVi9dff53+/fuzb98+4uLicHFx4bXXXsPBwUFvXVdXV5588km0Wi179+7lgQceYN68ea3mkbaxsWHq1KlER0fzxRdfcPnyZZ588kkCAgIAOH78uHQOddzd3ZHL5SLxvHDX6lbBEfQD5Lx589q1bf/+/QkJCWHLli1s3LiRuLg4Pv74Y7KzswkPDweQAuDN+Za9vLyor6+XWm4jR47k9OnT/POf/+R///sfO3bsIDY2lqFDh+oF1Y5wc3PDwsKCdevW8eOPP5KYmMjKlSu5fv06arUaaEwnm5mZCUB2djapqam3zKU9Y8YMZDIZH3zwAT/++CP//e9/+fDDDzE3N+exxx7TW9fMzIylS5fyww8/sGvXLjZs2IClpSVDhw5ttd6rVq0iJSWFkydPsnLlSgoLC1s8p56entTW1raax0UQuqsuv60ePnx4s9HX/Px8Kioq2r0vHx8fGhoaSEtLw87OTlqekZGBRqPB0tKSK1eu8Pvvv+Pt7U1CQgIAKpWKs2fP8scffwDw2WefNdt3bm4uAEqlktLS0nbX7ea6REVF6S2rqamhpKSEHj16tGtflpaWuLu7k5SUpFenK1eucPjwYSknty6/9aVLlygoKJDW093mKxSKVsu6OVtfQ0MDhw8fZsqUKSgUCs6dO0dJSQk+Pj7ExsYCjcExLS2txdzagtCddXlwVKlUqFSqTtmXg4MDMpmMDz/80ODrNjY21NTU8Msvv/DII49gZWWFqakpLi4ubNq0SVrPxMSEgIAAgoODcXBwaNOoe3v0/H/t3U1IVF8cxvHv3JmJXrBhbkWUESHIZJmJCTk0gZUaEUNBGa2DoEUgtIsIolW4iDYRLUuIUqlFbiJCKMTCkUbSXijoTcLM0WbhNOQd/S9ihm4eLVPzrzyfnee+nDt3mIffOefKXbWK/fv3EwqFsG0bn+/H1zDV14WuXLkSr9frmlvM+vTpEz6fD9u2XYH4s2w16vV6p/gJYGBgAIBgMMjg4CCdnZ3s2bOHvLw8bNsmGAzmKnGR+WjOw3Em+Xw+0uk0V65cMW7PVoadnZ3U1tZSUlKC3+9nbGwsNzSEH0PVqqoq7t+/z507dxgeHqaoqIja2tppX6Nt25w9e5bBwUHu3r3L58+f+f79OydPnpzyuTweD4BxyJ0N3NHR0eld8ASy53ccB/gxtK6urqa0tJRAIIDjOHR1dc1K3yL/woIKx6GhIRYvXsz79+8nHZZ/+fKFd+/eUVZWhmVZvH79Ore/ZVlUVlbS3t5OY2Nj7pjVq1dP2nc2JCxr8mnccDjMsmXLOH/+fK76AhgZGfnt5/tVIpFgbGyMtWvXjtu2Zs0aHMfh69evUz7vn8jej0QiAZAbWpeWlrJ8+XJevHjBt2/fZqVvkX/hf7cgMx3ZSuXAgQOu9o0bN1JRUeFqi8VibN68mU2bNtHR0ZFr93g8WJbFokWLXPsXFxfntpukUinS6TTr1693tRcWFrr+zoan3+/PteXn57NixQrXuf9kyJtKpXj58iXhcJhAIJBrt22b8vJyuru7Z2zO7+eH0ZcsWUI4HObDhw+uR5uy93TDhg1apZZ5b0FVjj09PcRiMXbt2kUwGKSnpwfbttm9ezfd3d08fvw4t29HRweHDx9mdHTUNTeWyWSIx+OUl5dz/Phx+vv72bJlC+vWrQNg6dKlE/Yfi8WIRCIcO3aMvr4+ioqKCIVCrn3i8TjRaJS6ujqePHlCXl4e27dvx+PxuM7d39+P4zhUV1cTDAZpbW01VmI3b97k9OnTnDlzhocPHwJQWVmJ4zg0NTVN6f4FAgHOnTvHgwcPaGlpcW2rq6ujtbWVdDpNJBIhEAi45mnhx3RFTU0NmUyGp0+fTqlvkf+bBRWOAFevXqW3t5cdO3ZQXFxMMpnk3r17437sAwMDvH37lpGREZLJpGvbtWvXSKVSucWieDxOU1MTR48eda2C/+rWrVt4vV5KSkrYtm0bz58/zx2X9fHjRy5fvszBgwepqakhkUjQ0NBAQUEBBQUFeL1eMpkMw8PDNDc3E41Gqaqq4tGjR8Y+e3t7uXDhAocOHWLfvn1kMhlevXrF7du3J1yImYhlWViWZaxWr1+/TjQaJT8/n2QySUNDw7jq8M2bNwwNDdHX16d3jsu8N6uvZr148SJbt2796+O7uro4derUDF6RTNWRI0fYu3cvJ06c+O28qN/v59KlSzQ3N+f+e0ZkvlpQc44ytyoqKvD7/XqERxaEBTesln8vFApRVlZGJBKhvb193DSFyHykylGmrbCwkJ07d/Ls2TNu3Lgx15cjMiNUOcqkGhsbXc97mrS0tIxb8BKZ72a1cpzuiqVWPEVkrsxqONbX19PW1vZXx7a1tVFfXz/DVyQi8mdm9VEeEZH5SgsyIiIGCkcREQOFo4iIgcJRRMRA4SgiYqBwFBExUDiKiBgoHEVEDBSOIiIGCkcREQOFo4iIgcJRRMRA4SgiYqBwFBExUDiKiBgoHEVEDBSOIiIGCkcREQOFo4iIgcJRRMRA4SgiYqBwFBExUDiKiBgoHEVEDBSOIiIGCkcREQOFo4iIgcJRRMRA4SgiYqBwFBExUDiKiBj8B9aUm7Ed9n4EAAAAAElFTkSuQmCC)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uGXgQ5c1TGj9"
      },
      "source": [
        "from dont_patronize_me import DontPatronizeMe\n",
        "# Initialize a dpm (Don't Patronize Me) object.\n",
        "# It takes two areguments as input: \n",
        "# (1) Path to the directory containing the training set files, which is the root directory of this notebook.\n",
        "# (2) Path to the test set, which will be released when the evaluation phase begins. In this example, \n",
        "# we use the dataset for Subtask 1, which the code will load without labels.\n",
        "dpm = DontPatronizeMe('.', 'dontpatronizeme_pcl.tsv')"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zxkWFeZR8liD"
      },
      "source": [
        "# Load Subtask 1 data\n",
        "\n",
        "Note that the original dataset contains fine-grained tags (0 to 4, both inclusive). The interface we provide collapses them into binary labels, which is the labeling we use for the task."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 198
        },
        "id": "xBNfZFPq6VgX",
        "outputId": "fd3cfd7d-217f-4c10-92b7-c814e868aa88"
      },
      "source": [
        "# This method loads the subtask 1 data\n",
        "dpm.load_task1()\n",
        "# which we can then access as a dataframe\n",
        "dpm.train_task1_df.head()"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>par_id</th>\n",
              "      <th>art_id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>country</th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "      <th>orig_label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>@@24942188</td>\n",
              "      <td>hopeless</td>\n",
              "      <td>ph</td>\n",
              "      <td>we 're living in times of absolute insanity , ...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>@@21968160</td>\n",
              "      <td>migrant</td>\n",
              "      <td>gh</td>\n",
              "      <td>in libya today , there are countless number of...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>@@16584954</td>\n",
              "      <td>immigrant</td>\n",
              "      <td>ie</td>\n",
              "      <td>\"white house press secretary sean spicer said ...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>@@7811231</td>\n",
              "      <td>disabled</td>\n",
              "      <td>nz</td>\n",
              "      <td>council customers only signs would be displaye...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>@@1494111</td>\n",
              "      <td>refugee</td>\n",
              "      <td>ca</td>\n",
              "      <td>\"\"\" just like we received migrants fleeing el ...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  par_id      art_id  ... label orig_label\n",
              "0      1  @@24942188  ...     0          0\n",
              "1      2  @@21968160  ...     0          0\n",
              "2      3  @@16584954  ...     0          0\n",
              "3      4   @@7811231  ...     0          0\n",
              "4      5   @@1494111  ...     0          0\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_IFlGsmx9RZe"
      },
      "source": [
        "# Load Subtask 2 data\n",
        "\n",
        "Similarly, you can use our data manager code to have a binary vector representing the labels of each paragraph in the multilabel classification subtask (PCL categories, Subtask 2)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 198
        },
        "id": "vfLpiw568iCQ",
        "outputId": "d1a4614f-7ae6-4c7c-d9f5-2d642ddaad71"
      },
      "source": [
        "# Same for subtask 2\n",
        "dpm.load_task2()\n",
        "# which we can then access as a dataframe\n",
        "dpm.train_task2_df.head()"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>par_id</th>\n",
              "      <th>art_id</th>\n",
              "      <th>text</th>\n",
              "      <th>keyword</th>\n",
              "      <th>country</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4046</td>\n",
              "      <td>@@14767805</td>\n",
              "      <td>we also know that they can benefit by receivin...</td>\n",
              "      <td>hopeless</td>\n",
              "      <td>us</td>\n",
              "      <td>[1, 1, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1279</td>\n",
              "      <td>@@7896098</td>\n",
              "      <td>pope francis washed and kissed the feet of mus...</td>\n",
              "      <td>refugee</td>\n",
              "      <td>ng</td>\n",
              "      <td>[0, 0, 1, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8330</td>\n",
              "      <td>@@17252299</td>\n",
              "      <td>many refugees do n't want to be resettled anyw...</td>\n",
              "      <td>refugee</td>\n",
              "      <td>ng</td>\n",
              "      <td>[0, 0, 0, 1, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4063</td>\n",
              "      <td>@@3002894</td>\n",
              "      <td>\"budding chefs , like \"\" fred \"\" , \"\" winston ...</td>\n",
              "      <td>in-need</td>\n",
              "      <td>ie</td>\n",
              "      <td>[1, 1, 0, 0, 1, 1, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4089</td>\n",
              "      <td>@@25597822</td>\n",
              "      <td>\"in a 90-degree view of his constituency , one...</td>\n",
              "      <td>homeless</td>\n",
              "      <td>pk</td>\n",
              "      <td>[0, 1, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  par_id      art_id  ... country                  label\n",
              "0   4046  @@14767805  ...      us  [1, 1, 0, 0, 0, 0, 0]\n",
              "1   1279   @@7896098  ...      ng  [0, 0, 1, 0, 0, 0, 0]\n",
              "2   8330  @@17252299  ...      ng  [0, 0, 0, 1, 0, 0, 0]\n",
              "3   4063   @@3002894  ...      ie  [1, 1, 0, 0, 1, 1, 0]\n",
              "4   4089  @@25597822  ...      pk  [0, 1, 0, 0, 0, 0, 0]\n",
              "\n",
              "[5 rows x 6 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pX5JzaId9rWm"
      },
      "source": [
        "# helper function to save predictions to an output file\n",
        "def labels2file(p, outf_path):\n",
        "\twith open(outf_path,'w') as outf:\n",
        "\t\tfor pi in p:\n",
        "\t\t\toutf.write(','.join([str(k) for k in pi])+'\\n')"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LyPF_Gtd9xT9"
      },
      "source": [
        "# Generate predictions\n",
        "\n",
        "In this example, our code will just generate a bunch of random predictions for both subtasks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kljiRAHxA1zh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e7f1607e-6614-46e9-b832-52854be25a1c"
      },
      "source": [
        "# first, we need to create the res/ and ref/ folders, which the evaluator expects\n",
        "!mkdir ref res"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "mkdir: cannot create directory ‘ref’: File exists\n",
            "mkdir: cannot create directory ‘res’: File exists\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qNhV7Raz-6my"
      },
      "source": [
        "dpm.load_test()"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V6S7L0BiT-yD"
      },
      "source": [
        "# random predictions for task 1\n",
        "preds_task1 = [[random.randint(0,1)] for k in range(0,len(dpm.test_set))]\n",
        "labels2file(preds_task1, os.path.join('res/', 'task1.txt'))"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lsFszPYdUIUO"
      },
      "source": [
        "# random predictions for task 2\n",
        "preds_task2 = [[random.choice([0,1]) for k in range(7)] for k in range(0,len(dpm.test_set))]\n",
        "labels2file(preds_task2, os.path.join('res/', 'task2.txt'))"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z63hEpsm_Qgn"
      },
      "source": [
        "# Evaluate predictions\n",
        "\n",
        "After generating two prediction files (`task1.txt` and `task2.txt`), we can just call the scorer (`evaluation.py`), which will generate a `scores.txt` file."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69wGJxmqDuVD"
      },
      "source": [
        "## Get gold labels\n",
        "\n",
        "We will use the gold labels of the training set and compare our random predictions to them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FVfa-UhvCa2v"
      },
      "source": [
        "# for subtask 1 (we convert our list of labels into a list of lists to make \n",
        "# it compatible with the labels2file function)\n",
        "labels2file(dpm.train_task1_df.label.apply(lambda x:[x]).tolist(), os.path.join('ref/', 'task1.txt'))\n",
        "# and for subtask 2\n",
        "labels2file(dpm.train_task2_df.label.tolist(), os.path.join('ref/', 'task2.txt'))"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h0DTuPlhE3Uz"
      },
      "source": [
        "## Call task scorer and print results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hcfPPtez__BU"
      },
      "source": [
        "# Now, we can just call the official scorer, which takes an input_directory and an output_directory\n",
        "# as arguments. In this example, both will be the root directory of this notebook.\n",
        "!python3 evaluation.py . ."
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P9Eu-lCVDE3O",
        "outputId": "c5f7c122-5f76-499f-8403-7d571b0512df"
      },
      "source": [
        "# The scorer generated a results file called \"scores.txt\". \n",
        "# We can now see the performance of a random baseline on the training set.\n",
        "!cat scores.txt"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "task1_precision:0.09203054630898766\n",
            "task1_recall:0.473313192346425\n",
            "task1_f1:0.1540983606557377\n",
            "task2_unb:0.301255230125523\n",
            "task2_sha:0.5839536807278742\n",
            "task2_pre:0.291350531107739\n",
            "task2_aut:0.2896174863387978\n",
            "task2_met:0.48283038501560877\n",
            "task2_com:0.2952243125904486\n",
            "task2_the:0.06177606177606177\n",
            "task2_avg:0.3294296696688647\n"
          ]
        }
      ]
    }
  ]
}