analysis_round2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DynaSent version 1: Round 2 analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = 'Christopher Potts'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Set-up](#Set-up)\n",
    "1. [Load the splits](#Load-the-splits)\n",
    "1. [Label distributions](#Label-distributions)\n",
    "1. [Relationship between validation labels and model predictions](#Relationship-between-validation-labels-and-model-predictions)\n",
    "1. [Adversarial rates](#Adversarial-rates)\n",
    "1. [Distribution of validation response distributions](#Distribution-of-validation-response-distributions)\n",
    "1. [Worker distribution](#Worker-distribution)\n",
    "1. [Estimates of human performance](#Estimates-of-human-performance)\n",
    "1. [Examples for the paper](#Examples-for-the-paper)\n",
    "1. [Random cases](#Random-cases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "\n",
    "import dynasent_utils as utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.style.use(\"dynasent.mplstyle\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "src_dirname = \"dynasent-v1.1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename_template = os.path.join(\n",
    "    src_dirname, \"dynasent-v1.1-round02-dynabench-{}.jsonl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_filename = filename_template.format(\"train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_filename = filename_template.format(\"dev\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_filename = filename_template.format(\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = utils.load_dataset(train_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'hit_ids': ['y22661'],\n",
       " 'sentence': \"We enjoyed our first and last meal in Toronto at Bombay Palace, and I can't think of a better way to book our journey.\",\n",
       " 'sentence_author': 'w250',\n",
       " 'has_prompt': True,\n",
       " 'prompt_data': {'indices_into_review_text': [2093, 2213],\n",
       "  'review_rating': 5,\n",
       "  'prompt_sentence': \"Our first and last meals in Toronto were enjoyed at Bombay Palace and I can't think of a better way to bookend our trip.\",\n",
       "  'review_id': 'Krm4kSIb06BDHternF4_pA'},\n",
       " 'model_1_label': 'positive',\n",
       " 'model_1_probs': {'negative': 0.29140257835388184,\n",
       "  'positive': 0.6788994669914246,\n",
       "  'neutral': 0.029697999358177185},\n",
       " 'text_id': 'd16069',\n",
       " 'label_distribution': {'positive': ['w43', 'w26', 'w155', 'w23'],\n",
       "  'negative': [],\n",
       "  'neutral': [],\n",
       "  'mixed': ['w174']},\n",
       " 'gold_label': 'positive'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ternary = utils.load_dataset(\n",
    "    train_filename,\n",
    "    labels=('positive', 'negative', 'neutral'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev = utils.load_dataset(dev_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = utils.load_dataset(test_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "positive        6518\n",
       "negative        5059\n",
       "mixed           3334\n",
       "neutral         2928\n",
       "No Majority     2136\n",
       "Total          19975\n",
       "Name: gold_label, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(train, dev, test, dist_labels=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "positive    32551\n",
       "negative    24994\n",
       "mixed       18765\n",
       "neutral     16365\n",
       "Total       92675\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(train, dist_labels=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "positive        6038\n",
       "negative        4579\n",
       "mixed           3334\n",
       "neutral         2448\n",
       "No Majority     2136\n",
       "Total          18535\n",
       "Name: gold_label, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(train, dist_labels=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neutral     240\n",
       "positive    240\n",
       "negative    240\n",
       "Total       720\n",
       "Name: gold_label, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(dev, dist_labels=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neutral     240\n",
       "positive    240\n",
       "negative    240\n",
       "Total       720\n",
       "Name: gold_label, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_distribution(test, dist_labels=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relationship between validation labels and model predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  model_1_label\n",
       "mixed       negative         1612\n",
       "            neutral           584\n",
       "            positive         1138\n",
       "negative    negative         3693\n",
       "            neutral           214\n",
       "            positive          672\n",
       "neutral     negative          406\n",
       "            neutral          1473\n",
       "            positive          569\n",
       "positive    negative          255\n",
       "            neutral           261\n",
       "            positive         5522\n",
       "dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_model_relationship(train, model_colname='model_1_label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  model_1_label\n",
       "negative    negative         80\n",
       "            neutral          80\n",
       "            positive         80\n",
       "neutral     negative         80\n",
       "            neutral          80\n",
       "            positive         80\n",
       "positive    negative         80\n",
       "            neutral          80\n",
       "            positive         80\n",
       "dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_model_relationship(dev, model_colname='model_1_label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gold_label  model_1_label\n",
       "negative    negative         80\n",
       "            neutral          80\n",
       "            positive         80\n",
       "neutral     negative         80\n",
       "            neutral          80\n",
       "            positive         80\n",
       "positive    negative         80\n",
       "            neutral          80\n",
       "            positive         80\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_label_model_relationship(test, model_colname='model_1_label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adversarial rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3,337 of 17,839 examples (18.71%) are adversarial\n"
     ]
    }
   ],
   "source": [
    "adversarial, total = utils.get_adversarial_rate(\n",
    "    train, dev, test,\n",
    "    model_colname='model_1_label',\n",
    "    labels=('positive', 'negative', 'neutral'))\n",
    "\n",
    "print(f\"{adversarial:,} of {total:,} examples ({adversarial/total:0.2%}) are adversarial\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distribution of validation response distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5    6439\n",
       "3    5322\n",
       "4    4638\n",
       "dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_dist_of_majority_dists(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    393\n",
       "5    327\n",
       "dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_dist_of_majority_dists(dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    411\n",
       "5    309\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_dist_of_majority_dists(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Worker distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(625,)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worker_dist = utils.get_global_worker_dist(train, dev, test)\n",
    "\n",
    "worker_dist.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "w23     2080\n",
       "w79     1953\n",
       "w193    1946\n",
       "w135    1837\n",
       "w439    1821\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worker_dist.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worker_dist.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worker_dist.mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAFLCAYAAAAEZYwbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXgUVd728buzayAQQwzKHvawpE1YBGSJLCHgiAsyIGpANh8WcQRhFJBldISRGRBmnGERGBBZRHTkYRGQdUA2AVFhUCBs6kA0hBASICHn/YM3/dh0CB3opEjn+7muXJpTp6p+p7uv9E3VqSqbMcYIAADAIj5WFwAAAEo2wggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEv5WV3AncZms1ldAgAAXsHdu4cQRvLg6VuvJCcnKzw83KPbvJN48/i8eWwS4yvuvHl83jw2qWSM795773W7P6dpAACApQgjAADAUoQRAABgKcIIAACwFGEEAABYijACAAAsRRgBAACWIowAAABLEUYAAIClCCMAAMBShBEAAGApnk1TBC71StApq4twU6WVe6wuAQBQwnBkBAAAWIowAgAALEUYAQAAliKMAAAASxFGAACApQgjAADAUoQRAABgKcIIAACwFGEEAABYijACAAAsRRgBAACWIowAAABLEUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEsRRgAAgKUIIwAAwFKEEQAAYCnCCAAAsBRhBAAAWIowAgAALEUYAQAAliKMAAAASxFGAACApQgjAADAUoQRAABgKcIIAACwFGEEAABYijACAAAsRRgBAACWIowAAABLEUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEsRRgAAgKUsDSMZGRkaOnSofHx8tGnTJitLAQAAFrEsjGzbtk0PPPCAtmzZImNMnn2OHz+uUqVKyW63u/wsXLjQpf/Zs2fVu3dv1apVS/Xq1VPz5s21bt26wh4KAAC4DX5W7Xjs2LGaNWuWjh07pt69e9+wX6NGjdw6anLx4kW1adNG1apV04EDBxQUFKR58+YpISFBq1evVvv27T1YPQAA8BTLjoysWbNGrVq18tj23nnnHR06dEjTpk1TUFCQJKlXr15q0qSJhgwZ4rH9AAAAz7IsjPj5efagzKJFixQZGanq1as7tcfHx+vw4cPau3evR/cHAAA8446/mubMmTPq1auXYmJiVLNmTcXHx2vZsmVOfS5fvqyDBw+6BBFJqlGjhiRp3759RVIvAAAoGMvmjLjD19dXOTk56tWrlx566CHl5ORo1qxZ6tatm1599VW9+eabkqSUlBTl5OQoJCTEZRu5bcnJyTfcz8yZMzVz5kzH7/n19Xa3Mvbs7Gyvfc28eWwS4yvuvHl83jw2qWSMryDu6DBSqVIlHT582Klt0KBB2rBhgyZOnKi+ffuqWrVqjqtxbDbbLe2nf//+6t+/v2Mb4eHht1f4dU55dGuF61bGnpyc7PHX7E7hzWOTGF9x583j8+axSSVjfAVxx5+myUuLFi2Uk5Oj7du3S5LCwsLk4+Oj8+fPu/RNS0uTdGtfsgAAoPDd0WHk/PnzyszMdGn38blWdk5OjiQpMDBQUVFROnr0qEvfI0eOSJLsdnshVgoAAG7VHR1Ghg4dqilTpri079mzR5LUuHFjR1v37t117NgxHTt2zKnv2rVrVatWLcXGxhZusQAA4Jbc0WFEkv72t7/p0KFDjt8/+ugjLV68WH369FGdOnUc7S+99JLq1KmjoUOH6tKlS5Kk+fPna+fOnZo+fXqR1w0AANxj2QTWadOmac6cOUpJSZEk9e3bV6VKldLo0aPVtWtXSdLw4cMVGhqq7t27S5JSU1N1zz33aMqUKRo0aJDT9oKDg7Vp0yaNHDlSDRo0UGBgoEqXLs3dVwEAuMPZzI0eDFNC2Wy2Gz4r51ad6tzIo9srTJVW7inwOt48K9ybxyYxvuLOm8fnzWOTSsb47r33Xre/T+/40zQAAMC7EUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEsRRgAAgKUIIwAAwFKEEQAAYCnCCAAAsBRhBAAAWIowAgAALEUYAQAAliKMAAAASxFGAACApQgjAADAUoQRAABgKT+rCwC83anOjawuIV+n/v9/K63cY2kdAEoujowAAABLEUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJZyO4xcuXJFP/74oy5cuOBoW7BggYYPH661a9cWSnEAAMD7uR1GJk2apOrVq2vGjBmSpGnTpikxMVGzZs1S586d9eGHHxZakQAAwHu5HUZWrVql9evXa/jw4ZKkyZMnKz4+XikpKdq0aZP+/Oc/F1qRAADAe7n9bJqsrCy1aNFCkrRr1y6dPn1a//znP+Xr66sWLVro8uXLhVYkAADwXm4fGcnKynL8/5IlS3TfffcpLi7O0Waz2TxbGQAAKBHcPjISERGht956SzVq1NDs2bPVv39/x7JDhw7JGFMoBQIAAO/mdhh58803FR8fr9TUVN1///2OuSOTJk3S5MmT1adPn0IrEgAAeC+3w0jjxo11/PhxHTp0SPXq1VOpUqUkSQkJCWrUqJEaNmxYaEUCAADv5XYY6datm6Rrl/TmBhFJhBAAAHBb3J7Aunz5csXFxSk0NLQw6wEAACWM20dGoqOj9T//8z+FWQsAACiB3D4yEhkZqbNnz95weUJCgkcKAgAAJYvbR0ZefvllPffcc+rXr5/q16+v0qVLOy0/ffq0x4sDAADez+0wknv31XXr1hVaMQAAoORxO4xUqlRJr7/+ep7LjDF64403PFYUAAAoOdwOI7Gxsfne2Gz79u0eKQgAAJQsBbq0Nz9z5sy57WIAAEDJ43YYybVnzx5NnDhRo0ePliTt2LGDJ/YCAIBbVqCn9nbr1k1NmzbVa6+95jgSMnfuXNntdp05c6bQigQAAN7L7TAyYcIE7dmzR9OnT9eWLVtUrlw5SdKMGTOUmJh4w8mtAAAA+XF7AuvSpUu1adMmVa5c+dqKfv+36ogRI9SoUSPPVwcAALye20dG/Pz8HEHEZSM+Prpy5YrHigIAACWH22Hk0qVL+vHHHx2/G2Mc///DDz8oMzPTs5UBAIASwe0w8thjj6lNmzZauHChTp06JZvNptTUVK1fv16PPvqounXrVph1AgAAL+X2nJEJEyboiy++0LPPPiubzSZJCgsLkyQ1b96cCawAAOCWuB1GgoODtWXLFs2fP1+fffaZkpOTFR4eroSEBD3zzDNOE1oBAADcVaCbnvn5+en555/XkiVLtGHDBi1ZskS9evWSn5+fPvzwwwLvPCMjQ0OHDpWPj482bdpU4PUBAEDx53YYadu2bb7LC/qgvG3btumBBx7Qli1bnCbDXi89PV2/+93vVLt2bTVs2FAxMTFatGhRnn3Pnj2r3r17q1atWqpXr56aN2/OU4YBALjDuX1u5fjx47p69ap8fX2d2lNTUzVgwAB98803Bdrx2LFjNWvWLB07dky9e/fOs09OTo4eeeQRpaena8eOHQoNDdX69evVsWNHZWZm6vnnn3f0vXjxotq0aaNq1arpwIEDCgoK0rx585SQkKDVq1erffv2BaoPAAAUDbePjJw+fVq//e1vlZOT42j77LPPVL9+fX366acqVapUgXa8Zs0atWrVKt8+ixcv1ubNmzVp0iSFhoZKktq1a6du3bpp+PDhysjIcPR95513dOjQIU2bNk1BQUGSpF69eqlJkyYaMmRIgWoDAABFx+0wEhUVpXr16unZZ59Venq6Bg4cqE6dOqlcuXLatWuXoqOjC7Rjdya8Llq0SIGBgS6hJT4+XufOndOaNWuc+kZGRqp69eoufQ8fPqy9e/cWqD4AAFA03A4jW7du1fjx43XfffepQoUKmjVrloYNG6Zdu3apQYMG2rJli8eL27dvnypXrix/f3+n9ho1ajiWS9Lly5d18OBBlyCSV18AAHBncXvOSO5pmMmTJyszM1M//fST/vSnPzmWP/bYY/rkk088WlxycrIaNGjg0h4SEuJYLkkpKSnKyclxtOfXNy8zZ87UzJkznfZbUt3K2LOzs732NfPmsV3PG8fp7e+fN4/Pm8cmlYzxFcQNw8gf//jHG650//33a9myZXruuedUp04dSdLu3bsLtGN35d5gLT+5V+O40zcv/fv3V//+/R3bCA8Pv6Xt3Mgpj26tcN3K2HPvOeONPDG24vL+e+N76M2fTcm7x+fNY5NKxvgK4oZhZPTo0Tdd+f3333f8/60GgfyUK1dO58+fd2lPS0uT9H9/PMPCwuTj4+NWXwAAcGe54ZyR6OhoZWVlufVz5coVNWzY0OPFxcTE6OTJk8rKynJqP3LkiCTJbrdLkgIDAxUVFaWjR4+6bOP6vgAA4M5ywzAyePBg+fr6uvXj5+ensWPHery47t276/Lly/r3v//t1L527VqFhoaqY8eOTn2PHTumY8eOufStVauWYmNjPV4fAAC4fTcMI3369HH6PTw8XPfee6/Ll32uxx57zLOVSerRo4dat26tESNGKDU1VZK0YcMGLV26VJMnT1ZwcLCj70svvaQ6depo6NChunTpkiRp/vz52rlzp6ZPn+7x2gAAgGe4fTXNpUuXtHPnTkVGRnpkx9OmTdOcOXOUkpIiSerbt69KlSql0aNHq2vXrpIkHx8frVixQmPGjFGTJk0UFBQkX19fzZ8/Xz169HDaXnBwsDZt2qSRI0eqQYMGCgwMVOnSpbn7KgAAdzibye/BML/SrFkzffHFFzdc/sMPP6hChQoeK8wqNpst32fl3IpTnRt5dHuFqdLKPQVex5tnhXvkappi8v7fynt/p/Pmz6bk3ePz5rFJJWN89957r9vfp27f9OzBBx/Url27brj8N7/5jbubAgAAcHD7NE29evX03HPPqX379qpfv75Kly7ttPzcuXMeLw4AAHg/t8NI7k3BvvvuuzyXF8Z9RgAAgPdzO4zUrVtXn376aZ7LjDHq0qWLx4oCAAAlh9thpGfPnnk+iC7XoEGDPFIQAAAoWdyewPraa6/lu3zgwIG3XQwAACh53A4jknThwgWNHz9ezZo1U/Xq1dWsWTNNmDBBFy5cKKz6AACAl3P7NM3PP/+shx56SN999538/PwUGhqqU6dOaefOnVq8eLG2bt2qsLCwwqwVAAB4IbePjIwaNUoRERHauXOnrly5ojNnzujKlSvauXOnIiIiNGrUqMKsEwAAeCm3j4ysW7dOBw4cUKlSpZzaGzdurH/961+Kjo72eHEAAMD7uX1kJCgoyCWI5AoJCVFQUJDHigIAACWH22EkJydHhw8fznPZf/7zH+Xk5HisKAAAUHK4fZrmmWee0cMPP6xhw4bpwQcf1D333KOUlBTt2LFDf/nLXzR48ODCrBMAAHgpt8PIq6++qv3792v48OFOt343xqhbt24aMWJEoRQIAAC8W75h5MMPP9RTTz0lSfL19dWyZcv0+eef67PPPtPPP/+s8PBwJSQkqE2bNkVRKwAA8EL5hpGxY8eqS5cuCggIcLS1bdtWbdu2LfTCAABAyZDvBNbTp0+rTp06euSRR/T3v/9dJ06cKKq6AABACZHvkZHq1atr37592rNnj1avXq0ePXooPT1dHTt2VKdOndSyZUv5+voWVa0AAMAL5XtkZMWKFZKkRo0aacyYMdq+fbs2btwou92u2bNnq3bt2nrqqac0d+5c/fe//y2SggEAgHfJN4xUrFjRpS0sLExPP/20Zs+erSlTpigpKUl9+/bNsy8AAMDNuH1prySdPHlSK1eu1MqVK7Vx40ZdunRJxhjVqVNHnTp1KqwaAQCAF8s3jOTk5Gjbtm2OAHLw4EEZY3TXXXcpLi5OnTp1UqdOnVS1atUiKhcAAHibfMNIWFiY0tLSJEmRkZEaNGiQOnXqpLi4OAUGBhZJgQAAwLvlG0bi4uL0+eefKzIyUi+++KISEhJUvnz5oqoNAACUAPmGkeXLlys7O1tbt27VypUrNXnyZN11112O0zNNmzZ1ujU8AABAQd30qb1+fn6Ki4vT5MmT9e2332rZsmWKiIjQhAkTVKVKFT377LNatGiRzp07VxT1AgAAL3PTMHK9qlWratCgQfr44481depU7d69W88884wiIiIKoz4AAODl8j1N88Ybb2j06NGO30+cOKFVq1Zp1apV2rhxozIzM2WMUdWqVbm0FwAA3JJ8w8jSpUv10EMPadWqVVq5cqX+85//yBgjf39/tWjRQp06dVLnzp1Vt27doqoXAAB4mXzDyDfffKO2bdvKGKP77rtPvXr1UqdOndShQweVLl26qGoEAABeLN8wUrZsWb388svq1KmTYmJiiqomAABQguQbRux2u9OcEQAAAE/L92qaDRs2FFUdAACghCrwpb0AAACeRBgBAACWIowAAABLEUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEvl+6A8AMCtO9W5UeHvw0PbqbRyj4e2BBQcR0YAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEtx0zMAkormBl2ewg26AO/CkREAAGApwggAALBUsTlNExAQoKioKJf2nj176pVXXnH8np6erjFjxmjVqlUKDAyUn5+fXnnlFfXo0aMoywUAAG4qNmHk/vvv1/79+/Ptk5OTo0ceeUTp6enasWOHQkNDtX79enXs2FGZmZl6/vnni6haAIWpIPNbPPUgOQCFx6tO0yxevFibN2/WpEmTFBoaKklq166dunXrpuHDhysjI8PiCgEAwPW8KowsWrRIgYGBatWqlVN7fHy8zp07pzVr1lhUGQAAuJFiE0YyMjI0ePBgxcbGqnbt2mrTpo1mzpwpY4yjz759+1S5cmX5+/s7rVujRg3HcgAAcGcpNnNGAgMD1bZtW/35z3+Wr6+vli9frsTERG3evFkLFy6UJCUnJ6tBgwYu64aEhDiW52XmzJmaOXOm4/cb9SsJbmXs2dnZXvuaefPYgF+7E+8zk9d8n6B5q4u8jsLg7X9bsrOzC9S/2ISRU6ecP5bdunXTzp079Ze//EUDBgxwnJqx2WwF3nb//v3Vv39/x/rh4eG3X/CvFKcJdLcy9uTkZI+/ZncKT4ytOL3/wJ3OW/7WePPfTang/7AtNqdp8tKiRQtJ0tatWyVJ5cqV0/nz5136paWlSfKeDzEAAN6kWISR9PR0paenu7T7+FwrPycnR5IUExOjkydPKisry6nfkSNHJEl2u72QKwUAAAVVLMLI5MmT9fLLL7u079lz7fkUjRs3liR1795dly9f1r///W+nfmvXrlVoaKg6duxY+MUCAIACKTZzRj744AP16tVLzZs3lyRt2bJF06dPV7t27RQfHy9J6tGjh2bNmqURI0Zo3bp1Klu2rDZs2KClS5dqxowZCg4OtnIIAIDbdCdOtL0RHujovmIRRhITE5Wdna0hQ4YoOztb6enpCgwM1LBhwzRy5EjHpFUfHx+tWLFCY8aMUZMmTRQUFCRfX1/Nnz+f28EDAHCHsplf36gDstls8vRL4u1J3ptnhXvkappi9P4D8Jz8/p56899N6dr47r33Xre/T4vFnBEAAOC9CCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEsRRgAAgKUIIwAAwFKEEQAAYCnCCAAAsBRhBAAAWIowAgAALEUYAQAAliKMAAAASxFGAACApQgjAADAUoQRAABgKcIIAACwFGEEAABYijACAAAsRRgBAACWIowAAABLEUYAAICl/KwuAAAAb3Sqc6P8lxdRHe6otHKPpfvnyAgAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEsRRgAAgKUIIwAAwFKEEQAAYCnCCAAAsBRhBAAAWIowAgAALEUYAQAAliKMAAAASxFGAACApQgjAADAUoQRAABgKcIIAACwFGEEAABYijACAAAsRRgBAACW8rO6ANxZTnVudGvrebiOm6m0ck8R7xEAUFg4MgIAACxFGAEAAJYijAAAAEsRRgAAgKUIIwAAwFKEEQAAYCnCCAAAsBRhBAAAWMrrwkhWVpYmTJigunXrqmHDhqpfv76mT58uY4zVpQEAgDx43R1Y+/Tpoy1btmjr1q2qVKmSDhw4oIceekjJycmaMGGC1eUBAIDreFUY2b59uxYsWKDZs2erUqVKkqSGDRtq8ODBmjhxovr27avKlStbXCU84VZvW39L+yqyPQFAyeRVp2kWLVokSerQoYNTe3x8vLKysrRs2TIrygIAAPnwqjCyb98+BQQEqGLFik7tNWrUcCwHAAB3Fq8KI8nJySpdurRsNptTe0hIiGM5AAC4s3jVnBFjjEsQccfMmTM1c+ZMx++3sg0AAIoti7/3vOrISHh4uNLS0lwu401LS3Msz0v//v21Z88e7dmzR8YYj//ExsYWynbvlB9vHp83j43xFf8fbx6fN4+tJI3PXV4VRmJiYnTlyhWdPn3aqf3IkSOSJLvdbkVZAAAgH14VRrp37y5JWr9+vVP72rVr5e/vr65du1pRFgAAyIfvuHHjxlldhKdUrlxZx44d04IFC9S1a1eFhITo66+/1gsvvKDhw4fr8ccft6y22NhYy/ZdFLx5fN48NonxFXfePD5vHpvE+H7NZgpyUqcYyMrK0h//+EctWrRIAQEBunr1qgYMGKAhQ4YwMRUAgDuQ14URAABQvHjVnBEUnYyMDA0dOlQ+Pj7atGmT1eUAeVqzZo1sNpt69epldSkA8uFV9xm5k2RlZemtt97SokWL5O/vr5ycHA0YMECDBw8u9qeLtm3bpueff1533313gS7dKg6OHTum9957T+vWrVN2drYyMzMVERGh3/3ud+rSpYvV5d22kydPasaMGY7xXb58WZL09NNPa9iwYQoKCrK4Qs+5fPmyhgwZYnUZHhcQEKCoqCiX9p49e+qVV16xoCLP2rJliyZOnKj//ve/yszM1KVLl9SsWTPNnDlTpUqVsrq8W9arVy9t2LBB99xzj8uyb775Rq+99lqxfpjrsWPHNH78eO3atUsBAQHKzs5W+/btNXr0aJUrV+7mGzAoFM8++6ypUqWKOXnypDHGmK+++sqULl3ajBkzxuLKbl/btm3N5s2bzdy5c40ks3HjRqtL8pj4+HgTFRVlTpw4YYwxJicnx4wbN85IMpMnT7a4uts3d+5cc/fdd5vNmzc72rZu3WqCgoJMz549LazM8yZMmGAef/xxI8kkJiZaXY7HVKlSxeoSCs2CBQtMhQoVzJ49exxtK1asMJLMTz/9ZGFlty8xMdHMnTvXpf3AgQPGZrOZ7777ruiL8pDU1FRTvnx507JlS5OammqMMebHH380tWvXNna73Vy9evWm2yCMFIJt27YZSWb27NlO7a+++qrx9/d3fNEVV1lZWcYY47VhZPHixU5tOTk5pkKFCiY8PNyiqjxn9erVZuzYsS7tjz76qLn77ruLvqBCkpSUZMLCwszx48cJI8XEjz/+aIKDg82iRYtclq1YscJcunTJgqo8Z/v27ebo0aMu7S+88IKJj4+3oCLPWbZsmZFkPvroI6f2P//5z0aS2b9//023wZyRQuDtTw/28/Pes3srVqxQt27dnNpsNpsqVKig8+fPF/vTUh07dlReV/OnpaW5PGCyOHvppZc0dOhQValSxepS4Kb58+crMzNTjz76qMuyRx55RIGBgRZU5TnNmjVTZGSkU9uFCxf0/vvva/DgwRZV5Rm5z3/Lyspyas/93Z33jjBSCHh6cPHl7+/vMqcnOztbR44c0cMPP1zs5/tcLzMzU1OnTtXevXs1bdo0q8vxiNWrV+ubb77RiBEjrC6lUGRkZGjw4MGKjY1V7dq11aZNG82cObPYB+WtW7eqfPny+vLLL5WQkKB69eqpXr16evHFF3X27FmryysU8+fPV3h4uDp16mR1Kbfl4YcfVrt27TRp0iSdOnVKknTo0CH97W9/0yOPPKI6dercdBuEkULA04O9y8KFC3Xx4kVNnDjR6lI8KjY2VmXKlNE777yj999/X/Hx8VaXdNsuX76sF198UVOnTi32/5K+kcDAQLVt21bbt2/Xt99+q4EDB2ro0KF65plnrC7ttpw4cUIpKSnq27ev3n77bX377bf6+OOPtXbtWjVr1kznzp2zukSP+8c//qGBAwfKx6d4fxX7+vrq448/VnR0tKpVq6YKFSrogQce0LPPPqtPPvnErW0U71fgDmXMrT09GHee77//XsOGDdPs2bMVHR1tdTke9eWXXyojI0OTJ09Wz549NXDgQKtLum2TJk1SnTp19Mgjj1hdSqE5deqUHn/8cQUGBsrPz0/dunXTwIED9cEHH2jLli1Wl3fLcq+cGTVqlOrXry9JqlWrlv70pz/p2LFjevfddy2u0LO2bNmio0eP6vnnn7e6lNt24sQJxcTE6OzZs/rpp5/0ww8/aO/evfroo4+UmJioq1ev3nQbhJFCcKtPD8adJSkpSR06dNCkSZOK/b86b8TPz09PPvmkRowYob///e/67LPPrC7plh0/flxTp07VO++8Y3UpRa5FixaSrp3qKK5yjxw/8MADTu2NGzeWJO3YsaPIaypM7777rnr06JHnpb7Fzauvvqrvv/9es2fPdny/RUVF6a233tLChQs1Z86cm26DMFIIeHpw8Xfo0CE9/PDDmjhxovr06WN1OR6TmZmZ579Scr8Adu/eXdQlecy6detUqlQpPfHEE7Lb7Y4fSfr0009lt9vVqFEji6u8Penp6UpPT3dpzz3Mn5OTU9QleUzuvVOuH4Ovr2+e7cXZmTNntHz58mI/cTXXvn37FBISovvuu8+pPXeuiDtBkjBSCHh6cPG2d+9edezYUe+++65++9vfOtoHDBigL7/80sLKbl9CQoKWLFni0n78+HFJcu/mRHeofujlRsQAABPFSURBVP366eTJk9q/f7/TjyQ9+uij2r9/v/bs2WNxlbdn8uTJevnll13ac8eVexShOHriiSckyfGe5cr9/cEHHyzymgrLrFmz1LhxY5ejQMVV+fLllZaWppSUFKf2pKQkSW6eDSiMa45x7aZnVatWNadOnTLGXLuxTUhIiFfc9CyXN95nZMuWLSYkJMT079/fLFiwwOmnTp06xX6srVu3NvXq1XO638HevXvNfffdZypVqmRSUlIsrK5wyIvuMzJ27FgTHBxstm3b5mjbvHmzCQkJMe3atTM5OTkWVnd7cnJyTNu2bU3NmjUdN4tMTk42sbGxplq1al7z2czOzjaVKlUyH3zwgdWleMyKFSuMzWYzPXv2NJmZmcYYY3766ScTExNjSpcubb7//vubboMH5RUSb3568LRp0zRnzhylpKTo1KlTql69ukqVKqXRo0cX+6M+MTEx+V56vXHjRrVp06boCvKwL774QvPmzdO2bdvk4+OjjIwMSdfugTN69GiXw6zFWe4pmq+++kqhoaGqXLmyHn300WJ9y+2kpCS99957Wr16tbKzs5Wenq7AwEB1795dI0eOLPZXEF28eFETJkzQsmXLFBAQoMzMTHXo0EF/+MMfFBERYXV5HvGvf/1LAwYM0KlTp+Tv7291OR7z73//WxMnTtSRI0cUEBCgK1euqGnTpho1apRq1ap10/UJIwAAwFLMGQEAAJYijAAAAEsRRgAAgKUIIwAAwFKEEQAAYCnCCAAAsBRhBAAAWIowAhQzX3/9tex2u0qVKlWkN2CbO3eu7Ha7bDabxo0bV+D1H3/8cVWuXFk2m81x+/nbVRjbHDRokGrUqCGbzaZNmza5vd7ly5c1evRopaenW/YeWenNN99UVFSUbDab5s2bV+D19+/fr1mzZnm+MBQLhBF4hdTUVNntdpUvX142m01RUVFOD0uz2+2KiopS1apVrS71tjVo0ED79+8v8oe+9e7d2+W5IQXx8ccfe/zup4Wxzb/97W+aPXt2gdZJT093hI5SpUpZ9h5ZadSoUVq1atUtrx8VFaX//d//Vf/+/T1YFYoLwgi8QtmyZbV//3698MILkqRVq1a5PDDtdv5QAvkZMGCAwsPD9cYbb1hdSrEVEBCgJUuWaOXKlZoxY4bV5aCIEUZQYkRERGjOnDlWlwEvs3//fn3wwQcaO3as1aUUe0FBQRo2bJjGjh2rK1euWF0OihBhBCVCmzZttHPnTj388MOSpCVLljjmBURERGjatGm6evWq7Ha7/P39VbduXX366aeSpIMHDyoxMVENGjTQAw88oOjoaL388stKS0tzbP/gwYNOcwSWLl2qNm3aqGLFioqLi9ORI0f0yy+/6JlnnlGDBg1Us2ZNLViwwLH+L7/8IrvdrnvuuUdVq1bVxx9/rObNm6tWrVqqWLGi/vCHP8jdx0h99NFHaty4sWrUqKGqVauqS5cuOnjwoFOfI0eO6Mknn1T9+vUVExOj2NhYjRw5UsnJybf0+k6dOlXNmjVTbGys7Ha7WrZsqc8///yG/Y8fP67HHntMDRs2VFhYmBITE3Xu3DmnPtnZ2XrzzTdVu3Zt1alTR5GRkRo4cKBLvxtJSkpS9+7dVblyZdWuXVt2uz3PuQyHDx9WQkKC7rnnHsXGxuq5555zeRR6ft5//32Fh4crNjbW7XWWL1+upk2bqnr16qpcubI6d+6c5wMaV61apQceeEARERFq1KiRhg0bpldffVU2m012u11z587Ndz9//etf1bBhQ8fn9tFHH9Unn3zi1OfEiRPq2bOnqlSpoujoaDVo0ED9+vXTnj17HH1Wrlypzp07Kzo6Wna7XTExMXr33Xfd/kxevHhRw4cPV2RkpOrWrauaNWvqtdde0+XLl136xsfH68yZM1q3bp1b24aXKLyHCgNFb+zYsUaSSUpKcmpv3bq12bhxo1PbpUuXTP369U2FChVMcnKyMcaYDz/80OVR7NOnTzdxcXEmPT3dGGNMenq66dKli3niiSdc9t+6dWsTERFhJk6c6NhH06ZNTaNGjcwrr7xiUlNTjTHGTJkyxfj6+ppjx445rZ+YmGgCAwNNt27dzKVLl4wxxqxatcr4+fmZkSNHuuyrdevWTm1///vfjSTzz3/+0xhjzNWrV03//v1N2bJlnV6TWrVqmd69e5urV68aY4w5fPiwKVeunMtrlBdJZuzYsU5t5cqVM6tWrXL8/vnnn5u7777b7N2716nf3LlzjSTTqlUrc/r0aWOMMUePHjUVK1Y0zZs3d9RjjDG//e1vTWhoqNm/f78xxphffvnFNG7c2DRq1MhcuXLFZZu/Ht+JEydMeHi46dSpk8nIyDDGGLNhwwbj7+9vpkyZ4uj33//+19x7772mZcuW5vz588YYYw4cOGDq169vJLn1ejRp0sS0atUqz2V5vUfvvvuusdlsZvHixcYYY3Jycszw4cPNXXfdZXbu3Onot379euPr62t+97vfOT6PixYtMsHBwcadP90LFy40/v7+5vDhw8aYa4+uf+mll5zqOXnypImIiDBPPvmk4/N26tQpU7NmTZOYmOjo9+STT5rf//73jvfnxIkTJjIy0kybNs1pn0lJSUaSmTt3rqMtKyvLPPTQQ6ZatWrm+PHjTus/9thjLnVnZWUZHx8fl887vBthBF4lN4zUrVvXREdHO36Cg4Pz/GLZv3+/CQgIMJ07dzZJSUmmWrVq5tSpU059zp49a86cOePUtnr1aiPJnD171qm9devWJjQ01OnL8u233zaSzIcffui0TUlm9uzZTusnJiYaSebEiRNO7V26dDEBAQFO+7v+i+7ChQsmJCTEtGzZ0mnd1NRUExQUZPr162eMMSY5OdlIMm+88YZTvzlz5pjvv//e5TW6Xl5h5Ouvv3bp17RpUzNw4ECnttzgkBuWck2ZMsVIMp988okxxpjNmzcbSWbMmDFO/dasWWMkmYULF7ps89dhpHfv3kaSOXr0qNP63bt3N2XKlHEElJEjRxpJZseOHU79cj9H7oSRsLAw07Vr1zyXXf8epaWlmdKlS5sOHTo49bt8+bIpV66cad68uaOtadOmJjQ01GRmZrps050wMmjQIBMSEuIYqzHG/Pjjj2b69OmO33v37m18fX0dwTDXe++9ZwYMGOD4/ejRo07bMebaaxcVFeXUllcY+ec//2kkmffee8+p7z/+8Q8jyWzbts2l9vxeU3gnTtPAK10/gfVGVzVER0drwoQJWrlypZo2bao//vGPqlixolOfkJAQffDBB3rooYdUr1492e12DR48WNK10x3Xq169uvz9/R2/h4WFSZLq1q3raCtXrpwk6aeffnJZv2zZsqpcubJT24MPPqgrV65o27ZtNxzz9u3blZaWptatWzu1lylTRlWqVNH69esd9cTExGj8+PHq27evNmzYoKysLPXu3Vs1atS44fbzk52draefflrR0dGOQ/nffPNNnq+PdO11v358krRx40ZJ0po1ayTJZSy56+WO5UbWrFmjihUrKjIy0mX98+fPa/fu3Y79+fr6unw+7HZ7vtv/tfPnz+uuu+5yq+/27dt14cIFNWvWzKk9ICBAMTEx+uKLL3T+/HldvHhRu3fvVnR0tIKCgm6ptvbt2+vChQtq1KiR/vrXv+r06dO67777HJ9dSVq9erUqVqyoChUqOK37/PPP6x//+Ifj9+DgYI0aNUqNGzdWgwYNHKe8bvT+/tqtvJd33XWXzp8/79Y44R38rC4AKAr53S/ilVde0YIFC3Ts2DG1bNnSZfmAAQO0aNEirVy5Uu3atXNsLy4uLs9z3sHBwU6/22w2l/bctuzsbJf1y5Qp49J2zz33SLo2t+RGcud7zJs3TytWrHBalpGR4dinzWbTxo0bNWXKFM2fP1/vvfeewsLC9MILL+j1119XQEDADfeRl2+//VYtWrRQhw4dtHXrVoWEhEi6Nk8nr9cnrzFeP77csQwePFiBgYFOfSMiIpSZmZlvTcnJyQoICHD54s7MzFRERIRjTsgvv/yiMmXKyNfX16lf2bJl893+r/n4+Lg9d+Lnn3+W9H/j/bWwsDAZY/Tzzz8rMDBQOTk5efZzt7YuXbro888/1zvvvKNhw4ZpyJAhatmypSZPnqwmTZo46mnQoEG+28nIyFBcXJyysrK0Zs0aVa9eXZI0btw4jR8//qZ15L6XTzzxhOMzKElXr15VRESE0tPTXdYxxsjHh38rlySEEZR4u3btkp+fnwICAtSrVy+tXbvW8Ufz0qVLWrhwoeLj4x1BpLDl9S/C3C/p3KMseQkPD5d07Qt85MiR+e4jJCREY8eO1dixY7Vr1y5NmzZNb775pmw2m/7whz8UqN4PPvhAGRkZGjVqlCOI3Mz1Y7x+fLljmTdvnpo2bVqgenLXL1OmzE3vixIWFqakpCRdvXrVKZC4O0lWuhYsLl686HZdkvKcIPvLL7/IZrOpXLly8vPzk4+PT57hsyC1xcXFKS4uTqmpqVq6dKnGjx+v9u3b6/jx4woNDVV4ePhNJ+tu27ZNhw4d0ttvv+0IIgWRO+a1a9cqIiLCrXUuXryYZxCD9yJ6okTJPT2S6+LFi+rXr58WLlyoadOmaf369Zo+fbpjeXZ2tq5everyr7S8Tq94Smpqqk6ePOnUtmPHDgUEBKhFixY3XK958+YKCQnRl19+6bIs94tIks6ePasXX3zRsaxJkyZ6//33Vb9+fX311VcFrjf3EsyCvEbX72fHjh2S5LjaKSEhQZLyHMvrr7+ujz76KN+aEhISlJSU5PLF/fPPP+uJJ55wfAG3bdtWOTk5jtM2N6ovP3Xq1HH785D7HuWON9eVK1e0b98+NWvWTGXKlFFwcLCaNm2qr776SpcuXbql2qZOnerYT9myZdW/f39NmzZNaWlpSkpKknTtdTp9+rR++OEHp3Xff/999evXz1GbVLD399fyey9feOEFbd261ant0qVLSk1NVe3atd3aPryExXNWAI+60dU0ua7/yPfr18+88847jt+feOIJExQUZA4ePOhoa9eunQkKCjK7d+82xhiTkpJiYmNj85zgmNfVE3lNsMyt5fqJoImJiaZUqVLm2Wefdbma5ve///1N9zVjxgzj6+trli1b5mjbv3+/qVChglm5cqUx5tokQ39/f7NhwwZHn++++86UKVPGaXLjjVxf99atW43NZjM9evRwTNzNnbR4o9ciNjb2plfTPP3006ZixYpOk2MXLlxoypcv7zTJOK/XN/cqkR49ejgmXqalpZmnnnrK9OjRw9Hv7Nmzpnz58qZly5YmLS3NGHNtMm7VqlXdnsA6fvx4ExIS4lR7rhu9Rz4+Pmbp0qXGmGtX04wYMcLlapqNGzfmeTVNRESEWxNYExMTTZcuXRxXgWVnZ5shQ4aYChUqOCbF5nU1zbFjx0yVKlXMRx99ZIy59nkPDw83devWdUzk/vrrr01ISIhLHTe6mqZVq1amQYMGjonZV69eNW+//bapUaOGo75cu3btMpLM1q1bbzpGeA/CCLxCSkqKqVKliilTpoyRZCpUqGCqVKni8pP7x3P37t2mYcOGRpJp2LChOXPmjNmwYYOpXr26kWTCw8NNnz59jDHGnDlzxvTs2dNERESYRo0amY4dO5px48YZSaZ69epm/Pjx5ocffnBctRMcHGyio6PNmTNnzMCBA02lSpUcV/jMmjXLrFmzxkRHRxtJJiIiwiQkJDjGkZiYaKpUqWLWr19vWrVqZWrWrGnuv/9+M2HCBMcX0oEDB1z29e233zq28fHHH5umTZuaypUrm9jYWNOmTRuzevVqx/KMjAwzfvx4Exsba6Kjo43dbjfR0dFm6tSp+b7Gc+bMcaq7WbNmjmWLFi0y9erVMxUqVDBxcXFm+PDhplGjRk6vxWOPPeZ4LVasWGHat29v6tevb0JDQ81zzz1nUlJSnPaXnZ1tJk6caGrXrm1q1qxp7Ha7eeqpp8yhQ4ccfX69zbp165q33nrLsez48eOmZ8+epmLFiqZhw4bGbrebcePGmcuXLzvt57vvvjMJCQkmNDTUREdHm8cff9wsWLDA8f5ef0XQ9U6cOGH8/PzM2rVrHW03e4+WL19umjRpYiIjI02lSpVMQkKC+fLLL122vXr1amO3201ERIRp0qSJef31183o0aPdCiNbtmwx3bp1M1FRUcZut5t69eqZrl27mv/85z9O/ZKSkszTTz9tKlWqZBo2bGgaN25slixZ4tTnyy+/NG3atHFcBt2jRw/HlV/R0dFm5cqV5o033jB169Y1kkylSpXMk08+6Vj/4sWLZuTIkSYyMtJxpVufPn3MDz/84FL3a6+9ZmrXru10eT28n80YN2deASh0vXr10qZNmzz20DcUjdGjR2vt2rX64osvXCbDetqQIUM0b948XbhwoVD3Y4WzZ8+qVq1aWrx4sTp27Gh1OShCzBkBgNs0YcIE1a9fX127dtXVq1c9ss09e/bkOZl4//79Bbrba3GRkpKi+Ph4vfnmmwSREoiraQDgNvn4+GjOnDlavny5rly54vZ9R/KTmpqqyZMn6ze/+Y3jEuX58+dr+/btjnt3eJNz585pxowZjsuOUbJwmga4A/zyyy9q27atTp48qfT0dEVFRWnKlCmKi4uzujRY5PTp0xo3bpy2bt2qoKAgpaamqnLlyhozZow6dOhgdXmARxFGAACApZgzAgAALEUYAQAAliKMAAAASxFGAACApQgjAADAUoQRAABgqf8HRBZMIQn6Ey0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "utils.plot_global_worker_dist(\n",
    "    worker_dist, os.path.join('img', 'round2-workers.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimates of human performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.681"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_fleiss_kappa(dev).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.667"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.get_fleiss_kappa(test).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>support</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>positive</th>\n",
       "      <td>0.925</td>\n",
       "      <td>0.896</td>\n",
       "      <td>0.910</td>\n",
       "      <td>240.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>negative</th>\n",
       "      <td>0.926</td>\n",
       "      <td>0.899</td>\n",
       "      <td>0.912</td>\n",
       "      <td>240.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>neutral</th>\n",
       "      <td>0.900</td>\n",
       "      <td>0.878</td>\n",
       "      <td>0.889</td>\n",
       "      <td>240.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Macro avg</th>\n",
       "      <td>0.917</td>\n",
       "      <td>0.891</td>\n",
       "      <td>0.904</td>\n",
       "      <td>240.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           precision  recall     F1  support\n",
       "positive       0.925   0.896  0.910    240.0\n",
       "negative       0.926   0.899  0.912    240.0\n",
       "neutral        0.900   0.878  0.889    240.0\n",
       "Macro avg      0.917   0.891  0.904    240.0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.estimate_human_precision_recall_f1(dev).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>support</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>positive</th>\n",
       "      <td>0.923</td>\n",
       "      <td>0.894</td>\n",
       "      <td>0.909</td>\n",
       "      <td>240.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>negative</th>\n",
       "      <td>0.929</td>\n",
       "      <td>0.892</td>\n",
       "      <td>0.910</td>\n",
       "      <td>240.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>neutral</th>\n",
       "      <td>0.894</td>\n",
       "      <td>0.872</td>\n",
       "      <td>0.882</td>\n",
       "      <td>240.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Macro avg</th>\n",
       "      <td>0.915</td>\n",
       "      <td>0.886</td>\n",
       "      <td>0.900</td>\n",
       "      <td>240.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           precision  recall     F1  support\n",
       "positive       0.923   0.894  0.909    240.0\n",
       "negative       0.929   0.892  0.910    240.0\n",
       "neutral        0.894   0.872  0.882    240.0\n",
       "Macro avg      0.915   0.886  0.900    240.0"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.estimate_human_precision_recall_f1(test).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "agr_df = utils.get_worker_agreement(dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "116"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agr_df[agr_df.rate == 1].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "244"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agr_df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5163934426229508"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Percentage of workers with 95% agreement or more:\n",
    "\n",
    "agr_df[agr_df.rate >= 0.95].shape[0] / agr_df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAEDCAYAAAC8vCafAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3hU1b3A/e/ee/aemSQzSYAQwi0Ywv1ipN7hVVGEU09P7cVaag89bVW8VOqxPW2tR0WhpfZVa1Vq24jSVlqkVevt1JZDq9VXjtZSEKwXQLlKgISQ+8zsPXuv949hhgwJYUBgkpnf53l8ZNZea8+aMPyy9rpqSimFEEKIHunZroAQQvQFEiyFECIDEiyFECIDEiyFECIDEiyFECIDEiyFECIDvmxXIFs0Tct2FYQQvdDhZlPmbbCEw/9Q8kl9fT1lZWXZroboBeS70HMjSh7DhRAiAxIshRAiAxIshRAiA1kNlh0dHdx4443ous5LL72UzaoIIUSPshYsX331VU477TRefvnlYxpoWbVqFeeeey4TJ05k9OjRfPnLX2bv3r0noKZCCJHFYDl//nwefvhhbrzxxqMuu3LlSv7lX/6Fa6+9lrfeeov169eze/duLrjgAtrb209AbYUQ+S5rwfKPf/wj55133jGVnTdvHueccw5f+tKXAAgEAtx///288847PPDAA8ezmkIIAWQxWPp8xzbFc82aNWzcuJGZM2empY8ZM4YRI0awfPny41E9IYRI0+cmpa9duxaAkSNHdrlWXV3Niy++iG3bWJZ1squWVXZTI87WzXiFhWzd/AH721vRYg5awI/uD6CZJk5TE7S34vkD0NGOFiyiXfPoVzEEt6MNd/cevKICiMRQHe0QDGIUFGKYFkMGDyLYth9v61YCNWcQnjgFIxjM9scW3YjG26hv3UYk1syG9W8TjTbhaR7hQDk+M0g01oTpK8D0FeLE2/EZfooC/WhqbKCwuIhorIW4crGdNny6H13TQDfx6QamL0j16JFYfh+GBqWFQwn4irL9kU+KPhcs6+vrAQiHw12uhcNhXNelsbGRQYMGneyqZY3d1Ej9nTfh7d0FbW3Me+1d/rq36bi+x12TKrli2AAA2p5eRuTCT1B+/XckYPYy0Xgb/7vhR+zYt5aYauG7n/4rsQ73uL7HLbUXUXFKAQVWMQOLxzJjwo15ETD7XLBMjpwfy9ru2tpaamtrU6+Tgbevs9/bgNfSDPjATfzDOKXQz3+PrwQN8Ach7nDV/72d8T2XfCzRcm+y4/zXhm1wyIwF94ON1G/ZjFGWP7+U+oKmyA6a2utx1cEAOeGcAZz1LxWdcuk8Ov/NjO/51TsnAbDln628+NutuMrBVS6u59HS3sCOXRspCQ47Xh+h1+pzwTK5drW5ubnLtZaWFgzDoLS0tNuyc+fOZe7cuUAi2ObKOljbrKE+XIwXrQPDADRKTB8zBw8ATUHpAIg7bC8vBts+WFDTIRSGkv6wpw6cGHjprZDdsTiwLZG3E6NqNGWnVEvLspcJxYO80zCAlsiOVNqAiiATz+n8Xdf50coZJIYsdCw9QMAXJqD1p9nZiue5OLQdcmcd5ZjAVgzNxNAMDN0gXDiAYYNHS8uyN5oyZQoA77//fpdrmzdvZty4cfj9/pNdrayySvpRNv++VJ+l+ZW5GK0tlNx2L1phECNQgGb5sffVQ0szbjAIra3ohWFaNI+BI6qItzbj7tiBGw5BewTV1gKFRYTaI/DnT1Bw2ZewzjlN+ix7OcsIcmrlJxjafzKOG8Onv8bQkhqmVX+JfgXDMK1C2iMN+M0iLCuMbbdg+oIUFwykYe9uQv3CRKL7cbw4UbsF0wigaxqabhH48EWW8g9mTPoGo8ZWSZ9lb1NXV0d5eTm6nmjZTJkyhVGjRrFq1Spuu+22VL5NmzaxdetWFi1alK2qZpVV0g+r5kwAjKIQRtwl9LGz0/IEBg/vUs6ur8ffvwx//0EwYkyX65EPP0yUrRhK+cWfOgE1F8eT69nomo8hpRNw4lF0zWBgcTVnnHLZwUzdP3gRC0BZYRkUVna55imX0oL3ACgrGs7QkrEnovq9Wq9eG75ixQoGDx7MvHnzUmmaprF48WJeffVVli1bBkA0GuXGG29k7NixfP3rX89WdXOecl28aATlHt8BA3H8GLqFrhlE7GY0LbN/3nHPpsPeT9yLA4nA6LgRvAP9nnHP5sP9G9jf/mHq+qF58kHWguUDDzxATU0Nt99+OwBXXXUVNTU1PPHEE6k8JSUlhEIhKioq0srOnDmTF154gYceeoiJEycyadIkysrKeOmllygsLDypn6M3MgyDN954gxtuuIGGvXuPKcAp1+WN1a/y+csvB0DXNOzNb2Nvegd789sSMHs17cBA6MFB0MMFt7hn886uVbxb9yJbGl/GdiPsad7I7uaN7GneSNyzqdv/NvWtHxC1E+MEcTeWlidfAqam8nQHXE3Tcnbz33379jF//nx++tOfUhIKceu1V3PlJy8hOO5UfOFiNMNI5e1uw9e6HTu45cZ5/PLpZynrV8r3Fv2AL3/xCtwPNqIHgnixCFb1OPSA9Fn2No4bYXfzRixfECceZfyIaVRUVPCTJXcxZkIVPt2ivHg0upb4DnTY+3m37kWCVoj6xt2cWjWDlsjeVPl+RcNoaN3Klh3v8d0bf8j/vbyONWtfp2y4P5WnvHgUppEb34We4kKvfgwXmVOui9vehtveRr+SEhYvXsw/Xn+NSaNH8Y0f3kPx1Aux+vVH9/nQNC3138CBA9Nea5rG4OHD+fVzzzPv0//Gut//jiu/NAdfIIhmmXixCJppopn5Nem/rzB0C59u4cSjGLrJb5b/mpaWZi656HJ+/sBj2E4Ux42kWpmWrxDTCBC12/DpFkGrJK285Stk1Qv/H5+ZcR3/eP1t7vnR3UyedFpaHkPPj++CtCxzgHJdYu+9hbN1I0ppWFWj8I+eCMD/Lf8ll827ibqmlqO65yWnTeaBL36GsjHjKZp5KbploVwX5dhoppXWOhW9i6dcXM9G0wyUcmnYt4+r536V/3nmf/nYmZO598EFVAwrxe8rYEj/yQDY8Xb2N3YwYEAxnnLRNYP21hg3/ec3+NWvfkXNaafyy1/9kskTT017j2Qfaa7oKS5IsMwBXjRC9J9rcffuAeWhl/anZVAlt33/+zz66KP0CxVx6ydn8cXzp1J05jSskWNTj9D1dbsoqxiMZhgo1+XD117hju99n6V/+jPFhYXc9rVr+Nqt87GKQln+lOJoeMplT/NGHDdKc6SOImsAjz/+OD+4rZbWlqPbmeumb1/H1276MpZlUVEyHl0zcjJQQs9xoddPHRJHppkWerCQuG0T37OT5X9cxS1LH6MjGuWGL87mlmuvpd+Y0WiGhbt7B84HG1HKQ0UjxNsj2O1NWNXjUY7NgHCYB+++m/84Yxm3PPk/fOOH9/Lo83/k17/8BRMnTUY5iUnteiAorctezHEjROOt+AwLx40C0LivCdt2jvpe9fUNNLbWYVguyvMwDAtPuV36P3OdBMscoBkGVvU4vLYWvFiE36/+NUWWxf8u+G/GjR2NURDAV1QMQPzACp7I639F2TbKZ+GWlhx8vLZMvJYWJtfU8MfPfJ7fPPc8c29fwCtPP0Xlh++DUuimgTliNP4xEyVg9kKecmlo3UZzx24Uivq6JuZ9+y5Wv/IGF86cxg23/RvF/QLousWI/lPY37ELTTeoa9hEWfEQdN2kLFxN/+BI/vu//5v7f/wAf/3LK/zggW9z7tlDcIlTYJXgxKOJeZ05MrhzJDLAkys8F80fwNevP7hxBvcvZVS/MM62D4h/uA10A3SD+J4Psd/dQHxfA1q4BBWJoMWdVD+kecqYxACOZeJ+uI3pZ52RuH1zI/bWjcS3v48bjeG1t6VamaJ3cT0bT8UpLx7NX55dy+WzbmL92nf4ee3PeOF//sS/njOPM0dewRlVl9MvlJiAbqBTaA2gpGgIpYWD0TUd3VT8x39O56e/vhXbsfnSp/+LH921BFwz7wZ3QFqWOUMzLfRAAMoq0IIFaE4crTCEUdoPo2Joas23r3woDBxKvLUZI1iEXhHAmjDlYAvRc9EMH/5RE3DbWjELSwBw7Rhuw17spgb0D7ej+QMEdGlV9kaaZoBSxONR7rjlHqqqRvLb3y1nVPWY1COzT7dSfZqG7qMoMAA35sfQfASsYpx4jB371tHYvp3JZ1ax4oUf8eM7n2TxfUu4+is3MG78qJzss+yJtCxzROJRfDxW9dhE4NN0vOZGVNzF27cHdONAQPWj+XQKzrmAggs/jnH+rLQ13slHceXYGEVF6P4AAL5wKb4B5ZhDq7AGV2IUl3TZdENkn6dc6lveR6FAAw2ds6eeRkF/J20CuevZxD0b0xck5C+jf+gUDM1HY/tOdjdvROHi0/34fUVE7TaKS0qY/fkrANA1A9MI5lWgBGlZ5hTNMBKB0lNouoZROgDf4GFo/kCixWhZqYGc5GO3fsg2dcmgqxwbdAPntZcTFwJBjIoheJvfxY204bY0JR7tRa+SDIJ+sxAnHgXUgfmUwbQ+xkSr0Mfe5k0oQNM19kd3UhwOE4/bFBcMoT3awNB+k/GUy5DSSWzUngPImxU7h5JgmWM00wI9Mf3Ba23G3a2jFRTAhNMyv4dhABZuazNeNNEvqZr2YQ6aDnEPs2oUuhWQlmUvlJyUHnM6SGz5qqFrBrbTARx4RCfROhwQqsR22wmYIaJ2C0rF0dBoc/bR2LoVvxliUPGY1Oqc5o5dAOxr3YanJuZdy1Iew3OIZ9u4rc1ogQI0K4A5dhJW1RiMgYPxOtrwbDuj9d3KdbE3v42z7X3chrpEmgKjXxmqow13xzbc+l3SsuyFdM2gLDwSjcRG2QqPgBnGwyXuRVOP4snWoWUU4LoOlq+Q/gVVFPpLCfvLCfpLUAcmpyfnVboqsdGG7XYQi7fl3UYa0rLMEZ5t0/7SH1CRCO7+BpQdx2vYi608MA4ucVQKjIJCvFiEXdu38bkv/jtPPfVU2jEcXjSC19aGXhTGGJDYxMRXNhBiHfjKKzArR4KnpGXZS7meTVzZBMwQKHCVw/72XeiahlL19CsaTlP7h8QPbOdWFq7CNIL4nAZK+hXS0LoN17VTo93J4JrclqMlsodNu1+hpGAIphHIm7mW0rLMEV5HGyoSwQgVJ46W8Fz81ePwDRiIMWAgRkERSoGmkVrf/b0f/r+8/vrrLFy4MHUf5bo4294nvmcnsc3voFmJ36e+8iFYYxItVU3TEgNFsj681+k8x3Jv82YAdE0HDvzlc7Bf0/IFU61HAM+zMY0gFSVjKS8eRXnxaADqmt5l1/638Q78ciwtHHpgBY8P13NwvfyYQibBMkfoBUVowSDxpkYAlKbjtjahTAvN8OF2tKMH/JijJuAbOoKGQJhf/PKXeJ7H0qVL2bV1S2rtt3JsjCEjMEr64atInK2i6TpGYRH+MROxqsdhVY+XCem9UHKOZVl4JIX+fmiajqGZlIVGEvYPZECoioAZRtd8RGLNaJqBphnsad5IQ8eWtBFzT7l02PvZ27KJ1the2mKJ75bjRACNuOvk1VxLeQzPEbplETx3Bu2rnk70VXW0Etv0Dl5rC75Bg7GGV+EbXk18+2aU7bBg0Q/xPA8ANx7njm99kwfunI9v+EjsbR/g1u0E00c8kr48LjHinh8rNvqi5NzHhpYPUCiUSvwdV5SMTa3nTlAoTQHq4DQiw0/cjVHX9DZKKZojdRT4i2mJ7qU0eHBP2ZbYXgYxAjRFWXhkXjyCgwTL3BJP7DSj+Xwo28Gr340XjUD5YPBcVCyCsh30QJDX1q7DPrD00XYcXvvHWmKb3yHesAe3uQl08A0cgnJiWf5Q4mgkRrlHYMc7CFiJ46KV8hIDNQd+yTluBNdzsIwgrhdP9El6LlG7mQI1AE8pFHGidjPhYDmF/n4ErRIK/YnzKAzDwqf7E/sLyACP6IuSj+LE46BpaMFCVLQjMccyWJC4fmBPyr898wRW9Xj2friD4Ia/g6bhbN+Cs30LXqQD5cbxDRiEVtj1fHbRu5lGkIAZwnXtxMT0Q46XUEBj23Y8Fcdn+EGDxvbttMSaKNUG0Nz+Ifvad9ARa6LdbqI4WI7hs1AkWqltkXriahjmgfma+UKCZY5QBwZ1Cv6fWRj9H8DwIFBzJugG1uhxGAVFqbXfXkdbInAaBpo/gDliJN6+vXj9B6La9qMHAujhEgJTL8IfzY/O+1yiawblxaMTe1qipwXLxNZt7+EqB8MwCfhCxJwWGutb+OZ1P+Snj/6Q/gNKKC10KQmUY7tR+odG0NCyhdZIYgFDeXgUo8qnUmCV5s0jOEiwzAnJeZHKdtAsE81nogcLCEyakrZRr3JdnC3vpfJZ1eOBAy2PYCF6QQH4/SjPxaoeh9m/DG337mx+NHGMOj92d+Z6NkopLF8h+9q24lgxDN3g5/f/hvVr3uOnP/4137/7ViJOM0rXKfAFcF0HTTOwfAUAmL5A3gVKkGCZE5RjHwiAfrz2lkQrU6kuO5on8+mBIG6knXhTI15LM14sii9YCIOG4qsYghEKp1qiqbKehxeNyC7pfZyhW5hGgCJ/P2JOK5ZZQN2uOp594kWUUjyx/FkWLbybsYOnE/ec1F6YPqOAAmsTAANC+TOo05kEyxygmRb4DGKb3wHl4jbt42//eJNQOASHHoeqvESnlUqMhCoFWnIOnlKJVTkaqXLJEfP43l3Ym95JtUglYPZNyUd0x42gayaN7VtZuvgZPC+xO7jrutx6+3eo/fmjvL/nJfa1b0fXfFSVnUFxwYEFCrqZzY+QNRIsc4BmGJjDR+J1tKObfj559ulMnDgJUOjFpYnHbF1H03WU56EcG7exAa+jjcVP/w83fOJf0ENhvI42fCX9USiMkv5ovsTXQ8XjTDilKnWyo3JsmT7Uh+magd9XxJB+E9F1nQ1rN+Ec2EHdth3eeH0tEbuJmNOGafhxvfiBtebpfZ+5erTE4UiwzBF6IIhRFMKLxfjyJz6Or395Yl9LXU8EN03DGjsZ3bLwbJu2vzxPfMdWFj/9P3zvqi9hDhmB27gHY+BgdF3HHDUhtZxRMy2cLe/JyY45xqdbDCmdxBtrXqO+5QN2NWzBH9AoLRqG3wzhN4tot/cngqsZQtcS4SJ5vk/cs/PqaAkJljkiubWaF40kHqVjUYg7eJ7Ca9yL1544pMo/4TTwXMyK4ZiDhgNgDBqCbplY584g/sG7eK5L+0svgBsHTceqGoU1clxiCpL0WeYUXTMImGEsXwEFvv4ovQUdg/1tOxg7+CLseDuGbuL3FaFrGwFwPYe4R5dt33KdBMscktzPUjk2ynPx2lrR/H689nb0gkIUiU0ygMQGv5HEtl1mST+8WBSvrRmlFLrhw2tpSpwAaRp4zfvxYhF0aVH2SUp5OG7ksI/MyX5Mu93ANYNYZiGua6MBRf4B3d4PNCJ2c17NtZRgmWOUgsja17A3vYOug/+MC7AqR6AZPjTTxNn2fmKjDcPAHJWYOuS2NhNv2INyXeK7P0TFbby2FtzWpkQ/Vdwhvv/PmIOHowf8MsDTx9Q+XMuK3z4OgK770FL7Bx2kUMTjNmiJgR5dMzAOGcjp6Ej8ct3Xtg1fezARSI2u98pVEixzjIpFMIIF+Ir7gaFDtANz+EiMwiI8x8bZ+DZ6URjlxNAO7EepHBucOEZBIWpAOVrcwV89AXd/PQowQsU4H2xCNwyU48gATx/yxX//Avtb9qJpOlG7BdMIoOsGQasUXdNRShFxmnE9h9/95jlm//uncb04ATOET/ejaQeDoafiROwWSktLiXst+M1C1IGBHnkMP4Ecx+EHP/gBy5cvxzRNPM/jmmuu4YYbbkj7Czpc2fvvv59f/epXQGK6w7Bhw7j11luZNm3ayah+r6UXFKEXl6K0LeB5aKEwRqgYzTCIf7CR+J6dqN0aVtWoVBlfv4G4jftw21oxCosSU4dcB6O0PyhQjoMWDOB5LrpftmbrS37yk4fY07yRWLyN/e0fYhp+bDdCWVEVQ/pNwvVsdjdvxGf4+d1vnmPRj75DW2wfxcGKLntVJgd2HDdGc8TG81x8hl8ew0+0K6+8kpdffplXXnmFYcOGsX79eqZNm0Z9fT0LFizosezXv/51li5dyosvvsg555yD67p861vfYvr06bz66quceeaZJ+lT9D6aYVAw7WL8HzsHTTfwFfdLjIBHI+C6WCPHJU5tHD4y0ScJKCeGVTXqkLTEOT3KdfE62rBGT0DTkAGePiZ9XqVBQ9tW/L5CgNTUH59uJdaRA2XhkRhtVuoMn86txs7LKAeXjkcpN6+mDmlKKXWy33T16tVMnTqVJUuWcOWVV6bSb7nlFu655x42b97M8OHDD1s+FAoxffp0nn322VRaY2Mj/fv35z//8z+57777jliHxK7hJ/2jn1CHLnvs3LeYuuY4aObBa5qm4UY6ug2CPd1P9D1xz6Zu/9sA+Ax/qtWYnDNp+QpwvTh7mjfieom9KvNlWlBST3EhK5v/Ll++HICZM2empc+aNQvHcXjiiSd6LB8Oh3Gc9H0Wk6/9fv9xrGnf0nk5Y7JvEUht6usbXo1v6AjMU8akBT09EEyMorsubnsbbnvbwY2Au7mf6N085aadj5N8rWsGQ/pNYlDJGMrCI1P7WHbe59JxI5QUDqEsXJXaKT0WbyMWb8ur83a6k5XH8LVr12JZFkOHDk1Lr66uTl3vycKFC7n++ut55plnuPTSS+no6ODb3/42JSUlXHXVVSes3r1d8szvzpPHk61DLxolvudDfOVDux3RVq5LbONb2B9sQtMU5ojRWNXjutxP9G6HThgvC4+kvuX9tAnkhm4d6HuM0hypozhYkQqW7+z6C6BRFjqF8uLR7GneyN6W9/FUnIHh0QwpHZ9XLc3OshIs6+vrCYVCXQZywuFw6npPvvrVr+Lz+bjqqquYO3cu+/fvZ9q0aaxZs4aqqqrDlqutraW2tjatHrlGlZSB44BpojU2omJRvL31oOt49Q3ohcXQ2oJeuAvMxNSQvXt2g+Pg7qpDdXQkJpbsrsMI94ND7id6t7gbpaGj/sCu583EO0yaogdfEy0CoKGjHl3TaOzYh+aEcN3EJs/t7e1oaDTYu/EiARraP2Rv+1Y85dHe1oYeC6d2H8o3WQmWSqkjjnj35JprruHJJ5/kqaee4rzzzqO9vZ1vfvObzJw5k2effZbx48d3W27u3LnMnTsXSPRNlJWVHXMd+grlusRaGvHa24j374cZCqH7/ZgDy3G2vAdAcVM95iljsFsbsaNtaBqYgyrwVwxO3OPAYI/0V/Z+nnKhue1An2OYsvBwfC1O6nV5ceLvlOY2HDeGZ/anOBhC1/sBUFSYCKYDQoMoLx6O19hMs2th6CZFgRIGlPXH7yvK1sfLqqwEy7KyMrZt29YlaLa0tKSuH87KlSupra1l0aJFnHfeeQAUFhbywAMPMGTIEK6++mpeffXVE/sB+hotceCYOfQUrFNGHeiDTPRHQmJqEJ6Lf/REzOEjAVKj4olH+BiaRmptuei9Oo9+d3596KYX3Y1qA4wdPB1I7LYOMLC4GoWb2gPTzIP5lIeTlWA5ZcoUVq9ezc6dOxk2bFgqffPmxNGdNTU1hy2b7M8cO3ZsWrplWVRVVfG3v/0Nz/PQdTm4Eg5MOI+7GOGSRN/jgSWRkOjfBFL9kZphJOZZHuBFI3jRGO7eOrxIG5BYWy4tzN6voXUrttuBZRRQUTKuS5A73ObAyVZj575P0yhkQKgS0wjmbX8lZGk0fPbs2QCsWrUqLX3lypWYpslll12WSqurq0vtqQgwaNAgALZu3ZpW1vM8tm/fTv/+/SVQdtLdoA8c3HgDOOyUIM20EueMRxLHUCiFjIj3AY4bob51C62RvTS0bkm1Mo/G4c4W7zzKnm+yElWmTp3KnDlzWLBgATt37gRgw4YNLF68mJtvvpnKykoAVqxYweDBg5k3b16q7Gc/+1lOOeUU7r33Xt59910gESjnz5/P7t27+cY3vnHyP1AvlgyK3Z31nfzz4VqKmmFgjZ2MNWIURlkFekBW7/QdBzZ0PkbJyepOPIqhm6mzxXc3b0w7WzyfZG0FzyOPPMKiRYuYMWMGlmXhui4LFy5MC4wlJSWEQiEqKg6eWVxUVMRrr73GD3/4Qz772c+i6zqO4zBw4EB+97vfpbVKRcJHOetbtyz8E06TQZ4+xDSClIVG4sQ7MH0Fx9TPeGhfZ+eWZj5ty9ZZVlbw9Aa5uILnWBz6c0hORpfA2Lcdy07mPf2bSPZh5vrKnp5+BhIsc0R3QS6Zhm4kdj0/8P/OeZI/B+W6eNFIagu3zssbJYD2LclAqWlGt+u3DxdID/03cWi+I903F/QUF2SLthzQ3RpuIG3ljlFWgVtf1+0KntQqn7Y24nt2Yo0cl5ha5NiAJevD+5CDOwMdXJ3TefegTI+EOGy+A6t/8u1ICcjSAI84vrpbw506HtdnoiJRNEBFoml7UnYpXxRGKQ23rfXgcklZH96nJPsWDd1H3I1i6D5cz8H17LTrli+Yln64+1i+II4bI+q0pFqWmZTPRdKyzAGHnR5kmYkJ5cEACu2we1Imyx+6VVvn+ZiyPrxvSI5ix90YPiOA67mYnfacPHSUu7u9KD3l4h2YLhRzOmiO7AIUphGgLDwSn24dOO0RtDxpVYL0WWa7GsfN8eizPFy/pGfbeB2JuZaygqf3+yh9lskt2uKeja75KC4op7FtB36zENvpoH+oEp8RYE/Te6gDATSXHsWlzzIPdDc96GimDB0ur3JdnC3vSZ9lH3K41TmZXD90ipBpBDCNALbTQVNkF4rEEmWF6naD4FwmwVL0qHOfpReLyPk7Oe7Qx3TTCFJePJqo05IKkDGnAw2tx0f5XCTBMocdjyk/yf5MN9Ke2LpNl1ZlLjvcxhsBM4xpBA60Nv2UhUfm7PShw5E+yxyV6ZEQmfwcPNvGfnd94kzxQEAexXPUkb4LyR3XgZzdVEP6LPPQcX189lzQdIygPIrnu8a2HXk5xxJknmXOOtx0omzfS/Rd+TzHEuQxPEs28XwAAB9DSURBVNvVOKEy6bPM9OcgSx5z3+G+C52nIiVWB0UwjSAVJeNyrmUpj+F5pnNgS+54/lF9lJ2LRN/VedmjrvnwlHcgmBz79m99lQTLHCNnfYvjqfOjdyTWjNIUBf6SvJpfmSR9ljnmo6zlTu48pNz829hVdK/zvEvTV4BlFOTd/MokaVnmmGMdjJEWaX7qvPSxO4fOuwSOep/MXCHBMsckj5FI9llC4uCx5J8Pu/47GsFra0MvCuNFI7itzRih4lQ+WR/ed8Q9GzvejuUrxHeYjTJcz0YBe5reS22acWieznMqk7ulG7qFaQRT1/MpaEqwzEHJwZjOrUUMI9EnH3e7tByV6+Jse5/4np2oOsCXSHcP7HupXJf2l/6AikTQgkEKL7hEAmYvFfds3tm1CsdNrOseN3hGWsDsvN9lY/t24m6MWLydoBlOXQeoa3qH+tYtgKJ/0SnomoanXHy6RVl4JPUt7+fdfEvps8xhaRPTIx2ojo5u+zKVY4PrYo0chzFgIL7+5RgFhal8XkcbKhLBCBWjolG8jrYsfirREzvejuNGCVoh4m4MO96edr3zfpeeF0fXfdjxNnzGwUds17Ox3Q50TUPXDWJ2S+JY3QPzK+14e17Ot5SWZQ7r3H+pBwtAo9u+zIP7WdoY4ZIu+XTdQAsGcdta0AIB9IKiHt5VZJPlK8Q0AkTtNnyGH8tXmHY9NWDjxjB9QUL+geiaQXHB4NR1AMsoSEwRUgq/FUbXDm6ckXy8z7eBHpmUnuM6z7mErn2Wh+5niW6kWp0HNwCWPsu+JNM+y+R+l8n/W76C1L+JQ/ssIX1g51gOROsL5MCybuRLsDySzj8HGRHPb/JvQlbwiAM0rftVF92lRzaskQ0zhOhEgmUe6e43Zn19PWVlZYnryZal48iGGUIcQoKlSDl0jqY8ggtxkARLkUY2zMhvuTpwczxkbZ6l4zgsWLCAcePGMXnyZCZOnMiDDz6YcQdzfX09N954I6eddhqnnnoqw4cP58ILL+TPf/7zCa65ELlrT/NGdjdvZE/zxtQEdZGQtZbllVdeycsvv8wrr7zCsGHDWL9+PdOmTaO+vp4FCxb0WHbXrl1MnTqV6667jr///e8YhkF9fT3nn38+b7zxBhdddNFJ+hRC5JbOJzvm265CR5KVqUOrV69m6tSpLFmyhCuvvDKVfsstt3DPPfewefNmhg8fftjyn/3sZ2lvb+ePf/xjWvqGDRuwLIsxY8YcsQ4yTSKh8wCPyG+aprFr/9u4noOhm3mzjLGznuJCVh7Dly9fDsDMmTPT0mfNmoXjODzxxBOHLbtr1y6efvppPv3pT3e5NmnSpIwCpRCie+XFoykvHpWXgfJIshIs165di2VZDB06NC29uro6df1wVq9ejed5hMNh5s2bR01NDdXV1XziE5/gpZdeOpHV7rN62qdSebKHZa5LrsY5lj7Ij1I212QlWNbX1xMKhbpMhg6Hw6nrh7Nt2zYA5s6dy6RJk/j73//Ohg0bqKys5KKLLuL3v//9iat4H5ScO2lveicxh7JTUFSui9qysdtrIjckdxnKdNCmc964Z8uATydZGeBRSh12NcmRRCKJ9apnnHEGc+fOBcDn83Hffffx5JNP8q1vfavbR3SA2tpaamtrU697Csq5QsWieHvr0QIBVHMTeuEuNH8gdS0ejdJs+LpcE7kh7kZp6KjHNPzE3WaIFuEzAgwcOLDb/INLx3dJe2fra2ll81VWgmVZWRnbtm3rEjRbWlpS1w8n2fo87bTT0tIty+LUU09l5cqV7Nu3j/79+3cpO3fu3FSA1TQtLwY2lOtitzclVuWEQ1gVg9P2sdy7cwvFwUCXayI3eMqF5rYDgzZhyosHo2tGt4MYe/buwbMaUwM8yX0rDy2br7ISLKdMmcLq1avZuXMnw4YNS6Vv3rwZgJqamsOWHT8+8ZvP87wu14zkDjndXMtXPa3K0QwD7ZTRWMXFsmInRx16LERPwU7XdMoOyZtp2Xxw1H2W27dv59prr6W6upqqqioAFixYwJ/+9KeM7zF79mwAVq1alZa+cuVKTNPksssuS6XV1dWlBb/zzjuPAQMGsG7durSyruvy1ltvUVVVlRctxqOhGUbadmtp1/TDXxO5QdcMTCOYUbA7NO/RlM156ii8/fbbqrS0VAWDQTV27FhVWVmplFJqxYoVaujQoWrlypUZ32vOnDlqxIgRaseOHUoppdavX6/C4bC67bbbUnkef/xxBajrr78+rewvf/lLpWma+s1vfpNKmz9/vtI0TT355JMZvf9RfvSctXfv3mxXQfQS8l3oOS4cVcS49NJL1TXXXKOam5uVUkrV1NSkrq1bt05deOGFGd/Ltm11xx13qDFjxqhJkyap8ePHq/vvv195npfK88c//lGFQiG1cOHCLuWffPJJdcYZZ6iRI0eqyspKNX36dLVq1aqM31+CZYL8AxFJ8l3oOS4c1QqeqqoqPvjgg9TrKVOm8I9//CP1+swzz+Rvf/vbcWv1nkiygidBVvCIJPkuHMcVPD5fz+NB+TAVRwiRn44qWJaUlPDrX/+622u///3vGTRo0HGplBBC9DZHNXXoO9/5Dp/73OdYsmQJ5513Hvv27ePuu+9mzZo1PPXUUz2u6RZCiL7sqHcd+tnPfsbNN9+cmkAOiYnid999N1dfffVxr+CJIn2WCdJPJZJ6+i7ky6bAx/10x/b2dlavXp364Z577rkUFhYeuWAvIsEyQYKlSDrcdyG5vjzu2fh0K6d3JDpupzu+++67jB07lsLCQi6++OJU+ptvvsnzzz/P1772NUpKSj5abYUQvYrr2bIpMEc5wHPFFVd0m+73+3nzzTcPe10I0XcZuoVPt3DiUQzdxNDz89TPo2pZHq55OnbsWH7729/2uKZbCNE3yRrxhCMGy+3bt7N161Yg0Vf5yiuvdAmanuexY8eOtEEfIUTu0DUjLx+9OztisFy6dCl33nlnaiu1Cy64oEsepRS6rrNw4cLjXkEhhOgNjjgavm3bNrZu3YpSiquvvpolS5Z0yWOaJiNGjGDw4MEnrKLHWz6NhivXTW3RBqT9ub5uF2UVg9PSO+932d3WbqJvOXg0hIeu6YfdRehIMyPyYfrQRxoNr6yspLKyEoCvfOUrnH/++ce3duKESh4roWwHDAM0IO6CzwAF3r59xFoaU+maZWJVJ/YMTZZLpknA7Hs85VLX9C57WzbREt1DOFDOwPAoKkrGHlXAy6fpQ4dzVKPht9xyS4/XDz2aVmSfcmyU7aAHgniRDlRHB3ogiIp04EXa0QKB9HTHSZTpVC6ZJvoe17Ox4+2ASvyngRPvwPWO7u+z8/Qh13OOunwuOK4Hlh0pmIqTTzMtNMvEi0XQgwVoBQV4sQhasAA9WIiKRdPTTTNRplO5ZJroewzdwvIVAjpoGprSMH0FRz39R6YPHcMKnp/85Cf87Gc/Y/Pmzdh2198ubh85IVD6LKXPMl9In2XmjtsKnp/+9Kfcd999fO5zn+MXv/gF1157LQC7du1ixYoVfOYzn/notRXHnWYYaJ2mfaT92R9IBULtkKkhh5YTfZOuGfh9RcflPvk8feioWpann346Tz/9NEOHDu2y8e9bb73FXXfdxbJly05IRY+3fGpZ9kTWhosk+S4cx81/Xddl6NChqT93NnHiRHbu3HmMVRRCiN7tqIJl54jr9/tTK3sg8VvpvffeO24VE0KI3uSoguXQoUO59dZbicViTJs2jVmzZnHffffx4x//mOnTp6fO9BZCiFxzVH2Wf/jDH1i6dCl33303BQUFXHDBBbz77rsAlJWV8cwzz3D22WefsMoeT9JnmSD9VCJJvgsnYPPfpEgkwquvvopt25x77rlceOGFaYM+vZkEywT5ByKS5LtwHKcOHSoYDDJjxgwAHn/8cbZt2/ZRbieEEL3WEfsslVIsWrSI6upqSktL+exnP0tDQ0Pq+u9+9zsmTpzIFVdcgd/vP6GVFUKIbDlisFy8eDG33nor9fX1xONxfv/733PjjTeyZcsWPvaxj/H5z3+eSCTCQw89xJYtW05GnYUQ4qQ74mP4ww8/zL333stNN92EUopbb72VBx98kA8//JBoNMqyZcuYPXs2un5cl5kLIUSvcsQBnsrKyrS+yFgsRmFhId/+9rf53ve+l/UgGYlEGD9+fGrPzUzJAE+CdOqLJPkufMQVPIee1uj3+xk2bNhHDpSO47BgwQLGjRvH5MmTmThxIg8++OBRB7BFixalTY4XQogT4YiP4UY3u82UlpZ2Gygvuugi/vznP2f0xldeeSUvv/wyr7zyCsOGDWP9+vVMmzaN+vp6FixYkNE9Nm/ezM9+9jNmzJjBqlWrMiojhBDH4ojB0rZtduzYkdbi6y4NYM+ePRm96erVq3nsscdYsmQJw4YNA2Dy5MnccMMN3HXXXVx11VUMHz78iPf5+te/zre//W3++c9/ZvS+QghxrI4YLN9++21GjBiRlqaU6pJ2NJYvXw7AzJkz09JnzZrFD37wA5544gm+8Y1v9HiPp59+mi1btvDMM89w9dVXH3NdhMhHyb0pNc1AKbfbzXzzYf/Ko3HEYFleXp7at7InSilqa2szetO1a9diWVZqB6Ok6urq1PWeRCIRbrrpJmprazFNM6P3FEIkJM/TcdwozZE6ioMVmEYAXfXrkiefz9w51BGD5aBBg5g/f35GN3vmmWcyyldfX08oFEodr5sUDodT13vy/e9/n9NPP52LL744o/dLqq2tTQvoR3qffBCPx+XnkGfibpSGjno0TWN/xz50J4RSrRT7A6nvQjKPafiJu80QLcJnBLJc8+w6YrB86aWXMr5ZpnmVUl0CZaY2b97MQw89xJtvvnnUZefOncvcuXOBxBSBfJ8mATJdJB95yoXmNuJuDGX2JxQMYRp+dLsg9V1I5nE9B0MPU148WFqWR8pQXFyc8c0yzVtWVsa2bdu6BM2WlpbU9cOZN28e//Vf/5UaGBJCHB1dMygvHo3r2VSUjk/1We5raOw2j/RZJnykjTSO1ZQpU1i9ejU7d+5MC3qbN28GoKampttyLS0trF27lp07d/Lb3/42lb59+/a0cosWLeKSSy45UdUXos/L5DydfD9z51BZWX4ze/ZsgC5zI1euXIlpmlx22WWptLq6OjzPAxJ9mrt372bDhg2sW7cu9d8nP/lJgNRrCZRCiOMtK8Fy6tSpzJkzhwULFqTO7dmwYQOLFy/m5ptvprKyEoAVK1YwePBg5s2bl41qCiFESlYewwEeeeQRFi1axIwZM7AsC9d1WbhwYVpgLCkpIRQKUVFR0e09br/9dp599tkuj+Hr1q078R9ACJFXPtJO6X2ZbKSRIKPhIkm+C8fxKFwhhMhXEiyFECIDEiyFECIDEixFRpTr4kUjKNfNdlWEyIqsjYaLvkO5Lvbmt1G2g2aZWNXj0brZ51SIXCYtS3FEyrFRtoMeCKIcB+XY2a6SECedBEtxRJppoVkmXiyCZppoZte9D4XIdfIYLo5IMwys6vEox04ETnkEF3lIgqXIiGYYaLKpgshj8hguhBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZkGAphBAZyFqwdByHBQsWMG7cOCZPnszEiRN58MEHUUr1WC4SifCLX/yCmTNn8rGPfYxJkyYxadIk7rjjDlpaWk5S7YUQeUdlyZw5c1RlZaXavn27UkqpN998U4VCIXXbbbf1WG758uVK13W1YsWKVNqaNWtUSUmJmjJlirJtO6P3z+JH71X27t2b7SqIXkK+Cz3Hhay0LFevXs1jjz3GbbfdxrBhwwCYPHkyN9xwA3fddRfbt2/vsfzMmTO5/PLLU6+nTJnCddddxz/+8Q/+93//94TWXQiRn7ISLJcvXw4kgl5ns2bNwnEcnnjiicOWvfzyy3nuuee6pCeD7v79+49jTYUQIiErpzuuXbsWy7IYOnRoWnp1dXXq+uHouo6ud43x77zzDqZpct555x3fygohBFlqWdbX1xMKhdA0LS09HA6nrh+NxsZGfvOb3/DNb34z1cIUQojjKSstS6VUl0B5rFzX5atf/SpnnXUWCxcu7DFvbW0ttbW1qddHG5RzUTwel5+DAOS7cCRZCZZlZWVs27atS9BMTv0pKyvL6D6e53HllVcSjUZ5+umn8fl6/jhz585l7ty5AGialvH75LL6+nr5OQhAvgtHkpXH8ClTpmDbNjt37kxL37x5MwA1NTVHvIfjOHzhC1+gvb2dZ599lkAgcELqKoQQkKVgOXv2bABWrVqVlr5y5UpM0+Syyy5LpdXV1eF5Xlq+SCTCpz71KYqKinj88cexLAuA559/njvvvPME114IkY+yEiynTp3KnDlzWLBgQap1uWHDBhYvXszNN99MZWUlACtWrGDw4MHMmzcvVba1tZWPf/zjbN26lenTp7N8+XKWLVvGsmXLeO6559iyZUs2PpIQIsdlpc8S4JFHHmHRokXMmDEDy7JwXZeFCxemBcaSkhJCoRAVFRWptKVLl/LXv/4VgDlz5nS573/8x3+c+MoLIfKOdmCJT97RNO2I69DzgXTqiyT5LvQcF2TXISGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyIAESyGEyEDWgqXjOCxYsIBx48YxefJkJk6cyIMPPohSKqPyq1at4txzz2XixImMHj2aL3/5y+zdu/cE11oIka982XrjK6+8kpdffplXXnmFYcOGsX79eqZNm0Z9fT0LFizosezKlSu55JJLePTRR/nSl75ENBrlU5/6FBdccAFvvPEGhYWFJ+lTCCHyhsqCV199VQFqyZIlaenf/e53lWmaatu2bT2WHz16tJo2bVpa2rvvvqsAtWjRoozqkKWP3uvs3bs321UQvYR8F3qOC1l5DF++fDkAM2fOTEufNWsWjuPwxBNPHLbsmjVr2LhxY5eyY8aMYcSIEal7i54p18WLRvDiNl40gnLd9HQ7PV30PZ5ycdwIcc8mFm8jFm/DU13/Pg/mi+O4kW7zANhuhOZIHfaBPD3lzUVZeQxfu3YtlmUxdOjQtPTq6urU9Z7KAowcObLLterqal588UVs28ayrONY49yiXBd789t40RjepneIjRqLHghgnjIGZ8t7eNEo8T0f4isfih7wY1WPRzOMbFdbHAVPuexp3ojjRmnq+BDXc9E0jbLQSCpKxqJrRpd8Oxs34vhGYxoByotHp/JAIlC+/v5vcNwOfHqAEWVnoKHh060ueXNVVoJlfX09oVAITdPS0sPhcOp6T2U75z20vOu6NDY2MmjQoC7Xa2trqa2t7XKvfKNiUby99aBrxCMdNLe1Q2srmi+A2lsPuo5X34BeWAytLeiFu9D8gWxXWxyFuBuloaMeTdNobNuLoZvouolm16HHwviMQFo+XdOIORFaWtpAtUK0KJUHoC22l+bWRoJWiGa7gT1sIxQYRNxt7pI3V2UlWCqlugTKoykLHFP5uXPnMnfu3FT5srKyY6pDX6dcF7u9CS8WoylYQHFREXrAjzmsEicexYvGiJcNwBcKofv9WBWDpWXZx3jKheY24m4MzzcQV7no6AwIVVBeMjitZUlzG44bozkapDgcwmf4KS8enNZaLHaL2NnRj7gbodgaQHlZJTo6hh7ukjdXZSVYlpWVsW3bti5Bs6WlJXW9p7IAzc3NXa61tLRgGAalpaXHuca5RTMMrOrxKMdGL6vAX9oPzbTS0v3jJoPnptJF36JrBuXFo3E9m4rS8bieDYBpBNMCW+d8ZrycASUlGLrVJfhZRpCzRl5BxG4iaJXg0y1cz+42b67KygDPlClTsG2bnTt3pqVv3rwZgJqamh7LArz//vtdrm3evJlx48bh9/uPY21zk2YY6IEgus9CDwRTATGVbqWni75H1wxMI4hPt/D7ivD7iroNbAfz+boE084sI0hxsALrQJ6e8uairATL2bNnA4mJ5Z2tXLkS0zS57LLLUml1dXV4npd6PWXKFEaNGtWl7KZNm9i6dStXXHHFCay5ECJvnaz5S4eaM2eOGjFihNqxY4dSSqn169ercDisbrvttlSexx9/XAHq+uuvTyv7pz/9SRmGoR577DGllFKRSER9/OMfV2PHjlVtbW0ZvX8WP3qvInPrRJJ8F3rhPEuARx55hC9/+cvMmDGDyZMnM3v2bBYuXMidd96ZylNSUkIoFKKioiKt7MyZM3nhhRd46KGHmDhxIpMmTaKsrIyXXnpJVu8IIU4I7UA0zTuapmW8Dj2X1dfX5+2sAJFOvgs9xwXZdUgIITIgwVIIITIgwVIIITIgwVIIITKQtf0se4NjXXIphMg/eRssZSQ84fTTT+fvf/97tqshegH5LvRMHsOFECIDEiyFECIDEizzXHLLOiHku9CzvF3BI4QQR0NalkIIkQEJljnsrLPOYtCgQTJFSojjQIJlDnv99de59tprs10NcRI1NTVxxx13sG7duhP2HuvWreOOO+6gqanphL1HbyTBUogc0tTUxJ133nnCg+Wdd94pwVIIIURXEixzyB/+8AdOO+00Bg0axFlnncX8+fPTjuRIWrduHZ/4xCeorKykurqas88+m+effz51ffr06ViWRVFRETU1NezYsQOA22+/nREjRlBaWspNN9100j6XyMzSpUu55JJLgMTfVU1NDTU1NakjWF588UUuuOACTjnlFKqqqrjoootYvXp12j327NnDV77yFSZMmMCUKVOoqanhhhtu4IMPPgDga1/7GrfffjsAl1xySeo98qKVeTK2ahcn3qpVq5RhGOqmm25SnucppZRavny5Ki8vT9sqf82aNaqgoEBdddVVKh6PK6WUeuyxx5Smaeqpp55K5fvCF76gQqFQl2M6rrvuOvWTn/zkJHwicSy2bNmiALV06dK09Oeee04ZhqHuvPPOVNr3vvc9ZVmWeu2111JpF198sbr44ouVbdtKKaXq6urUqFGj0u63dOlSBagtW7acyI/S60iwzBFnnXWWKi0tVZFIJC39/PPPTwuW06dPV4WFhaq5uTkt39lnn61GjRqVev2Xv/xFAWrJkiWptPb2dlVRUaGamppO0KcQH1V3wdLzPFVVVaVGjBiR+gWplFLxeFwNHTpUXXzxxam0wsJCddVVV6Xd8+mnn1avv/566nW+Bkt5DM8B7e3tvPHGG5x66qkEAoG0a52PFe7o6ODll1/mtNNOIxwOp+U79dRT2bRpE9u3bwcSj+KjRo3i4YcfTuVZsWIFs2bNori4+AR+GnG8bdq0iQ8++IBp06ZhdDra2DAMJkyYwF//+lccxwES51s98sgjXH755Tz//PNEIhEuvfRSzjzzzGxVv9fI212Hcsn+/fvxPI9+/fp1uVZSUpKWz3Vd3nrrrS5ns7e1tVFeXk59fT3Dhw8H4KqrruI73/kO69evZ/LkydTW1nLvvfee2A8jjrv6+nogcdT0oX/vzc3NlJaWsn//fgYOHMjjjz/OT37yEx599FH+7d/+jaKiIubMmcNdd93V5RdsvpFgmQNKS0vRdZ19+/Z1ubZ///60fIZhcPbZZ/PCCy8c8b5f+cpXuO2223j44Ye55ppraGlp4dxzzz2udRcnXvIQss985jP89Kc/7TGvZVncdNNN3HTTTfzzn//k5z//OYsXL6a1tZXHHnvsZFS315LH8BxQWFjIWWedxZtvvkk0Gk279uabb6b+XFBQwPnnn8/69euJx+Np+d5++22uuOKKtPSysjIuvfRSli1bxv3338/VV199Yj+I+MhM0wRIzYLYtm0bDQ0NVFVVsWbNmi75X3rpJb72ta+lXs+ePTv15wkTJvDAAw/wr//6r2nfo0PfY82aNWzcuPH4f5heRoJljli0aBGtra3ccsstqY2NH3/88S7/QO655x6am5u5+eabcV0XgIaGBq699lpGjBiBz5f+sHH11VfT1NTEsmXLmDNnzsn5MOKYlZeXU1BQkJrutXjxYh555BEWL17M2rVruf/++1N5t27dynXXXcfkyZNTaStWrGDZsmWp13v27OGf//wnM2fOTKVVVVUBsGPHDpRSXH/99bz++usn+qNlX7ZHmMTx88ILL6iamho1cOBANWXKFHXjjTeq7373uwpQp556qnr00UeVUkqtX79effKTn1RDhgxRNTU16vTTT1f3339/aspRZ8mR1C984Qsn++OIY/TII4+oESNGqMmTJ6upU6eq999/Xyml1Msvv6wuvPBCNXToUDVlyhR1zjnnqF//+tdpZe+55x517rnnqkmTJqmamho1ceJENX/+/NRUoqTrrrtOVVZWqgkTJqjPf/7zKhaLnbTPly2yRZsQQmRAHsOFECIDEiyFECIDEiyFECIDEiyFECIDEiyFECIDEiyFECIDEiyFECIDEiyFECIDEiyFECIDEiyFECID/z9JmFNIBbUL8wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "utils.plot_worker_agreement(\n",
    "    dev, test, os.path.join('img', 'human-agr-round2.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples for the paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>model_1_label</th>\n",
       "      <th>responses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The place was somewhat good and not well</td>\n",
       "      <td>negative</td>\n",
       "      <td>mix, mix, mix, mix, neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I bought a new car and met with an accident.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neg, neg, neg, neg, neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>The retail store is closed for now at least.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neu, neu, neu, neu, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Prices are basically like garage sale prices.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neg, neu, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>That book was good.  I need to get rid of it.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>mix, mix, mix, neg, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I REALLY wanted to like this place</td>\n",
       "      <td>neutral</td>\n",
       "      <td>mix, neg, neg, neg, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>But I'm going to leave my money for the next vet.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>neg, neu, neu, neu, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>once upon a time the model made a super decision.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>pos, pos, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I cook my caribbean food and it was okay</td>\n",
       "      <td>positive</td>\n",
       "      <td>mix, mix, mix, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>This concept is really cool in name only.</td>\n",
       "      <td>positive</td>\n",
       "      <td>mix, neg, neg, neg, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Wow, it'd be super cool if you could join us</td>\n",
       "      <td>positive</td>\n",
       "      <td>neu, neu, neu, neu, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Knife cut thru it like butter! It was great.</td>\n",
       "      <td>positive</td>\n",
       "      <td>pos, pos, pos, pos, pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             sentence model_1_label  \\\n",
       "0            The place was somewhat good and not well      negative   \n",
       "3        I bought a new car and met with an accident.      negative   \n",
       "6        The retail store is closed for now at least.      negative   \n",
       "9       Prices are basically like garage sale prices.      negative   \n",
       "1       That book was good.  I need to get rid of it.       neutral   \n",
       "4                  I REALLY wanted to like this place       neutral   \n",
       "7   But I'm going to leave my money for the next vet.       neutral   \n",
       "10  once upon a time the model made a super decision.       neutral   \n",
       "2            I cook my caribbean food and it was okay      positive   \n",
       "5           This concept is really cool in name only.      positive   \n",
       "8        Wow, it'd be super cool if you could join us      positive   \n",
       "11       Knife cut thru it like butter! It was great.      positive   \n",
       "\n",
       "                  responses  \n",
       "0   mix, mix, mix, mix, neg  \n",
       "3   neg, neg, neg, neg, neg  \n",
       "6   neu, neu, neu, neu, neu  \n",
       "9   neg, neu, pos, pos, pos  \n",
       "1   mix, mix, mix, neg, pos  \n",
       "4   mix, neg, neg, neg, pos  \n",
       "7   neg, neu, neu, neu, neu  \n",
       "10  pos, pos, pos, pos, pos  \n",
       "2   mix, mix, mix, pos, pos  \n",
       "5   mix, neg, neg, neg, neu  \n",
       "8   neu, neu, neu, neu, pos  \n",
       "11  pos, pos, pos, pos, pos  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.sample_short_examples(train, 'model_1_label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>model_1_label</th>\n",
       "      <th>responses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The menu was appealing.</td>\n",
       "      <td>positive</td>\n",
       "      <td>pos, pos, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Our food took forever to get home.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neg, neg, neg, neg, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Our table was ready after 90 minutes and usually I'd be mad but the scenery was nice so it was an okay time.</td>\n",
       "      <td>positive</td>\n",
       "      <td>mix, mix, mix, mix, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I left feeling like an initiator in a world of books that I never even knew existed.</td>\n",
       "      <td>positive</td>\n",
       "      <td>mix, neu, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I decided to go with josh wilcox, only as a last resort</td>\n",
       "      <td>neutral</td>\n",
       "      <td>mix, neg, neu, neu, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>: ] Standard ask if the food is a great happening!</td>\n",
       "      <td>neutral</td>\n",
       "      <td>neu, neu, neu, neu, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>The car was really beautiful.</td>\n",
       "      <td>positive</td>\n",
       "      <td>neu, pos, pos, pos, pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Food is a beast of its own, low-quality ingredients, typically undercooked, and a thin, very simple menu.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neg, neg, neg, neg, neu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>I want to share a horrible experience here.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neg, neg, neg, neg, neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>I tried a new place. The entrees were subpar to say the least.</td>\n",
       "      <td>negative</td>\n",
       "      <td>neg, neg, neg, neg, neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                       sentence  \\\n",
       "0                                                                                       The menu was appealing.   \n",
       "1                                                                            Our food took forever to get home.   \n",
       "2  Our table was ready after 90 minutes and usually I'd be mad but the scenery was nice so it was an okay time.   \n",
       "3                          I left feeling like an initiator in a world of books that I never even knew existed.   \n",
       "4                                                       I decided to go with josh wilcox, only as a last resort   \n",
       "5                                                            : ] Standard ask if the food is a great happening!   \n",
       "6                                                                                 The car was really beautiful.   \n",
       "7     Food is a beast of its own, low-quality ingredients, typically undercooked, and a thin, very simple menu.   \n",
       "8                                                                   I want to share a horrible experience here.   \n",
       "9                                                I tried a new place. The entrees were subpar to say the least.   \n",
       "\n",
       "  model_1_label                responses  \n",
       "0      positive  pos, pos, pos, pos, pos  \n",
       "1      negative  neg, neg, neg, neg, neu  \n",
       "2      positive  mix, mix, mix, mix, pos  \n",
       "3      positive  mix, neu, pos, pos, pos  \n",
       "4       neutral  mix, neg, neu, neu, neu  \n",
       "5       neutral  neu, neu, neu, neu, pos  \n",
       "6      positive  neu, pos, pos, pos, pos  \n",
       "7      negative  neg, neg, neg, neg, neu  \n",
       "8      negative  neg, neg, neg, neg, neg  \n",
       "9      negative  neg, neg, neg, neg, neg  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils.sample_examples(train, 'model_1_label')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}