worked-solutions/02_penguin_regression_solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2: Penguin regression with PyTorch\n",
    "\n",
    "<img src=\"https://allisonhorst.github.io/palmerpenguins/reference/figures/lter_penguins.png\" width=\"750\" />\n",
    "\n",
    "\n",
    "Artwork by @allison_horst\n",
    "\n",
    "In this exercise, we will again use the [``palmerpenguins``](https://github.com/mcnakhaee/palmerpenguins) data to continue our exploration of PyTorch.\n",
    "\n",
    "We will use the same dataset object as before, but this time we'll take a look at a regression problem: predicting the mass of a penguin given other physical features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1: look at the data\n",
    "In the following code block, we import the ``load_penguins`` function from the ``palmerpenguins`` package.\n",
    "\n",
    "- Load the penguin data as you did before.\n",
    "- This time, consider which features we might like to use to predict a penguin's mass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       bill_length_mm  bill_depth_mm  flipper_length_mm  body_mass_g  \\\n",
      "count      342.000000     342.000000         342.000000   342.000000   \n",
      "mean        43.921930      17.151170         200.915205  4201.754386   \n",
      "std          5.459584       1.974793          14.061714   801.954536   \n",
      "min         32.100000      13.100000         172.000000  2700.000000   \n",
      "25%         39.225000      15.600000         190.000000  3550.000000   \n",
      "50%         44.450000      17.300000         197.000000  4050.000000   \n",
      "75%         48.500000      18.700000         213.000000  4750.000000   \n",
      "max         59.600000      21.500000         231.000000  6300.000000   \n",
      "\n",
      "              year  \n",
      "count   344.000000  \n",
      "mean   2008.029070  \n",
      "std       0.818356  \n",
      "min    2007.000000  \n",
      "25%    2007.000000  \n",
      "50%    2008.000000  \n",
      "75%    2009.000000  \n",
      "max    2009.000000  \n",
      "Index(['species', 'island', 'bill_length_mm', 'bill_depth_mm',\n",
      "       'flipper_length_mm', 'body_mass_g', 'sex', 'year'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "from palmerpenguins import load_penguins\n",
    "\n",
    "data = load_penguins()\n",
    "\n",
    "# Note: ``pd.DataFrame.describe`` is a useful function for giving an overview\n",
    "# of what a ``pd.DataFrame`` contains.\n",
    "print(data.describe())\n",
    "\n",
    "print(data.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now discuss the features we will use to classify the penguins' species, and populate the following list together:\n",
    "\n",
    "#### Let's use:\n",
    "\n",
    "- ``\"species\"``\n",
    "    - Perhaps the most relevant aspect from which to estimate mass.\n",
    "- ``\"sex\"``\n",
    "    - Biologically relevant.\n",
    "- ``\"bill_length_mm\"``\n",
    "    - Biologically relevant.\n",
    "- ``\"bill_depth_mm\"``\n",
    "    - Biologically relevant.\n",
    "- ``flipper_length_mm``\n",
    "    - Biologically relevant.\n",
    "\n",
    "#### Let's reject\n",
    "- ``\"island\"``\n",
    "    - While island could be predictive if dominated by a particular species it would be acting as a proxy and we have already included species as an input feature. \n",
    "- ``\"year\"``\n",
    "    - This feature could also be important: then behaviour of certain species may be changing in response to time-dependent environmental factors such as melting ice. It does however seem like the least biologically-relevant feature, and the most likely source of bias, so we reject it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2: creating a ``torch.utils.data.Dataset``\n",
    "\n",
    "As before, we need to create PyTorch ``Dataset`` objects to supply data to our neural network.  \n",
    "Since we have already created and explored the ``PenguinDataset`` class there is nothing else to do here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3: Obtaining training and validation datasets.\n",
    "\n",
    "- Instantiate the penguin dataloader.\n",
    "  - Make sure you supply the correct column titles for the features and the targets.\n",
    "  - Remember, the target is now mass, not the species!\n",
    "- Iterate over the dataset\n",
    "    - Hint:\n",
    "        ```python\n",
    "        for features, targets in dataset:\n",
    "            # print the features and targets here\n",
    "        ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.0, 42.9, 13.1, 215.0) (5000.0,)\n",
      "(0.0, 46.1, 13.2, 211.0) (4500.0,)\n",
      "(0.0, 44.9, 13.3, 213.0) (5100.0,)\n",
      "(0.0, 43.3, 13.4, 209.0) (4400.0,)\n",
      "(0.0, 42.0, 13.5, 210.0) (4150.0,)\n",
      "(0.0, 46.5, 13.5, 210.0) (4550.0,)\n",
      "(0.0, 44.0, 13.6, 208.0) (4350.0,)\n",
      "(0.0, 40.9, 13.7, 214.0) (4650.0,)\n",
      "(0.0, 42.6, 13.7, 213.0) (4950.0,)\n",
      "(0.0, 42.7, 13.7, 208.0) (3950.0,)\n",
      "(0.0, 45.3, 13.7, 210.0) (4300.0,)\n",
      "(0.0, 47.2, 13.7, 214.0) (4925.0,)\n",
      "(0.0, 45.2, 13.8, 215.0) (4750.0,)\n",
      "(0.0, 43.6, 13.9, 217.0) (4900.0,)\n",
      "(0.0, 43.8, 13.9, 208.0) (4300.0,)\n",
      "(0.0, 45.5, 13.9, 210.0) (4200.0,)\n",
      "(0.0, 45.7, 13.9, 214.0) (4400.0,)\n",
      "(0.0, 43.3, 14.0, 208.0) (4575.0,)\n",
      "(0.0, 47.5, 14.0, 212.0) (4875.0,)\n",
      "(0.0, 46.2, 14.1, 217.0) (4375.0,)\n"
     ]
    }
   ],
   "source": [
    "from ml_workshop import PenguinDataset\n",
    "\n",
    "features = [\n",
    "    \"sex\",\n",
    "    \"bill_length_mm\",\n",
    "    \"bill_depth_mm\",\n",
    "    \"flipper_length_mm\",\n",
    "]\n",
    "\n",
    "data_set = PenguinDataset(\n",
    "    input_keys=features,\n",
    "    target_keys=[\"body_mass_g\"],\n",
    "    train=True,\n",
    ")\n",
    "\n",
    "for _, (input_feats, target) in zip(range(20), data_set):\n",
    "    print(input_feats, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4: Applying transforms to the data\n",
    "\n",
    "As in the previous exercise, the raw inputs and targets need transforming to ``torch.Tensor``s before they can be passed to a neural network.  \n",
    "We will again use ``torchvision.transforms.Compose`` to take a list of callable objects and apply them to the incoming data.\n",
    "\n",
    "Because the raw units of mass are in grams, the numbers are quite large. This can encumber the model's predictive power. A sensible way to address this is to normalise targets using statistics from the training set. The most common form of normalisation is to subtract the mean and divide by the standard deviation (of the training set). However here, for the sake of simplicity, we will just scale the mass by dividing by the mean of the training set.\n",
    "\n",
    "Note that this means that the model will now be trained to predict masses as fractions of the training mean.\n",
    "\n",
    "We grab the mean of the training split in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set = PenguinDataset(features, [\"body_mass_g\"], train=True)\n",
    "\n",
    "training_mean = train_set.split.body_mass_g.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create our real training and validation set, and supply transforms as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([  0.0000,  42.9000,  13.1000, 215.0000]) tensor([1.1856])\n",
      "tensor([  0.0000,  46.1000,  13.2000, 211.0000]) tensor([1.0670])\n",
      "tensor([  0.0000,  44.9000,  13.3000, 213.0000]) tensor([1.2093])\n",
      "tensor([  0.0000,  43.3000,  13.4000, 209.0000]) tensor([1.0433])\n",
      "tensor([  0.0000,  42.0000,  13.5000, 210.0000]) tensor([0.9840])\n"
     ]
    }
   ],
   "source": [
    "from torch import tensor, float32, eye\n",
    "from torchvision.transforms import Compose\n",
    "\n",
    "\n",
    "# Apply the transforms we need to the PenguinDataset to get out inputs\n",
    "# targets as Tensors.\n",
    "\n",
    "\n",
    "def get_input_transforms() -> Compose:\n",
    "    \"\"\"Return transforms which map from raw inputs to tensors.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    Compose\n",
    "        A composition of transforms (callable functions) to map the tuple\n",
    "        of input features (``Tuple[float, ...]``) to a ``torch.Tensor``.\n",
    "\n",
    "    \"\"\"\n",
    "    return Compose([lambda x: tensor(x, dtype=float32)])\n",
    "\n",
    "\n",
    "def get_target_tfms() -> Compose:\n",
    "    \"\"\"Return transforms which map from the raw targets to tensors.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    Compose\n",
    "        A composition of transforms (callable functions) to map the tuple\n",
    "        of target features (``Tuple[str]``) to a ``torch.Tensor``.\n",
    "\n",
    "    Notes\n",
    "    -----\n",
    "    Note the rescaling by dividing out the training mean.\n",
    "\n",
    "    \"\"\"\n",
    "    return Compose([lambda x: tensor(x, dtype=float32) / training_mean])\n",
    "\n",
    "\n",
    "train_set = PenguinDataset(\n",
    "    input_keys=features,\n",
    "    target_keys=[\"body_mass_g\"],\n",
    "    train=True,\n",
    "    x_tfms=get_input_transforms(),\n",
    "    y_tfms=get_target_tfms(),\n",
    ")\n",
    "\n",
    "\n",
    "valid_set = PenguinDataset(\n",
    "    input_keys=features,\n",
    "    target_keys=[\"body_mass_g\"],\n",
    "    train=False,\n",
    "    x_tfms=get_input_transforms(),\n",
    "    y_tfms=get_target_tfms(),\n",
    ")\n",
    "\n",
    "\n",
    "for _, (input_feats, target) in zip(range(5), train_set):\n",
    "    print(input_feats, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5: Creating ``DataLoaders``—Again!\n",
    "\n",
    "As before, we wrap our ``Dataset``s in ``DataLoader`` before we proceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([16, 4]) torch.Size([16, 1])\n",
      "torch.Size([16, 4]) torch.Size([16, 1])\n",
      "torch.Size([16, 4]) torch.Size([16, 1])\n",
      "torch.Size([12, 4]) torch.Size([12, 1])\n"
     ]
    }
   ],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "batch_size = 16\n",
    "\n",
    "train_loader = DataLoader(\n",
    "    train_set,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=True,\n",
    "    drop_last=True,\n",
    ")\n",
    "\n",
    "valid_loader = DataLoader(\n",
    "    valid_set,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=False,\n",
    ")\n",
    "\n",
    "for batch, targets in valid_loader:\n",
    "    print(batch.shape, targets.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 6: Creating a neural network in PyTorch\n",
    "\n",
    "Previously we created our neural network from scratch, but doing this every time we need to solve a new problem is cumbersome.  \n",
    "Many projects working with the ICCS have codes where the numbers of layers, layer sizes, and other parts of the models are hard-coded from scratch every time!\n",
    "\n",
    "The result is ungainly, non-general, and heavily-duplicated code. Here, we are going to shamelessly punt Jim Denholm's Python repo, [``TorchTools``](https://github.com/jdenholm/TorchTools), which contains generalisations of many commonly-used PyTorch tools, to save save us some time.\n",
    "\n",
    "Here, we can use the ``FCNet`` model, whose documentation lives [here](https://jdenholm.github.io/TorchTools/models.html). This model is a fully-connected neural network with various options for dropout, batch normalisation, and easily-modifiable layers.\n",
    "\n",
    "#### A brief sidebar\n",
    "Note: the repo is pip-installable with\n",
    "```bash\n",
    "pip install git+https://github.com/jdenholm/TorchTools.git\n",
    "```\n",
    "but has already been installed for you in the requirements of this workshop package.\n",
    "\n",
    "It is useful to know you can install Python packages from GitHub using pip. To install specific versions you can use:\n",
    "```bash\n",
    "pip install git+https://github.com/jdenholm/TorchTools.git@v0.1.0\n",
    "```\n",
    "(The famous [segment anything model](https://github.com/facebookresearch/segment-anything) (SAM) published by Facebook Research was released in this way.)\n",
    "\n",
    "One might argue that this is a much better way of making one-off codes available, for example academic codes which might accompany papers, rather than using the global communal package index PyPI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Back to work: let's instantiate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FCNet(\n",
      "  (0): InputBlock(\n",
      "    (0): BatchNorm1d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (1): Dropout(p=0.1, inplace=False)\n",
      "  )\n",
      "  (1): DenseBlock(\n",
      "    (0): Linear(in_features=4, out_features=16, bias=True)\n",
      "    (1): BatchNorm1d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (2): Dropout(p=0.1, inplace=False)\n",
      "    (3): LeakyReLU(negative_slope=0.1)\n",
      "  )\n",
      "  (2): DenseBlock(\n",
      "    (0): Linear(in_features=16, out_features=8, bias=True)\n",
      "    (1): BatchNorm1d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (2): Dropout(p=0.1, inplace=False)\n",
      "    (3): LeakyReLU(negative_slope=0.1)\n",
      "  )\n",
      "  (3): DenseBlock(\n",
      "    (0): Linear(in_features=8, out_features=4, bias=True)\n",
      "    (1): BatchNorm1d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (2): Dropout(p=0.1, inplace=False)\n",
      "    (3): LeakyReLU(negative_slope=0.1)\n",
      "  )\n",
      "  (4): DenseBlock(\n",
      "    (0): Linear(in_features=4, out_features=1, bias=True)\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from torch_tools import FCNet\n",
    "\n",
    "model = FCNet(\n",
    "    in_feats=4,\n",
    "    out_feats=1,\n",
    "    hidden_sizes=(16, 8, 4),\n",
    "    input_bnorm=True,\n",
    "    input_dropout=0.1,\n",
    "    hidden_dropout=0.1,\n",
    "    hidden_bnorm=True,\n",
    ")\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 7: Selecting a loss function\n",
    "\n",
    "The previous loss function we chose was appropriate for classification, but _not_ regression.  \n",
    "Here we'll use the mean-squared-error loss, which is more appropriate for regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn import MSELoss\n",
    "\n",
    "loss_func = MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 8: Selecting an optimiser\n",
    "\n",
    "``Adam`` is regarded as the king of optimisers: let's use it again, but this time specifying the learning rate.\n",
    "\n",
    "[Adam docs](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an optimiser and give it the model's parameters.\n",
    "from torch.optim import Adam\n",
    "\n",
    "optimiser = Adam(model.parameters(), lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 9: Writing basic training and validation loops\n",
    "\n",
    "\n",
    "As before, we will write the training loop together and you can then continue with the validation loop.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Dict\n",
    "from torch.nn import Module\n",
    "from torch import Tensor, no_grad\n",
    "from numpy import mean, sqrt\n",
    "\n",
    "\n",
    "def train_one_epoch(\n",
    "    model: Module,\n",
    "    train_loader: DataLoader,\n",
    "    optimiser: Adam,\n",
    "    loss_func: MSELoss,\n",
    ") -> Dict[str, float]:\n",
    "    \"\"\"Train ``model`` for once epoch.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : Module\n",
    "        The neural network.\n",
    "    train_loader : DataLoader\n",
    "        Training dataloader.\n",
    "    optimiser : Adam\n",
    "        The optimiser.\n",
    "    loss_func : MSELoss\n",
    "        Mean squared error loss function.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    Dict[str, float]\n",
    "        A dictionary of metrics.\n",
    "\n",
    "    Notes\n",
    "    -----\n",
    "    - The ``model.train()`` is very important:\n",
    "        - it turns on the dropout layers.\n",
    "        - it tells the batch norm layers to use the incoming statistics, and\n",
    "          let them contribute to their \"memory\".\n",
    "\n",
    "    \"\"\"\n",
    "    model.train()\n",
    "    metrics: Dict[str, float] = {\"loss\": []}\n",
    "\n",
    "    for batch, targets in train_loader:\n",
    "        optimiser.zero_grad()\n",
    "\n",
    "        preds = model(batch)\n",
    "\n",
    "        loss = loss_func(preds, targets)\n",
    "\n",
    "        loss.backward()\n",
    "\n",
    "        optimiser.step()\n",
    "\n",
    "        metrics[\"loss\"].append(loss.item())\n",
    "\n",
    "    return {key: mean(val) for key, val in metrics.items()}\n",
    "\n",
    "\n",
    "@no_grad()\n",
    "def validate_one_epoch(\n",
    "    model: Module,\n",
    "    valid_loader: DataLoader,\n",
    "    loss_func: MSELoss,\n",
    ") -> Dict[str, float]:\n",
    "    \"\"\"Validate ``model`` for a single epoch.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : Module\n",
    "        The neural network.\n",
    "    valid_loader : DataLoader\n",
    "        Validation dataloader.\n",
    "    loss_func : MSELoss\n",
    "        Mean squared error loss function.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    Dict[str, float]\n",
    "        Metrics of interest.\n",
    "\n",
    "    Notes\n",
    "    -----\n",
    "    - The ``model.eval()`` is also very important:\n",
    "        - It turns off the dropout layers, which are likely to impair the\n",
    "          validation performance and render it unrealistically poor.\n",
    "        - It tells the batchnorm layers to _not_ use the batch's statistics,\n",
    "          and to instead use the stats it has built up from the training set.\n",
    "          The model should not \"remember\" anything from the validation set.\n",
    "    - We also protect this function with ``torch.no_grad()``, because having\n",
    "      gradients enable while validating is a pointless waste of resources—they\n",
    "      are only needed for training.\n",
    "\n",
    "    \"\"\"\n",
    "    model.eval()\n",
    "    metrics: Dict[str, float] = {\"loss\": []}\n",
    "\n",
    "    for batch, targets in valid_loader:\n",
    "        preds = model(batch)\n",
    "\n",
    "        loss = loss_func(preds, targets)\n",
    "\n",
    "        metrics[\"loss\"].append(loss.item())\n",
    "\n",
    "    return {key: mean(val) for key, val in metrics.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 10: Training and extracting metrics\n",
    "\n",
    "- Now we can train our model for a specified number of epochs.\n",
    "  - During each epoch the model \"sees\" each training item once.\n",
    "- Append the training and validation metrics to a list.\n",
    "- Turm them into a ``pandas.DataFrame``\n",
    "  - Note: You can turn a ``List[Dict[str, float]]``, say ``my_list`` into a ``DataFrame`` with ``DataFrame(my_list)``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0-25 time: 4.234758 seconds\n",
      "Epoch 25-50 time: 4.243387 seconds\n",
      "Epoch 50-75 time: 4.252240 seconds\n",
      "Epoch 75-100 time: 4.297547 seconds\n",
      "Epoch 100-125 time: 4.254026 seconds\n",
      "Epoch 125-150 time: 4.280980 seconds\n",
      "Epoch 150-175 time: 4.243602 seconds\n",
      "Epoch 175-200 time: 4.236814 seconds\n",
      "Epoch 200-225 time: 4.238001 seconds\n",
      "Epoch 225-250 time: 4.244598 seconds\n",
      "Epoch 250-275 time: 4.229454 seconds\n",
      "\n",
      "\n",
      "     loss_train  loss_valid\n",
      "0      0.403945    0.258350\n",
      "1      0.334612    0.286643\n",
      "2      0.288723    0.248097\n",
      "3      0.235608    0.204395\n",
      "4      0.210375    0.182948\n",
      "..          ...         ...\n",
      "295    0.011671    0.008663\n",
      "296    0.009973    0.008757\n",
      "297    0.009214    0.008859\n",
      "298    0.010670    0.008706\n",
      "299    0.009724    0.008328\n",
      "\n",
      "[300 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "from time import perf_counter\n",
    "from pandas import DataFrame\n",
    "\n",
    "epochs = 300\n",
    "print_interval = 25\n",
    "\n",
    "train_metrics, valid_metrics = [], []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "\n",
    "    if epoch % print_interval == 0:\n",
    "        if epoch != 0:\n",
    "            print(\n",
    "                f\"Epoch {max(epoch - print_interval, 0)}-{epoch} time: {perf_counter() - tic:.6f} seconds\"\n",
    "            )\n",
    "        tic = perf_counter()\n",
    "\n",
    "    train_metrics.append(train_one_epoch(model, train_loader, optimiser, loss_func))\n",
    "\n",
    "    valid_metrics.append(validate_one_epoch(model, valid_loader, loss_func))\n",
    "\n",
    "print(\"\\n\")\n",
    "\n",
    "train_metrics = DataFrame(train_metrics)\n",
    "valid_metrics = DataFrame(valid_metrics)\n",
    "\n",
    "metrics = train_metrics.join(valid_metrics, lsuffix=\"_train\", rsuffix=\"_valid\")\n",
    "\n",
    "print(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 11: Plotting metrics\n",
    "\n",
    "- Use Matplotlib to plot the training and validation metrics as a function of the number of epochs.\n",
    "- Does this allow us to interpret performance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxUAAAGGCAYAAAANcKzOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKsElEQVR4nO3dd3xUVf7/8fdMKoEUQkgBQhEpIhCQEiNSXCPBRRQrTUEs+5WFXYVdFVCJ6P7Esiq6smJHFwQUATuIwVA0gjQR6U0QSEIITAqkzdzfH5CRIROYZJLMkLyej8c8cM49997Pzd3Jzjv3nnNNhmEYAgAAAIBKMnu6AAAAAAAXN0IFAAAAALcQKgAAAAC4hVABAAAAwC2ECgAAAABuIVQAAAAAcAuhAgAAAIBbCBUAAAAA3OLr6QK8hc1m0+HDhxUcHCyTyeTpcgAAAIAqZRiGcnNz1aRJE5nNVXttgVBxxuHDhxUbG+vpMgAAAIBqdfDgQTVr1qxKt0moOCM4OFjS6R9ySEiIh6sBAAAAqlZOTo5iY2Pt33urEqHijNJbnkJCQggVAAAAqLWq41Z/BmoDAAAAcAuhAgAAAIBbCBUAAAAA3EKoAAAAAOAWQgUAAAAAt3hlqFi5cqUGDRqkJk2ayGQyafHixeftv3DhQl133XVq3LixQkJClJCQoKVLl9ZMsQAAAEAd55VTyubn5ysuLk733HOPbrnllgv2X7lypa677jo988wzCgsL03vvvadBgwZpzZo16tq1aw1UDAAA4H1KSkpUUlLi6TJQDXx9feXr6z1f5U2GYRieLuJ8TCaTFi1apMGDB1dovcsvv1xDhgzRlClTXOqfk5Oj0NBQWSwWnlMBAAAuaidPnlRWVpby8/M9XQqqUf369RUREaGgoCCX+lfn913viTdVyGazKTc3V+Hh4Z4uBQAAoEYVFRXp4MGD8vPzU0xMjAICAqrlYWfwHMMwVFhYqOzsbB08eFCtWrWSv7+/R2uqlaHi3//+t/Ly8nTHHXeU26ewsFCFhYX29zk5OTVRGgAAQLXKzMyUj4+PWrRoIR8fH0+Xg2pSr149BQcHa9++fcrMzFSzZs08Wo9XDtR2x4cffqipU6fqo48+UmRkZLn9pk2bptDQUPsrNja2BqsEAACoeoZh6OTJkwoNDSVQ1AE+Pj4KDQ3VyZMn5ekRDbUqVMybN0/33XefPvroIyUmJp6376RJk2SxWOyvgwcP1lCVAAAA1aO4uFhWq1X16tXzdCmoIfXq1ZPValVxcbFH66g1tz/NnTtX99xzj+bNm6eBAwdesH9AQIACAgJqoDIAAICaYbPZJImrFHVI6bkuPfee4pWhIi8vT7t377a/37dvnzZt2qTw8HA1b95ckyZN0qFDh/TBBx9IOn3L06hRo/TKK68oPj5e6enpkk4nt9DQUI8cAwAAgKcwMLvu8JZz7ZW3P61bt05du3a1P2NiwoQJ6tq1q3162CNHjujAgQP2/m+++aZKSko0duxYxcTE2F8PPvigR+oHAAAA6hKvvFLRr1+/8w42mTVrlsP71NTU6i0IAAAAQLm88koFAAAA4C6TyVShV8uWLau8hpYtW3rNLUrVySuvVAAAAADuGjVqVJm21atXa8+ePYqLi1OXLl0clkVERNRQZbUPoQIAAAC10rm3zEvS3XffrT179mjw4MF68sknq72GlJQUj0/3WhMIFQAAAKgUq83Q2n3ZyswtUGRwoHq2CpePufbf6lMRrVu39nQJNYIxFQAAAKiwJVuO6OrnlmvYWz/qwXmbNOytH3X1c8u1ZMsRT5dWKbNmzZLJZNKTTz6pnTt3aujQoYqKipLZbNbixYslSbt379aTTz6phIQERUdHy9/fX82aNdPIkSO1c+dOp9t1NqZi//79MplM6tevn06dOqWJEyeqRYsWCggI0KWXXqrnnnvO40/IriiuVAAAAKBClmw5ojGzN+jcr73plgKNmb1Br995hQZ0jPFIbe7asWOHevTooUaNGumaa67R8ePH5efnJ0l6++239fzzz6tjx47q0aOHAgICtHXrVv3vf//Tp59+qlWrVqlz584u76uoqEj9+/fX1q1b1a9fP+Xn52vFihWaOHGicnNz9a9//au6DrPKESoAAADqAMMwdKrY6vZ2rDZDyZ/9WiZQSJIhySTpyc+2qtelEW7fClXPz6fGZ06aN2+exo0bp+nTp5d5MvngwYP1f//3f2rVqpVD+3vvvad77rlHDz30kJYvX+7yvtLS0tS3b1/t27dPISEhkk4/r+3KK6/Uyy+/rIkTJ6pBgwbuH1QNIFQAAADUAaeKreowZWm178eQlJ5ToE5PfuP2trY+laQg/5r9utq4cWM999xzZQKFJF155ZVO1xk9erTeeecdpaamymKxKDQ01KV9mc1mvfHGG/ZAIUndu3fX9ddfry+++ELr1q1Tv379KnUcNY1QAQAAAJyRmJiooKCgcpfn5eXp888/16ZNm5SdnW2f2enIkSMyDEN79uzRFVdc4dK+WrRooXbt2pVpb9u2rX2bFwtCBQAAQB1Qz89HW59Kcns7a/dl6+73frpgv1mje6hnq3C39lXPr+zVgurWvHnzcpctX75cQ4cO1dGjR8vtk5ub6/K+mjVr5rQ9ODhYklRYWOjytjyNUAEAAFAHmEymKrmVqHebxooJDVS6pcDpuAqTpOjQQPVu0/iinF42MDDQaXteXp7uuOMOZWdna8qUKRo6dKhatGihevXqyWQyafjw4Zo7d26FZm0ym2vPRKy150gAAABQ7XzMJiUP6iDpdIA4W+n75EEdLspAcT6rVq3SsWPHdOutt2rq1Km67LLLFBQUZB9IvnfvXg9X6FmECgAAAFTIgI4xev3OKxQd6vhX/ejQwIt6OtnzOX78uCTntyzt3r1bGzZsqOmSvAq3PwEAAKDCBnSM0XUdouvME7VLB08vXLhQkydPVuPGjSVJJ06c0L333msfsF1XESoAAABQKT5mkxJaN/J0GTWie/fuuu6667Rs2TK1bdvWPtVramqqIiIidNNNN+nTTz/1bJEexO1PAAAAgAs+/fRTPfbYY2rcuLG+/vprrV+/XkOHDtWPP/6osLAwT5fnUSajIkPUa7GcnByFhobKYrE4PIAEAADgYlFQUKB9+/apVatW5c5ihNqlIue8Or/vcqUCAAAAgFsIFQAAAADcQqgAAAAA4BZCBQAAAAC3ECoAAAAAuIVQAQAAAMAthAoAAAAAbiFUAAAAAHALoQIAAACAWwgVAAAAANxCqAAAAADgFkIFAAAAALcQKgAAAAC4hVABAACAWmn48OEymUx6+umnL9h37dq1MplMioqKUklJSYX2c/fdd8tkMik1NdWhvV+/fjKZTNq/f7/L25o1a5ZMJpOefPLJCtXgaYQKAAAA1Ep33XWXJGnOnDkX7Dt79mxJ0rBhw+Tr61utddVGhAoAAADUSv3791dUVJR27Nihn376qdx+JSUlmj9/vqQ/gkhV+OCDD7Rt2zY1bdq0yrbprQgVAAAAqJV8fHw0bNgwSX9ciXDmm2++UWZmpi677DJ169atyvbfvHlztW/fXn5+flW2TW9FqAAAAEDl2KzSvlXSLwtO/2uzerqiMu68805J0vz582W1Oq+v9PaoO++8UydOnNB//vMfJSUlqUWLFgoICFCjRo00YMAALVu2rEL7Pt+Yiu+//16JiYkKDg5WWFiYkpKStGbNmoodnBchVAAAAKDitn4mTe8ovX+D9Mm9p/+d3vF0uxfp1q2bLrvsMmVkZDgNBfn5+fr0009lMpk0YsQI/fjjj/r73/+unTt3ql27drr55pvVrl07ffPNN0pKStK7777rdk1ffPGF+vXrp5SUFHXo0EHXX3+9Dh48qD59+igtLc3t7XsCoQIAAAAVs/Uz6aORUs5hx/acI6fbvSxYlI6TcHYL1MKFC5Wfn68+ffqoRYsWateundLS0rRv3z598803mjdvnn744QetX79eoaGhGj9+vPLy8ipdS25uru655x6VlJTo3Xff1Zo1azR37lz9+uuvGj9+vN58881Kb9uTCBUAAAB1gWFIRfnuvwpypK8fkWQ428npf5Y8erqfu/synO2j4kaMGCGTyaTFixcrPz/fYVlp0Ci9TapVq1a68sory2yja9euGjt2rHJycvTdd99VupYFCxbo6NGj6tOnj0aPHm1vL536tlmzZpXeticxXxYAAEBdUHxSeqZJDezIOH0F49lY9zc1+bDkX9/tzTRv3lx9+vTRihUrtHjxYo0YMUKSlJGRoZSUFAUGBur222+397darUpJSdEPP/ygI0eOqLCwUJK0a9cuh38rY9WqVZKkoUOHllnm5+en2267TdOnT6/09j2FUAEAAIBa76677tKKFSs0e/Zse6iYO3eurFarbrnlFoWGhkqSfv/9d91www36+eefy91Wbm5upes4fPj0LWMtWrRwurxly5aV3rYnESoAAADqAr+g03/5d9dvP0hzbrtwvxELpBZXubcvvyD31j/LbbfdpnHjxunbb79VZmamIiMj7bc+nf1sivvuu08///yzbr31Vj3yyCNq166dgoODZTab9eabb+r//u//ZFTRbVm1iVeOqVi5cqUGDRqkJk2a2O9/u5DU1FRdccUVCggI0KWXXqpZs2ZVe50AAAAXDZPp9K1E7r5a/0kKaSLJVN6OpJCmp/u5uy9TefuouNDQUN14440qKSnR3LlztX37dq1fv14REREaMGCApNMzQS1btkxRUVGaP3++evbsqdDQUJnNp78y79271+06YmJiJEm//fab0+XltXs7rwwV+fn5iouL04wZM1zqv2/fPg0cOFDXXHONNm3apIceekj33Xefli5dWs2VAgAA1DFmH2nAc2fenPul/8z7Ac+e7udlSgdjz5kzx/5siiFDhtgfTmexWGSz2RQTEyMfH8f6i4uLtWjRIrdr6N27tyTpo48+KrOspKREn3zyidv78ASvDBXXX3+9/vWvf+nmm292qf/MmTPVqlUrvfjii7rssss0btw43XbbbXr55ZeruVIAAIA6qMON0h0fSCExju0hTU63d7jRM3VdwIABAxQREaGffvpJM2fOlOR461NkZKRCQ0O1ZcsWff/99/Z2q9WqRx99VDt37nS7httvv12NGjVSamqq3n//fXu7YRhKTk7WgQMH3N6HJ3hlqKiotLQ0JSYmOrQlJSWd9+EhhYWFysnJcXgBAADARR1ulB7aIo36Qrr1ndP/PvSL1wYK6fTsSqWzLmVlZalNmzaKj4+3L/f19dUjjzyikpIS9e3bV/3799fQoUN16aWXaubMmRo7dqzbNQQHB+udd96Rj4+P7r77bl155ZUaPny4OnbsqBdeeEH333+/2/vwhFoRKtLT0xUVFeXQFhUVpZycHJ06dcrpOtOmTVNoaKj9FRtbBdOeAQAA1CVmH6lVb6nTbaf/9cJbns519pWJ0tuhzjZ58mS9//776ty5s77//nt9++23iouL048//qju3btXSQ033XSTvvvuO11zzTXasmWLvvzyS8XExGjFihW66io3B7d7iMnw8uHrJpNJixYt0uDBg8vt07ZtW40ePVqTJk2yt3311VcaOHCgTp48qXr16pVZp7Cw0D7nsCTl5OQoNjZWFotFISEhVXoMAAAANaGgoED79u1Tq1atFBgY6OlyUAMqcs5zcnIUGhpaLd93a8WUstHR0crIyHBoy8jIUEhIiNNAIUkBAQEKCAioifIAAACAWq1W3P6UkJCglJQUh7Zly5YpISHBQxUBAAAAdYdXhoq8vDxt2rRJmzZtknR6ythNmzbZR8NPmjRJI0eOtPd/4IEHtHfvXj3yyCPavn27/vvf/+qjjz7S+PHjPVE+AAAAUKd4ZahYt26dunbtqq5du0qSJkyYoK5du2rKlCmSpCNHjjhMt9WqVSt9+eWXWrZsmeLi4vTiiy/q7bffVlJSkkfqBwAAAOoSrxxT0a9fv/M+/tzZ07L79eunjRs3VmNVAAAAAJzxyisVAAAAAC4ehAoAAAAAbiFUAAAA1DJe/hgyVCFvOdeECgAAgFrCbD791c5qtXq4EtSU0nNdeu49hVABAABQS/j5+cnHx0enTp3ydCmoIadOnZKPj4/8/Pw8WgehAgAAoJYwmUwKCgqSxWLhakUdYLVaZbFYFBQUJJPJ5NFavHJKWQAAAFROZGSk9u/fr99++03h4eEKCAjw+BdOVC3DMFRYWKjs7GzZbDZFRkZ6uiRCBQAAQG3i7++vZs2aKSsrS0eOHPF0OahG9evXV3R0tPz9/T1dCqECAACgtgkKClLz5s1VUlKikpIST5eDauDr6ytfX+/5Ku89lQAAAKBKedsXT9ReDNQGAAAA4BZCBQAAAAC3ECoAAAAAuIVQAQAAAMAthAoAAAAAbiFUAAAAAHALoQIAAACAWwgVAAAAANxCqAAAAADgFkIFAAAAALcQKgAAAAC4hVABAAAAwC2ECgAAAABuIVQAAAAAcAuhAgAAAIBbCBUAAAAA3EKoAAAAAOAWQgUAAAAAtxAqAAAAALiFUAEAAADALYQKAAAAAG4hVAAAAABwC6ECAAAAgFsIFQAAAADcQqgAAAAA4JYqCRUlJSXKyspSSUlJVWwOAAAAwEXEtzIr/fTTT/ryyy+1YsUKbdq0STk5OfZlISEh6tKli/r27auBAweqR48eVVYsAAAAAO9jMgzDcKWj1WrV22+/rZkzZ2rz5s06e7Xg4GCFhITIYrEoLy/vj42bTIqLi9MDDzyge+65R76+lcowNSInJ0ehoaGyWCwKCQnxdDkAAABAlarO77su3f60YMECXXbZZRozZoz27t2rO++8U++8845++eUXFRcXy2Kx6ODBg8rJyVFRUZE2b96st99+W8OHD9fu3bv1wAMPqEOHDvrkk0+qtHgAAAAAnufSlQqz2ay4uDhNnDhRN910kwIDA13eQUFBgRYtWqTnnntOv/zyi6xWq1sFVxeuVAAAAKA28/iVikWLFmnjxo0aMmRIhQKFJAUGBmrYsGHatGmTFi5c6PJ6M2bMUMuWLRUYGKj4+HitXbv2vP2nT5+udu3aqV69eoqNjdX48eNVUFBQoVoBAAAAVJxLoeKmm26qkp25up358+drwoQJSk5O1oYNGxQXF6ekpCRlZmY67f/hhx9q4sSJSk5O1rZt2/TOO+9o/vz5mjx5cpXUDQAAAKB8Xvmcipdeekn333+/Ro8erQ4dOmjmzJkKCgrSu+++67T/Dz/8oF69emn48OFq2bKl+vfvr2HDhl3w6gYAAAAA97kUKlauXKmdO3dWeOPffvutXn311QqtU1RUpPXr1ysxMdHeZjablZiYqLS0NKfrXHXVVVq/fr09ROzdu1dfffWV/vznP5e7n8LCQuXk5Di8AAAAAFScS6GiX79+eu6555wuCw8P19/+9jeny+bMmaPx48dXqKCsrCxZrVZFRUU5tEdFRSk9Pd3pOsOHD9dTTz2lq6++Wn5+fmrdurX69et33tufpk2bptDQUPsrNja2QnUCAAAAOM3l25/KmyTqxIkTys/Pr7KCKiM1NVXPPPOM/vvf/2rDhg1auHChvvzySz399NPlrjNp0iRZLBb76+DBgzVYMQAAAFB7eN3T6CIiIuTj46OMjAyH9oyMDEVHRztd54knntBdd92l++67T5LUqVMn5efn6y9/+Ysee+wxmc1ls1NAQIACAgKq/gAAAACAOsbrBmr7+/urW7duSklJsbfZbDalpKQoISHB6TonT54sExx8fHwklX+FBQAAAEDV8LorFZI0YcIEjRo1St27d1fPnj01ffp05efna/To0ZKkkSNHqmnTppo2bZokadCgQXrppZfUtWtXxcfHa/fu3XriiSc0aNAge7gAAAAAUD28MlQMGTJER48e1ZQpU5Senq4uXbpoyZIl9sHbBw4ccLgy8fjjj8tkMunxxx/XoUOH1LhxYw0aNEj/7//9P08dAgAAAFBneGWokKRx48Zp3LhxTpelpqY6vPf19VVycrKSk5NroDIAAAAAZ3M5VKxevVr33HNPhZatXr268pUBAAAAuCiYDBdGMjubPcnlHZhMslqtlV6/puTk5Cg0NFQWi0UhISGeLgcAAACoUtX5fdelKxXvvfdele4UAAAAQO3hUqgYNWpUddcBAAAA4CLldc+pAAAAAHBxqbLZnw4ePKilS5cqKytLTZs21fXXX6+IiIiq2jwAAAAAL+VSqPj555/1xhtvKD4+3umtUNOnT9ejjz6qkpISe1tQUJDeeustDR06tOqqBQAAAOB1XLr9acGCBXrjjTfUqFGjMstSUlI0YcIEFRcXq2nTprrlllvUuXNn5efna9SoUdq2bVuVFw0AAADAe7gUKlatWqXg4GANGDCgzLJnnnlGktSrVy/t2LFDH3/8sTZu3Kjk5GQVFxfrtddeq9qKAQAAAHgVl0LF/v371aNHD/n6Ot4tlZOTo5UrV8pkMmnatGmqV6+efdljjz2m6OhorVixomorBgAAAOBVXAoVmZmZioqKKtP+008/yWq1qmHDhurVq5fDMl9fX3Xp0kW//fZb1VQKAAAAwCu5FCpMJpOOHTtWpn3dunWSpO7du8tkMpVZHh4e7jB4GwAAAEDt41KoaNWqldatW6dTp045tC9dulQmk6nMVYpS5V3hAAAAAFB7uBQqBgwYoGPHjmncuHEqLCyUJM2fP1+pqamSpMGDB5dZxzAMbdiwQbGxsVVWLAAAAADv41Ko+Oc//6mGDRtq1qxZCgkJUUREhIYPHy5JuuGGG9SpU6cy6yxbtkzZ2dnq3bt31VYMAAAAwKu4FCqio6OVkpKiyy+/XMXFxcrOzpZhGEpKStKsWbOcrvPiiy9KkpKSkqqsWAAAAADex2QYhlGRFXbv3q2jR4+qWbNm5721afPmzTIMQx07dpSPj4/bhVa3nJwchYaGymKxKCQkxNPlAAAAAFWqOr/vVjhU1FaECgAAANRm1fl916XbnwAAAACgPL4X7iJ98MEHbu1k5MiRbq0PAAAAwHu5dPuT2Wx2+nC7CzEMQyaTSVartVLF1SRufwIAAEBtVp3fd126UlHqiiuuUPfu3au0AAAAAAAXN5dCxWWXXaZt27Zpw4YNys/P14gRIzRixAi1bNmymssDAAAA4O1cGqj966+/av369XrwwQdlsVj0xBNPqHXr1urdu7feeOMNHT9+vLrrBAAAAOClKjylrM1m07fffqs5c+Zo8eLFys3Nlb+/v66//nrdeeedGjRokPz9/aur3mrDmAoAAADUZl77nIqCggItWrRIs2fP1rJly2S1WhUSEqK77rpLr776alXWWe0IFQAAAKjNvDZUnO3IkSO69957tWTJEkVERCgzM7MqNltjCBUAAACozbxm9idn1qxZo9mzZ+ujjz7S0aNHZTKZlJCQUBW1AQAAALgIVCpU7Nq1S3PmzNGcOXO0d+9eGYahzp0765///KeGDx+upk2bVnWdAAAAALyUy6Hi6NGjmjt3rubMmaN169bJMAzFxsbq4Ycf1p133qmOHTtWZ50AAAAAvJRLoeL6669XSkqKrFarQkNDde+992rEiBHq27dvddcHAAAAwMu5NFDbbDbLZDKpa9euGjhwoAICAiq0k8mTJ1e6wJrCQG0AAADUZh6f/cls/uMZeSaTyf7fZ696dvvZy00mk6xWq7t1VjtCBQAAAGozj8/+lJycXKU7BQAAAFB7VNlzKi52XKkAAABAbVad33fNF+4CAAAAAOUjVAAAAABwi0uh4tFHH1V2drZbO8rKytIjjzzi1jYAAAAAeB+XQsX06dPVsmVLTZgwQRs3bqzQDtavX68HH3xQrVq10quvvlqpIgEAAAB4L5dmf/r11181adIkTZ8+Xa+88oratGmjfv36qUePHmrXrp0aNmyo4OBg5ebmKjs7Wzt27NBPP/2k1NRU7d69W4Zh6Pbbb9e0adOq+3gAAAAA1LAKzf60YcMGvf7665o/f77y8vKcPpuilGEYatCggYYNG6a//vWviouLq1BhM2bM0AsvvKD09HTFxcXpP//5j3r27Flu/xMnTuixxx7TwoULlZ2drRYtWmj69On685//7NL+mP0JAAAAtZnHH353rvz8fK1YsUKrVq3S5s2blZmZKYvFotDQUEVGRiouLk69e/dWnz59VL9+/QoXNX/+fI0cOVIzZ85UfHy8pk+fro8//lg7duxQZGRkmf5FRUXq1auXIiMjNXnyZDVt2lS//fabwsLCXA4zhAoAAADUZl4XKqpbfHy8evTooddee02SZLPZFBsbq7/97W+aOHFimf4zZ87UCy+8oO3bt8vPz69S+yRUAAAAoDarU8+pKCoq0vr165WYmGhvM5vNSkxMVFpamtN1PvvsMyUkJGjs2LGKiopSx44d9cwzz8hqtdZU2QAAAECd5dJA7ZqUlZUlq9WqqKgoh/aoqCht377d6Tp79+7V8uXLNWLECH311VfavXu3/vrXv6q4uFjJyclO1yksLFRhYaH9fU5OTtUdBAAAAFCHeN2Visqw2WyKjIzUm2++qW7dumnIkCF67LHHNHPmzHLXmTZtmkJDQ+2v2NjYGqwYAAAAqD28LlRERETIx8dHGRkZDu0ZGRmKjo52uk5MTIzatm0rHx8fe9tll12m9PR0FRUVOV1n0qRJslgs9tfBgwer7iAAAACAOsTrQoW/v7+6deumlJQUe5vNZlNKSooSEhKcrtOrVy/t3r1bNpvN3rZz507FxMTI39/f6ToBAQEKCQlxeAEAAACoOK8LFZI0YcIEvfXWW3r//fe1bds2jRkzRvn5+Ro9erQkaeTIkZo0aZK9/5gxY5Sdna0HH3xQO3fu1JdffqlnnnlGY8eO9dQhAAAAAHWG1w3UlqQhQ4bo6NGjmjJlitLT09WlSxctWbLEPnj7wIEDMpv/yEOxsbFaunSpxo8fr86dO6tp06Z68MEH9eijj3rqEAAAAIA6wyufU+EJPKcCAAAAtZnXPafi5MmTOnDggPLz8x3ajx8/rokTJ+qGG27QX//6V+3Zs6dKigQAAADgvSp1+9PTTz+t559/XmvXrlW3bt0knX7uw5VXXqndu3er9OLHggUL9PPPPysmJqbqKgYAAADgVSp1pWL58uVq3bq1PVBI0uzZs7Vr1y5dc801Wrp0qf7+978rKytLL7/8cpUVCwAAAMD7VCpUHDhwQG3atHFo++yzz2QymfTee+/puuuu0/Tp09W2bVt9/fXXVVIoAAAAAO9UqVBx/PhxhYWF2d8bhqHVq1erc+fODk+mjouL46FyAAAAQC1XqVARHR2tffv22d+vX79ex48fV9++fR36mUwm96oDAAAA4PUqFSq6dOmitWvXavHixcrNzdXTTz8tk8mkG264waHfrl271KRJkyopFAAAAIB3qlSoeOSRRyRJt956q8LCwvT5558rLi5Of/rTn+x9MjIy9PPPPzsM5gYAAABQ+1QqVFx11VVatGiRrr76arVv31533nmnPvvsM4enXM+dO1fBwcEaMGBAlRULAAAAwPvwRO0zeKI2AAAAajOve6I2AAAAAJSqVKjIyMjQypUrlZGR4dC+Z88eDR06VB07dtSf//xnpaWlVUmRAAAAALxXpULFs88+q2uuuUYWi8XelpOTo6uvvloff/yxtm7dqiVLligxMVG7du2qsmIBAAAAeJ9KhYrU1FR16NBBbdu2tbfNmjVLGRkZGjZsmHbs2KGXXnpJp06d0osvvlhlxQIAAADwPpUKFYcOHdIll1zi0Pbll1/K19dX06dPV5s2bfTQQw8pLi5OK1asqJJCAQAAAHinSoWK3NxcBQUF2d9brValpaWpW7duioiIsLe3b99ev//+u/tVAgAAAPBalQoVTZo00fbt2+3vV69erby8PPXr18+hX0lJifz9/d0qEAAAAIB3q1SoSEhI0ObNmzV9+nT98ssvevzxx2UymTRo0CCHftu2bVPTpk2rpFAAAAAA3qlSoWLSpEkKCAjQP/7xD3Xp0kXff/+9+vXrp6uuusreZ//+/dq6davi4+OrrFgAAAAA3se3MitdfvnlWr16tV555RVlZWWpW7duevjhhx36LF26VHFxcRo8eHBV1AkAAADAS5kMwzA8XYQ3qM7HlgMAAACeVp3fdyt1+xMAAAAAlKrU7U+lMjIy9O6772rVqlU6dOiQJKlp06bq06ePRo8eraioqCopEgAAAID3qvTtT5988onuuece5eXl6dxNmEwmBQcH65133tGtt95aJYVWN25/AgAAQG3mdbc/rVu3TsOGDVN+fr5uvvlmLVq0SBs3btSmTZu0ePFi3XLLLcrLy9Pw4cO1bt26Ki0YAAAAgHep1JWKW2+9VYsXL9aCBQt08803O+2zaNEi3Xrrrbrlllu0YMECtwutblypAAAAQG1Wnd93KxUqoqKi1LZtW61ateq8/Xr37q2dO3cqIyOj0gXWFEIFAAAAajOvu/3JYrGoefPmF+zXvHlzWSyWyuwCAAAAwEWiUqEiOjpaGzduvGC/TZs2KTo6ujK7AAAAAHCRqFSoSEpK0o4dOzR58mRZrdYyyw3D0OOPP67t27drwIABbhcJAAAAwHtVakzF77//rq5duyo7O1vNmzfXHXfcoZYtW0qSfvvtN3388cfav3+/GjVqpA0bNqhZs2ZVXXeVY0wFAAAAajOvG6gtSb/88otGjBihLVu2nN6QySRJ9mdWdOrUSXPmzFHHjh2rqNTqRagAAABAbVad33cr/UTtTp06afPmzUpNTdWqVat0+PBhSVKTJk3Uu3dv9evXr6pqBAAAAODFKn2lwhXvvvuufv/9d02ZMqW6dlFluFIBAACA2szrppR11VtvvaWpU6dW5y4AAAAAeFi1hgoAAAAAtR+hAgAAAIBbCBUAAAAA3EKoAAAAAOAWQsU51u7NltVWbRNiAQAAALUOoeIc97z/k65+brmWbDni6VIAAACAi4JLocLHx6dSr7Vr17pV3IwZM9SyZUsFBgYqPj7e5e3NmzdPJpNJgwcPrtR+0y0FGjN7A8ECAAAAcIFLocIwjEq/Kmv+/PmaMGGCkpOTtWHDBsXFxSkpKUmZmZnnXW///v365z//qd69e1d636VVT/18K7dCAQAAABfgUqiw2WyVflmt1koV9tJLL+n+++/X6NGj1aFDB82cOVNBQUF69913y13HarVqxIgRmjp1qi655JJK7beUIemIpUBr92W7tR0AAACgtvPKMRVFRUVav369EhMT7W1ms1mJiYlKS0srd72nnnpKkZGRuvfeey+4j8LCQuXk5Di8nMnMLaj4AQAAAAB1iFeGiqysLFmtVkVFRTm0R0VFKT093ek6q1ev1jvvvKO33nrLpX1MmzZNoaGh9ldsbKzTfpHBgRUrHgAAAKhjvDJUVFRubq7uuusuvfXWW4qIiHBpnUmTJslisdhfBw8edFhukhQTGqiercKroWIAAACg9vD1dAHOREREyMfHRxkZGQ7tGRkZio6OLtN/z5492r9/vwYNGmRvs9lskiRfX1/t2LFDrVu3dlgnICBAAQEBTvdvOvNv8qAO8jGbnPYBAAAAcJpXXqnw9/dXt27dlJKSYm+z2WxKSUlRQkJCmf7t27fXL7/8ok2bNtlfN954o6655hpt2rSp3FubyhMdGqjX77xCAzrGuH0sAAAAQG3nlVcqJGnChAkaNWqUunfvrp49e2r69OnKz8/X6NGjJUkjR45U06ZNNW3aNAUGBqpjx44O64eFhUlSmXZXfDO+j4ID/dw+BgAAAKAu8NpQMWTIEB09elRTpkxRenq6unTpoiVLltgHbx84cEBmc/VcaDliKSBUAAAAAC4yGe48oa4WycnJOT0L1EMf6d37+yixQ9SFVwIAAAAuEqXfdy0Wi0JCQqp02145psLTDh4/6ekSAAAAgIsGocKJA9mECgAAAMBVhAonDmaf8nQJAAAAwEWDUOHEQa5UAAAAAC4jVDhx8PhJMX4dAAAAcA2hwomTRVbNWXNAaXuOyWojXAAAAADn47XPqfAUs+n0v48v3iJJigkNVPKgDjxdGwAAACgHVyrOce6FiXRLgcbM3qAlW454piAAAADAyxEqLqA0Y0z9fCu3QgEAAABOECpcYEg6YinQ2n3Zni4FAAAA8DqEigrIzC3wdAkAAACA1yFUVEBkcKCnSwAAAAC8DrM/ucAkKTo0UD1bhXu6FAAAAMDrcKXiHKZy3icP6iAf87lLAQAAABAqzhEZEuDwPjo0UK/feQXPqQAAAADKQag4xzfj++rvf7pUktS6cX2tfvRPBAoAAADgPAgV5/Axm3TLFc0kSQePn5LN4NkUAAAAwPkQKs5ls6p5znrd7v+jrrBt0TsrdyltzzEefAcAAACUw2QY/CleknJychQaGirL/2ujkKIMe/thI1xTi0dqc3AfJQ/qwK1QAAAAuCjZv+9aLAoJCanSbXOl4hxGbrrD+2hl63W/6YrLXakxszdoyZYjHqoMAAAA8E6EinOcO2ls6SyyU/z+J7Nsmvr5Vm6FAgAAAM5CqHCB2SQ1MR1TD/N2HbEUaO2+bE+XBAAAAHgNQkUFROqEJCkzt8CzhQAAAABehFBRAZkKkyRFBgd6thAAAADAi/h6uoCLgc2Q0tVIP9naKyY0UD1bhXu6JAAAAMBrcKXiAkrHZD9VfJdsMit5UAf5mM8dzg0AAADUXVypuIB0NdLU4ru0ObiPXuc5FQAAAEAZhIrzMCS9UHy7ctRAy8f3Vr1Af0+XBAAAAHgdbn86D5Okl/1naq7/v+T7n87S1s88XRIAAADgdQgVLvLNT5c+GkmwAAAAAM5BqHCRSYYMScaSiZLN6ulyAAAAAK9BqKgAkwyZcg5pbernni4FAAAA8BqEikqY/e1PWrLliKfLAAAAALwCoaISMhWmqZ9vlbX0IRYAAABAHUaoOFe9RjLk/OF2NkM6bDTSWlt7HbEUaO2+7BouDgAAAPA+hIpzDZgm6Y8naZcqfT/1zJO1JSkzt6AmKwMAAAC8EqHiXO3/rJ19Zyhd4Q7N6QrXmOKHtNTW094WGRxY09UBAAAAXocnajtxad/h6pPWWK3yNug9v+fkZ7JpeNFj2m/E2PuE1/dTtxYNPVglAAAA4B24UuGEj9mkJ27spO9tnbTVaCFJuttnqa40b5VZNklSdn6x+r7wHbNAAQAAoM4jVJRjQMcYLbwmS23Mp0PD3b7faJ7/v7Q64O9KMq+VJKVbCjRm9gaCBQAAAOo0rw4VM2bMUMuWLRUYGKj4+HitXbu23L5vvfWWevfurYYNG6phw4ZKTEw8b/8L2vqZuqY9qHpyHIwdrWy97jddSea1Kh3LzfSyAAAAqMu8NlTMnz9fEyZMUHJysjZs2KC4uDglJSUpMzPTaf/U1FQNGzZM3333ndLS0hQbG6v+/fvr0KFDFd+5zSoteVRyMrms+UxDst//ZJZNhsT0sgAAAKjTTIZheOWf2OPj49WjRw+99tprkiSbzabY2Fj97W9/08SJEy+4vtVqVcOGDfXaa69p5MiRF+yfk5Oj0NBQWSwWhRz7WXr/hguuM7Tocf1o6yBJemVoF93UpekF1wEAAAA8weH7bkhIlW7bK69UFBUVaf369UpMTLS3mc1mJSYmKi0tzaVtnDx5UsXFxQoPD3e6vLCwUDk5OQ4vu7wMl/YRqRN//DfTywIAAKCO8spQkZWVJavVqqioKIf2qKgopaenu7SNRx99VE2aNHEIJmebNm2aQkND7a/Y2Ng/FjaIcrrOuY4qRCZJMaGB6tnKeXgBAAAAajuvDBXuevbZZzVv3jwtWrRIgYHOryBMmjRJFovF/jp48OAfC1tcJYU0kcqMqHD0kt9MJZnXKnlQB/mYz98XAAAAqK28MlRERETIx8dHGRmOtyFlZGQoOjr6vOv++9//1rPPPqtvvvlGnTt3LrdfQECAQkJCHF52Zh9pwHNn3pQfFqJM2Xrdf7quM7kxyxQAAABwkfPKUOHv769u3bopJSXF3maz2ZSSkqKEhIRy13v++ef19NNPa8mSJerevbt7RXS4UbrjAym4/BBjlmQY0tGPx2vJL7+7tz8AAADgIuWVoUKSJkyYoLfeekvvv/++tm3bpjFjxig/P1+jR4+WJI0cOVKTJk2y93/uuef0xBNP6N1331XLli2Vnp6u9PR05eXlVb6IDjdKN79x3i5mkxStY5o1dy4PwQMAAECd5OvpAsozZMgQHT16VFOmTFF6erq6dOmiJUuW2AdvHzhwQGbzH5no9ddfV1FRkW677TaH7SQnJ+vJJ5+sfCH5R13qFqkTmvr5Vl3XIZrxFQAAAKhTvDZUSNK4ceM0btw4p8tSU1Md3u/fv796inBxJqgWpnR9duYheAmtG1VPLQAAAIAX8trbn7zGmZmgyj5b+w+GIY33XaAk81pl5hbUYHEAAACA5xEqLsQ+E5Sh8p49bjqTN5L9/qffjubWWGkAAACANyBUuKLDjbL1nWQPD86YTVIT0zH98N3nDNgGAABAnUKocJFPxKUu9SsdsG21lXNZAwAAAKhlCBWucnHAdqbCdOTMgG0AAACgLiBUuMqFAdtWw6yGOj2mggHbAAAAqCsIFa6yD9iWyruzySybZvi9oiTzWkXUD6jB4gAAAADPIVRURIcbZbvtPRkm5z+2s2eBevijDQzYBgAAQJ1AqKggn/oR8pGt3OWls0C1yN+oMbMJFgAAAKj9CBUVlZfhUrf/+r2qJPNaZoICAABArUeoqCgXZ4EKVZ7+6zddnXNXMhMUAAAAajVCRUWdmQVK55kFSjp9G5R0enzFt78eqv66AAAAAA8hVFTUWbNAXeimptLxFb/+uFRfbT5c/bUBAAAAHkCoqIwON0p3fCDVC3Ope6JpvcbN3aivNjNoGwAAALUPoaKyOtwo0+0fuNT1Xt+vdZ1prf76IbNBAQAAoPYhVLij5dVnnrJ9foZOj60wy8ZsUAAAAKh1CBXuOGt8xXm7nRlb0dO8XUcsBcwGBQAAgFqFUOGuDjfKiB/jUtdInZAkpVtOVWNBAAAAQM0iVFQBc/uBLvVrYUqXJD395TbGVgAAAKDWIFRUhTPPrjDO8+wKw5DG+y7Q33w+0Yn8Aj0wewPTzAIAAKBWIFRUhTNjK0wyyh20bTKdHlvxD79P9H3A35VkXss0swAAAKgVCBVVpcONUr/JF3jO9mlRytbrftOZZhYAAAC1AqGiKjVq7VI385nkwTSzAAAAqA0IFVWpQZTLXc+dZvblZTuVtucY4QIAAAAXHV9PF1CrnBmwrZwj0gUfiXda6TSzr323W699t1th9fw0uldLjftTG/mYXbmZCgAAAPAsrlRUJRcfhne20mlmS504VayXv92lbv9axlgLAAAAXBQIFVWtw43SHR9IwTEX7Fo6zWySeW2ZZSdOFmvMbAZxAwAAwPsRKqpDhxul8b9K/SZLKv9GKJN9wPYHMstWZrkh6cnPfmWcBQAAALwaoaK6mH2kfo9ecJrZ0wO2szXWZ7HT5ek5hXpt+e5qKREAAACoCgzUrm4uTjM7wXeBdhrNtNTWs8yyl7/dKcnQmH6Xav1vx5WZW6DI4ED1bBVuH8xttRlauy/b6TIAAACgOhEqqpuL08wakv6f37tKKbxCJU5Oy8vf7tL0b3c53EoVExqo5EEdJElTP9+qI5YC+7Lw+n66uUtTJXaIJmAAAACgWpkMw+CGfUk5OTkKDQ2VxWJRSEhI1W3YZpWmd5RyDrvU/ZgRrMnF9zq9YlFZoYG+uq5DlHq1aazokNNXMSRp7b5spVtOKTu/SOENAhTZIEAySVl5hRe82sGVEQAAgItLtX3fFaHCrjp/yNr6mfTRXS51NYzTVy3+Wvx3LbFdWbV1nBEW5Cfp9AxT5xMdEqBhPZurZUR9RdT/I3DszzqpuWsPKD3njysjpVdNBnSMqZLAQWgBAACoWoSKGlCtoUKSUp+TUp9xubvNMOmVkpu11mivxspRpsK01tZeNi8fW5/QKlxb03NlOfVHYDn7VqxuLRqWGRciySFAHM8v0tNfOt7OdXZoKeUseJy7LcIIAADAaYSKGlDtoaKCt0E5c9yor3dLBmiG9WavDxflMclxit0GAT4yDCm/yOrS+vf2aqk/tY/ST/uzNeuH/TpxVngJ8veR2WRSXmGJve3sW79Kb+/KzCmw3/JVejsYwQMAANR2hIoaUO2hQjpzG9RIlf/kCtfkGQF6s2SgfjLaK1In1MiUo2yjgcJNeTpmhChTYZJkX3bMCFGGwt260mGWTT3N2xWpE8pUmNbZ2qq7eaf9/cVwFaU8YfX8NOqqFurZqpHTwCGVHX9S3rKwIH+dOHnhMSqlV1mcbbOyAefcKzfOrgpVZ3jiljUAALwboaIG1EiokE4Hiy8ekk4eq759lOO4UV/vlSS5HEZKlzXVUQ32/UGNTLn2bdkMk8ymP/6nk2UE61PrVTpkNHZpm66Gn9IwE6Vse5/zbbN0fUkOIchZ6Dk3KDnrc/bVj3NryfUJU5apkVYUtnEpUJVeNQmu56dPNx1Wdn5RmT7OAo6zoHLust9PnCqzzXOvCoXX99NNcU3UrGHQecPP+W5RcxaeokOc37IWVs9Po3u11Lg/tbGvf77Acb7b2c4Xvs5e7+yxP5UJVpUNRucLic6O/dw2V+t0JTi68rOuCu6G2Kq8fdHdn8v5zrs3hmVvrAlV43y/zzjPqAqEihpQY6FCkkqKpJcuk05mVe9+LlKl4UeSRvt+o4amvAqtn2MEyixDDUyFZbZZGqh6mbfoOp/1amjKt/c5XzDqYdpebi1VEai8adlxo4EamvLsfXJ9wnTM1FCniq2V2rafj1nRphNqYD1hX3bKr6GuuLy92keHqCD7kA4fPqg1GVJgscW+Xj0/H0UYxx3WK10WEuCj/s1N8i04VmY9Z8dy7npN/E/KXD9ctvxs+QY3VmDDJtqRnqP1v25XveLj5dZZkpfl0nrnOwZnP09X6jxcXF9LfzOUW1hS7nrO9ufKMUiq0DJXaqnoz9rZz6XAL1TxUVKTps3dquV8PxeTVO55L29Zt8sv081dm8qUn6HfDhxQli1IJXmu/zwLTxxWI+UqtmlTHTx0SMcUrIAw186Ds8/K+X7WFT23Hl12/Igy1VAFMd0VeGSdIk0nFBgW7X11VtMyZ+f27N9nrnwePH0M3rwsKLyp2kcHV/hzG9QoVu3jT38n2b5mqU4e+12ZaqiiZlcqMjjo9B/6LPnyP7RGEcYx2fKy3Pu8hzdTTmQPZeYVn/4jXj0fBR5eo0jTCQU1jDl9DCePaltukHYGXCb/wy58Vs5sM+tkieqrWNd1bUWoqE41GiqkCs0IBQBAXXHulXDA03KNQJkl1Tf9cTXe3T+Ank/pHyuDdarMH0DPVpHPSpYRrMXWXvr6VEctfGkKoaI61XiokKQti6VPRkuGrWb2BwAAgDorp9BQ6LO51fJ916tH1s6YMUMtW7ZUYGCg4uPjtXbt2vP2//jjj9W+fXsFBgaqU6dO+uqrr2qo0krqOFi67T1PVwEAAAC4xWtDxfz58zVhwgQlJydrw4YNiouLU1JSkjIzM532/+GHHzRs2DDde++92rhxowYPHqzBgwdry5YtNVx5BV0+WLrjf1K9hp6uBAAAAKgUr739KT4+Xj169NBrr70mSbLZbIqNjdXf/vY3TZw4sUz/IUOGKD8/X1988YW97corr1SXLl00c+bMC+7PI7c/nc1mlVb+W1rzunTqeM3vHwAAALVanbv9qaioSOvXr1diYqK9zWw2KzExUWlpaU7XSUtLc+gvSUlJSeX29zpmH6nfo9LDe6RRX0i3viPd9anUd6LkX9/T1QEAAADl8vV0Ac5kZWXJarUqKirKoT0qKkrbt293uk56errT/unp6U77FxYWqrDwjylHLRaLpNNXLDyuUZzU6Mx/N75C6vqA9P2r0rp3pIITnqwMAAAAF6mcwtM3KFXHjUpeGSpqwrRp0zR16tQy7bGxsR6oBgAAAKgZx44dU2hoaJVu0ytDRUREhHx8fJSRkeHQnpGRoejoaKfrREdHV6j/pEmTNGHCBPv7EydOqEWLFjpw4ECV/5DhGTk5OYqNjdXBgwc9M04GVYrzWftwTmsfzmntwvmsfSwWi5o3b67w8PAq37ZXhgp/f39169ZNKSkpGjx4sKTTA7VTUlI0btw4p+skJCQoJSVFDz30kL1t2bJlSkhIcNo/ICBAAQEBZdpDQ0P54NQyISEhnNNahPNZ+3BOax/Oae3C+ax9zOaqH1btlaFCkiZMmKBRo0ape/fu6tmzp6ZPn678/HyNHj1akjRy5Eg1bdpU06ZNkyQ9+OCD6tu3r1588UUNHDhQ8+bN07p16/Tmm2968jAAAACAWs9rQ8WQIUN09OhRTZkyRenp6erSpYuWLFliH4x94MABh5R11VVX6cMPP9Tjjz+uyZMnq02bNlq8eLE6duzoqUMAAAAA6gSvDRWSNG7cuHJvd0pNTS3Tdvvtt+v222+v1L4CAgKUnJzs9JYoXJw4p7UL57P24ZzWPpzT2oXzWftU5zn12offAQAAALg4eOXD7wAAAABcPAgVAAAAANxCqAAAAADgFkLFGTNmzFDLli0VGBio+Ph4rV271tMlwQVPPvmkTCaTw6t9+/b25QUFBRo7dqwaNWqkBg0a6NZbby3zkER41sqVKzVo0CA1adJEJpNJixcvdlhuGIamTJmimJgY1atXT4mJidq1a5dDn+zsbI0YMUIhISEKCwvTvffeq7y8vBo8CpS60Pm8++67y3xmBwwY4NCH8+ldpk2bph49eig4OFiRkZEaPHiwduzY4dDHld+1Bw4c0MCBAxUUFKTIyEg9/PDDKikpqclDgVw7n/369SvzOX3ggQcc+nA+vcfrr7+uzp07258nkpCQoK+//tq+vKY+n4QKSfPnz9eECROUnJysDRs2KC4uTklJScrMzPR0aXDB5ZdfriNHjthfq1evti8bP368Pv/8c3388cdasWKFDh8+rFtuucWD1eJc+fn5iouL04wZM5wuf/755/Xqq69q5syZWrNmjerXr6+kpCQVFBTY+4wYMUK//vqrli1bpi+++EIrV67UX/7yl5o6BJzlQudTkgYMGODwmZ07d67Dcs6nd1mxYoXGjh2rH3/8UcuWLVNxcbH69++v/Px8e58L/a61Wq0aOHCgioqK9MMPP+j999/XrFmzNGXKFE8cUp3myvmUpPvvv9/hc/r888/bl3E+vUuzZs307LPPav369Vq3bp3+9Kc/6aabbtKvv/4qqQY/nwaMnj17GmPHjrW/t1qtRpMmTYxp06Z5sCq4Ijk52YiLi3O67MSJE4afn5/x8ccf29u2bdtmSDLS0tJqqEJUhCRj0aJF9vc2m82Ijo42XnjhBXvbiRMnjICAAGPu3LmGYRjG1q1bDUnGTz/9ZO/z9ddfGyaTyTh06FCN1Y6yzj2fhmEYo0aNMm666aZy1+F8er/MzExDkrFixQrDMFz7XfvVV18ZZrPZSE9Pt/d5/fXXjZCQEKOwsLBmDwAOzj2fhmEYffv2NR588MFy1+F8er+GDRsab7/9do1+Puv8lYqioiKtX79eiYmJ9jaz2azExESlpaV5sDK4ateuXWrSpIkuueQSjRgxQgcOHJAkrV+/XsXFxQ7ntn379mrevDnn9iKxb98+paenO5zD0NBQxcfH289hWlqawsLC1L17d3ufxMREmc1mrVmzpsZrxoWlpqYqMjJS7dq105gxY3Ts2DH7Ms6n97NYLJKk8PBwSa79rk1LS1OnTp3sD7CVpKSkJOXk5Nj/mgrPOPd8lpozZ44iIiLUsWNHTZo0SSdPnrQv43x6L6vVqnnz5ik/P18JCQk1+vn06off1YSsrCxZrVaHH6QkRUVFafv27R6qCq6Kj4/XrFmz1K5dOx05ckRTp05V7969tWXLFqWnp8vf319hYWEO60RFRSk9Pd0zBaNCSs+Ts89n6bL09HRFRkY6LPf19VV4eDjn2QsNGDBAt9xyi1q1aqU9e/Zo8uTJuv7665WWliYfHx/Op5ez2Wx66KGH1KtXL3Xs2FGSXPpdm56e7vRzXLoMnuHsfErS8OHD1aJFCzVp0kSbN2/Wo48+qh07dmjhwoWSOJ/e6JdfflFCQoIKCgrUoEEDLVq0SB06dNCmTZtq7PNZ50MFLm7XX3+9/b87d+6s+Ph4tWjRQh999JHq1avnwcoAODN06FD7f3fq1EmdO3dW69atlZqaqmuvvdaDlcEVY8eO1ZYtWxzGruHiVd75PHsMU6dOnRQTE6Nrr71We/bsUevWrWu6TLigXbt22rRpkywWixYsWKBRo0ZpxYoVNVpDnb/9KSIiQj4+PmVGwWdkZCg6OtpDVaGywsLC1LZtW+3evVvR0dEqKirSiRMnHPpwbi8epefpfJ/P6OjoMpMqlJSUKDs7m/N8EbjkkksUERGh3bt3S+J8erNx48bpiy++0HfffadmzZrZ2135XRsdHe30c1y6DDWvvPPpTHx8vCQ5fE45n97F399fl156qbp166Zp06YpLi5Or7zySo1+Put8qPD391e3bt2UkpJib7PZbEpJSVFCQoIHK0Nl5OXlac+ePYqJiVG3bt3k5+fncG537NihAwcOcG4vEq1atVJ0dLTDOczJydGaNWvs5zAhIUEnTpzQ+vXr7X2WL18um81m/z9CeK/ff/9dx44dU0xMjCTOpzcyDEPjxo3TokWLtHz5crVq1cphuSu/axMSEvTLL784BMZly5YpJCREHTp0qJkDgaQLn09nNm3aJEkOn1POp3ez2WwqLCys2c9nVY0yv5jNmzfPCAgIMGbNmmVs3brV+Mtf/mKEhYU5jIKHd/rHP/5hpKamGvv27TO+//57IzEx0YiIiDAyMzMNwzCMBx54wGjevLmxfPlyY926dUZCQoKRkJDg4apxttzcXGPjxo3Gxo0bDUnGSy+9ZGzcuNH47bffDMMwjGeffdYICwszPv30U2Pz5s3GTTfdZLRq1co4deqUfRsDBgwwunbtaqxZs8ZYvXq10aZNG2PYsGGeOqQ67XznMzc31/jnP/9ppKWlGfv27TO+/fZb44orrjDatGljFBQU2LfB+fQuY8aMMUJDQ43U1FTjyJEj9tfJkyftfS70u7akpMTo2LGj0b9/f2PTpk3GkiVLjMaNGxuTJk3yxCHVaRc6n7t37zaeeuopY926dca+ffuMTz/91LjkkkuMPn362LfB+fQuEydONFasWGHs27fP2Lx5szFx4kTDZDIZ33zzjWEYNff5JFSc8Z///Mdo3ry54e/vb/Ts2dP48ccfPV0SXDBkyBAjJibG8Pf3N5o2bWoMGTLE2L17t335qVOnjL/+9a9Gw4YNjaCgIOPmm282jhw54sGKca7vvvvOkFTmNWrUKMMwTk8r+8QTTxhRUVFGQECAce211xo7duxw2MaxY8eMYcOGGQ0aNDBCQkKM0aNHG7m5uR44GpzvfJ48edLo37+/0bhxY8PPz89o0aKFcf/995f5Aw7n07s4O5+SjPfee8/ex5Xftfv37zeuv/56o169ekZERITxj3/8wyguLq7ho8GFzueBAweMPn36GOHh4UZAQIBx6aWXGg8//LBhsVgctsP59B733HOP0aJFC8Pf399o3Lixce2119oDhWHU3OfTZBiGUeFrKgAAAABwRp0fUwEAAADAPYQKAAAAAG4hVAAAAABwC6ECAAAAgFsIFQAAAADcQqgAAAAA4BZCBQAAAAC3ECoAAAAAuIVQAQCQyWS64Ovuu+/2dJkX9OSTT8pkMmnWrFmeLgUA6hRfTxcAAPAeo0aNKnfZ1VdfXYOVAAAuJoQKAIAdf+EHAFQGtz8BAAAAcAuhAgBQKSaTSS1btlRRUZGSk5PVunVrBQYG6pJLLtGUKVNUUFDgdL1jx47p4YcfVps2bRQYGKjw8HANGDBA33zzTbn7OnbsmB577DF16tRJ9evXV0hIiDp16qRHHnlER44ccbrOL7/8ohtvvFENGzZU/fr11bdvX/3www9VcuwAAEeECgBApRmGoVtvvVUvvPCCOnTooIEDByo7O1tPP/20brjhBlmtVof+hw4dUs+ePfXvf/9bRUVFGjx4sLp27apvv/1WSUlJevnll8vsY9u2berSpYueeeYZZWVlKSkpSYmJiTIMQy+88ILWrFlTZp1169bpyiuv1P79+5WUlKQ2bdpo5cqVuvbaa7Vly5Zq+3kAQJ1lAADqPElGRf8voXSdZs2aGXv27LG3Z2ZmGh07djQkGS+//LLDOjfccIMhyRg+fLhRWFhob1+1apURFBRk+Pj4GBs3brS3FxcXG+3atTMkGQ899JDDOoZhGFu2bDF2795tf5+cnGyv65VXXnHo+9BDDxmSjLvuuqtCxwkAuDCuVAAA7M43pezixYudrjNlyhRdcskl9veNGzfWCy+8IEl67bXX7O179+7VF198oQYNGug///mP/P397cuuvvpqPfDAA7JarZoxY4a9feHChdqxY4cuv/xy/fvf/3ZYR5Iuv/xytW7dukxNvXr10t///neHtscff1yStHLlShd/GgAAVzH7EwDA7nxTyjZv3txp+9ChQ8u0DRgwQA0bNtSePXt05MgRxcTEaPXq1fZl4eHhZda566679NJLL2nVqlX2tm+//VaSdN9998nHx8fl4+jfv3+ZtkaNGik8PLzcMRgAgMojVAAA7Co6pWzDhg0VHBzsdFmLFi10/PhxHT58WDExMTp8+LAkqWXLlk77l7YfOnTI3nbw4EFJcno14nyaNWvmtD04OFjZ2dkV2hYA4MK4/QkA4BVMJlOVbcts5v/eAKAm8VsXAFBpx48fV25urtNlBw4ckCQ1adLE4d/ffvvNaf/9+/dLkpo2bWpvi42NlSTt2bOnSuoFAFQPQgUAwC0fffRRmbZvvvlG2dnZuuSSSxQTEyPp9GBsSVqyZIlOnDhRZp3Zs2dLknr37m1vS0xMlCS98847stlsVV06AKCKECoAAG6ZOnWq/SqDJGVlZenhhx+WJI0dO9befskll2jgwIHKzc3Vgw8+qOLiYvuytLQ0vf766/Lx8XFY55ZbblHbtm21ZcsWPfLIIw7rSNKvv/6qvXv3VtORAQBcxUBtAIDd3XffXe6y5s2b66mnnirT1rlzZ11++eW69tpr5efnp+XLl+vEiRO65pprykzr+sYbb6h379764IMPtGLFCiUkJOjo0aNKTU2V1WrViy++qC5dutj7+/r66pNPPtF1112nF198UR9++KESEhJkGIZ27dqlLVu2aNGiRQ5T2gIAah6hAgBg9/7775e7LC4urkyoMJlMWrBggZ566il9+OGH9pmexo4dq8cee0y+vo7/N9O0aVP99NNPmjZtmhYvXqyFCxcqKChI1157rf7xj384nQq2Y8eO+vnnn/XCCy/os88+01dffaWAgAA1b95cjz76qK688sqqOXgAQKWZDMMwPF0EAODiYzKZ1KJFC4dbnwAAdRNjKgAAAAC4hVABAAAAwC2ECgAAAABuYaA2AKBSGJIHACjFlQoAAAAAbiFUAAAAAHALoQIAAACAWwgVAAAAANxCqAAAAADgFkIFAAAAALcQKgAAAAC4hVABAAAAwC2ECgAAAABu+f9PbrkOUgZkrAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from numpy import linspace\n",
    "\n",
    "\n",
    "quantities = [\"loss\", \"ratio\"]\n",
    "fig, axis = plt.subplots(1, 1, figsize=(8, 4))\n",
    "\n",
    "\n",
    "axis.plot(metrics.loss_train, \"-o\", label=\"Train\")\n",
    "axis.plot(metrics.loss_valid, \"-o\", label=\"Valid\")\n",
    "\n",
    "# for axis in axes.ravel():\n",
    "axis.legend(fontsize=15)\n",
    "axis.set_ylim(bottom=0.0, top=1.25)\n",
    "axis.set_xlim(left=0, right=epochs)\n",
    "axis.set_xlabel(\"Epoch\", fontsize=15)\n",
    "axis.set_ylabel(\"Loss (MSE)\", fontsize=15)\n",
    "\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this losses have decreased nicely, MSE isn't the most easily interpretable way to evaluate the model's performance. We could instead look at the ratio of the predictions to targets—and in particular, the distribution of this quantity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAAHECAYAAAADCgpRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABILUlEQVR4nO3deVyU5f7/8fewowguoIjimuaCS5GauYCm4lrayaXS1DI19zxW6k9ByqNl5qHUTFu0TM3lqHXKLPO4ZGblmmsH19xRj4KiosD9+6Ob+ToBIs7AMPh6Ph7zOM19X/d9f+5r7jjz7r7uayyGYRgCAAAAAMjN2QUAAAAAQEFBQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADB5OLuA/Jaenq5Tp06pWLFislgszi4HAAAAQD4wDEOXL19WSEiI3Nyyv090zwWkU6dOKTQ01NllAAAAAHCC48ePq3z58tmuv+cCUrFixST92TH+/v5OrgYAAABAfkhKSlJoaKg1D2TnngtIGcPq/P39CUgAAADAPSanx2yYpAEAAAAATAQkAAAAADARkAAAAADAREACAAAAANM9N0kDAADIW4ZhKC0tTampqc4uBcA9wMPDQ+7u7g77jVMCEgAAcAjDMHTp0iWdO3dOaWlpzi4HwD3E3d1dpUuXVkBAgN1BiYAEAAAc4syZM7p06ZL1pzQ8PDwc9l90ASArhmEoNTVVSUlJOn36tK5du6ayZcvatU8CEgAAsFtaWpoSExMVFBSkwMBAZ5cD4B5TrFgxeXt76/z58ypdurTc3d3vel9M0gAAAOx28+ZNGYahokWLOrsUAPeookWLyjAM3bx50679EJAAAIDDMKQOgLM46u8PAQkAAAAATAUqIG3cuFGdOnVSSEiILBaLVq5cmW3bgQMHymKxKC4uLt/qAwAAAFC4FaiAlJycrHr16mnmzJm3bbdixQpt2bJFISEh+VQZAAAA8KcJEybYNZwrMjJSkZGRjisIDlWgAlK7du00ceJEdenSJds2J0+e1NChQ7VgwQJ5enrmY3UAAOBeNW/ePFksFlksFm3atCnTesMwFBoaKovFoo4dOzqhwjv33Xff6fnnn1dYWJjc3d1VqVKlXO/jyy+/1IMPPigfHx9VqFBBMTExmX4YeO3atXruuedUvXp1FSlSRFWqVFG/fv10+vTp2+770qVLKl26tCwWi5YtW5br2m61e/duPfnkk6pYsaJ8fHxUrlw5tW7dWtOnT7dpN2nSpNuOXCrM9u/fr7Zt28rPz08lS5ZUr169dO7cuTvadvHixerZs6eqVasmi8Vyx6HvH//4hywWi8LCwmyWX716VTNnzlSbNm1UtmxZFStWTA888IBmzZqVr7+t5lLTfKenp6tXr156+eWXVbt27TvaJiUlRSkpKdb3SUlJeVUeAADIRqXRXzu7BB19o4Pd+/Dx8dHChQvVtGlTm+UbNmzQiRMn5O3tbfcx8trChQu1ePFiPfjgg3c1Guebb75R586dFRkZqenTp2v37t2aOHGiEhISNGvWLGu7V199Vf/73//UtWtXVatWTYcPH9aMGTP01VdfaefOnQoODs5y/9HR0bp69epdn1+GzZs3q0WLFqpQoYJeeOEFBQcH6/jx49qyZYveeecdDR061Np20qRJevLJJ9W5c2e7j+tKTpw4oebNmysgIECTJk3SlStXNHXqVO3evVu//PKLvLy8brv9rFmztG3bNjVo0EAXLly442NOmjQpyxkvDx8+rKFDh+rRRx/VyJEj5e/vr2+//VaDBg3Sli1b9Mknn9zVeeaWSwWkN998Ux4eHho2bNgdbzN58mTFxsbmYVUAnK0gfPEqiBzxZRCArfbt22vp0qV699135eHxf1+jFi5cqPDwcJ0/f96J1d2ZSZMm6YMPPpCnp6c6duyoPXv25Gr7UaNGqW7duvruu++sfeDv769JkyZp+PDhqlGjhiRp2rRpatq0qdzc/m/AUtu2bRUREaEZM2Zo4sSJmfa9Z88ezZo1S9HR0YqOjrbjLP+8SxEQEKBff/1VxYsXt1mXkJBg174Li0mTJik5OVnbtm1ThQoVJEkNGzZU69atNW/ePPXv3/+228+fP1/lypWTm5tbprtB2Rk1apQefvhhpaWlZfr3JTg4WLt377a5ETJgwAA999xzmjt3rsaPH6/77rsvl2eZewVqiN3tbNu2Te+88471FvedGjNmjBITE62v48eP52GVAACgMHvqqad04cIFrVmzxrrsxo0bWrZsmZ5++ukst0lPT1dcXJxq164tHx8flSlTRgMGDNDFixdt2n3xxRfq0KGDQkJC5O3trapVq+r111/PNLQoMjJSYWFh2rdvn1q0aKEiRYqoXLlymjJlyh2dQ0hIyF0/prBv3z7t27dP/fv3twmIgwYNkmEYNkPimjdvbhOOMpaVLFlS+/fvz3L/w4cPV5cuXdSsWbNsazhw4ID++OOPHGs9dOiQateunSkcSVLp0qWt/2yxWJScnKxPPvnEOoyyT58+1vWbNm1SgwYN5OPjo6pVq2r27Nk5HvtWc+bMUdWqVeXr66uGDRvqhx9+yLJdSkqKYmJidN9998nb21uhoaF65ZVXbEZChYWFqUWLFpm2TU9PV7ly5fTkk0/mqrZ//etf6tixozUcSVKrVq1UvXp1LVmyJMftQ0NDM33Gt7Nx40YtW7Ys20nWAgMDsxwllvH4TXbXjaO5TED64YcflJCQoAoVKsjDw0MeHh46duyY/v73v9927Ky3t7f8/f1tXgAAAHejUqVKaty4sRYtWmRd9s033ygxMVE9evTIcpsBAwbo5ZdfVpMmTfTOO++ob9++WrBggaKiomx+0HLevHny8/PTyJEj9c477yg8PFzR0dEaPXp0pn1evHhRbdu2Vb169fT222+rRo0aevXVV/XNN984/qRvsWPHDknSQw89ZLM8JCRE5cuXt67PzpUrV3TlyhUFBgZmWrd06VJt3rw5x6BXs2ZNPfvssznWWrFiRW3bti3HO2Tz58+Xt7e3mjVrpvnz52v+/PkaMGCApD+fYWrTpo0SEhI0YcIE9e3bVzExMVqxYkWOx5ekjz76SAMGDFBwcLCmTJmiJk2a6LHHHsv0H+zT09P12GOPaerUqerUqZOmT5+uzp0765///Ke6d+9ubde9e3dt3LhRZ86csdl+06ZNOnXqVLbXYFZOnjyphISETJ+l9OddpJw+y9xKS0vT0KFD1a9fP9WpUydX22acb1bXTV5wmSF2vXr1UqtWrWyWRUVFqVevXurbt6+TqgIAAPeap59+WmPGjNG1a9fk6+urBQsWKCIiIsvneTZt2qQPP/xQCxYssLnD1KJFC7Vt21ZLly61Ll+4cKF8fX2tbQYOHKiBAwfqvffe08SJE22ebzp16pQ+/fRT9erVS5L0/PPPq2LFivroo4/Url27vDp16wQLZcuWzbSubNmyOnXq1G23j4uL040bN2y+9EvStWvXNGrUKL300kuqVKmSjh49aneto0aNUrt27VS/fn01bNhQzZo106OPPqoWLVrY3EHr2bOnBg4cqCpVqqhnz542+4iOjpZhGPrhhx+sd1n+9re/3dEX/Js3b2rs2LGqX7++1q1bZ32ep1atWurfv79CQ0OtbRcuXKjvv/9eGzZssHm+LSwsTAMHDtTmzZv1yCOPqHv37oqOjtayZcs0ZMgQa7vFixfLz89PHTrc+dDqnD7L//3vf0pJSXHYc3Xvv/++jh07pu+//z5X2924cUNxcXGqXLmyGjRo4JBaclKg7iBduXJFO3fu1M6dOyVJR44c0c6dO/XHH3+oVKlSCgsLs3l5enoqODhY999/v3MLBwAA94xu3brp2rVr+uqrr3T58mV99dVX2Q6vW7p0qQICAtS6dWudP3/e+goPD5efn5/WrVtnbXtrOLp8+bLOnz+vZs2a6erVqzpw4IDNfv38/Gy+zHt5ealhw4Y6fPiwg8/W1rVr1yQpyy/NPj4+1vVZ2bhxo2JjY9WtWze1bNnSZt0bb7xhDRQ5MQxD69evz7Fd69at9dNPP+mxxx7Trl27NGXKFEVFRalcuXL68ssvc9w+LS1N3377rTp37mwzBK1mzZqKiorKcfutW7cqISFBAwcOtJnsoE+fPgoICLBpu3TpUtWsWVM1atSwuU4y+injOqlevbrq16+vxYsX29S5bNkyderUyeYayklOn+Wtbex14cIFRUdHa/z48QoKCsrVtkOGDNG+ffs0Y8YMm2GdealA3UHaunWrzbjKkSNHSpJ69+6tefPmOakqAACA/xMUFKRWrVpp4cKFunr1qtLS0rJ99iM+Pl6JiYk2z7zc6tbJAvbu3atx48bpP//5T6ZZdxMTE23ely9fPtMz2SVKlNBvv/12N6d0xzK+gN/6XEyG69evZ/sF/cCBA+rSpYvCwsL04Ycf2qw7evSo3nrrLc2cOVN+fn4OrbdBgwZavny5bty4oV27dmnFihX65z//qSeffFI7d+5UrVq1st323LlzunbtmqpVq5Zp3f33369Vq1bd9tjHjh2TpEzbe3p6qkqVKjbL4uPjtX///mzDw63XSffu3TV27FidPHlS5cqV0/r165WQkJDprlxOcvosb21jr3HjxqlkyZI2MwfeibfeeksffPCBXn/9dbVv394htdyJAhWQIiMjZRjGHbd3xO1XAACA3Hr66af1wgsv6MyZM2rXrl2WEwFIfz5bUrp0aS1YsCDL9RlfiC9duqSIiAj5+/vrtddeU9WqVeXj46Pt27fr1VdfVXp6us127u7uWe4vN9+j7kbGcKzTp0/bDBHLWNawYcNM2xw/flxt2rRRQECAVq1apWLFitmsj46OVrly5RQZGWn9bpfxzMm5c+d09OhRVahQIVeTAfyVl5eXGjRooAYNGqh69erq27evli5dqpiYmLvepyOlp6erTp06mjZtWpbrb+3r7t27a8yYMVq6dKlGjBihJUuWKCAgQG3bts3VMW/9LP/q9OnTKlmypEOG18XHx2vOnDmKi4uzGYJ5/fp13bx5U0ePHpW/v79Klixps928efP06quvauDAgRo3bpzddeRGgQpIAAAArqBLly4aMGCAtmzZYjPc6a+qVq2q77//Xk2aNLntf41fv369Lly4oOXLl6t58+bW5UeOHHFo3faqX7++pD9H/dwahk6dOqUTJ05kmhb6woULatOmjVJSUrR27dosn3f5448/dPDgwUx3VaQ/Z8eT/pyUIrsQmlsZkxLcGgyymiE5KChIvr6+io+Pz7Tu999/z/E4FStWlPRnQLh1SOHNmzd15MgR1atXz7qsatWq2rVrlx599NEcZ2uuXLmyGjZsqMWLF2vIkCFavny5OnfunOswU65cOQUFBWnr1q2Z1v3yyy/Wz9peJ0+eVHp6uoYNG5blT/VUrlxZw4cPt5nZ7osvvlC/fv30xBNPaObMmQ6pIzcK1DNIAAAArsDPz0+zZs3ShAkT1KlTp2zbdevWTWlpaXr99dczrUtNTdWlS5ck/d8doVvvAN24cUPvvfeeYwvPhZs3b+rAgQM2QaJ27dqqUaOG5syZYzP9+KxZs2SxWGyGGiYnJ6t9+/Y6efKkVq1aleVQNUmaOHGiVqxYYfPK6K9XXnlFK1asyPJHRXOybt26LO+oZQyNu/UZ9qJFi1o/iwzu7u6KiorSypUrbaYV379/v7799tscj//QQw8pKChI77//vm7cuGFdPm/evEzH6tatm06ePKkPPvgg036uXbum5ORkm2Xdu3fXli1b9PHHH+v8+fM2w+uy+tyy87e//U1fffWVzax6a9eu1X//+1917dr1rvb5V2FhYZk+3xUrVqh27dqqUKGCVqxYoeeff97afuPGjerRo4eaN2+uBQsW2HXn8G5xBwkAAOAu9O7dO8c2ERERGjBggCZPnqydO3eqTZs28vT0VHx8vJYuXap33nlHTz75pB555BGVKFFCvXv31rBhw2SxWDR//vw8GTL322+/WScpOHjwoBITE60/2lqvXj1r4Dt58qRq1qyZ6Vnwt956S4899pjatGmjHj16aM+ePZoxY4b69eunmjVrWts988wz+uWXX/Tcc89p//79Nr9h4+fnp86dO0uSzaxtGTLuFjVo0MDaLoPFYlFERESOEzUMHTpUV69eVZcuXVSjRg3duHFDmzdv1uLFi1WpUiWbWZDDw8P1/fffa9q0aQoJCVHlypXVqFEjxcbGavXq1WrWrJkGDRqk1NRUTZ8+XbVr187xeS9PT09NnDhRAwYMUMuWLdW9e3cdOXJEc+fOzXS3rFevXlqyZIkGDhyodevWqUmTJkpLS9OBAwe0ZMkSffvttzbTcXfr1k2jRo3SqFGjVLJkSZuZnrP73LIyduxYLV26VC1atNDw4cN15coVvfXWW6pTp45N/2S3z40bN2rjxo2S/hwOmZycbL2WmjdvrubNmyswMDDTZyjJesfo1nXHjh3TY489Zg3bS5cutdmmbt26qlu37m3PyREISAAAAHno/fffV3h4uGbPnq2xY8fKw8NDlSpVUs+ePdWkSRNJUqlSpfTVV1/p73//u8aNG6cSJUqoZ8+eevTRR+9oxrTc2L59u8aPH2+zLON97969b3tHTJI6duyo5cuXKzY2VkOHDlVQUJDGjh2r6Ohom3YZsxJ//PHH+vjjj23WVaxYMcsvzTm5cuWKpKynpv6rqVOnaunSpVq1apXmzJmjGzduqEKFCho0aJDGjRtnM2Rv2rRp6t+/v8aNG6dr166pd+/eatSokerWratvv/1WI0eOVHR0tMqXL6/Y2FidPn36jibE6N+/v9LS0vTWW2/p5ZdfVp06dfTll19m6n83NzetXLlS//znP/Xpp59qxYoVKlKkiKpUqaLhw4erevXqNu3Lly+vRx55RD/++KP69et31z/8Gxoaqg0bNmjkyJEaPXq0vLy81KFDB7399tt3NGTvP//5j2JjY22WZZxbTEyMzXDRO3HkyBHrhCSDBw/OtD4mJiZfApLFyOun+QqYpKQkBQQEKDExkR+NBQqJSqO/dnYJBdLRN+789zAAe12/fl1HjhxR5cqVrVMEA462atUqdezYUbt27cr1j42i8Mvp79Cd5gCeQQIAAIBLWLdunXr06EE4Qp5iiB0AAABcwltvveXsEnAP4A4SAAAAAJgISAAAAABgIiABAAAAgImABAAAAAAmAhIAAHCYe+zXQwAUII76+0NAAgAAdvP09JTFYlFycrKzSwFwj0pOTpbFYrnrH87NwDTfAADAbu7u7goICNC5c+eUkpIif39/eXh4yGKxOLs0AIWYYRhKTU1VUlKSkpKSVLx4cbm7u9u1TwISAABwiODgYPn6+iohIUFJSUnOLgfAPcTd3V1ly5ZVQECA3fsiIAEAAIewWCwqXry4AgIClJaWptTUVGeXBOAe4OHhIXd3d4fdsSYgAQAAh7JYLPLw8JCHB18zALgeJmkAAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAVKAC0saNG9WpUyeFhITIYrFo5cqV1nU3b97Uq6++qjp16qho0aIKCQnRs88+q1OnTjmvYAAAAACFSoEKSMnJyapXr55mzpyZad3Vq1e1fft2jR8/Xtu3b9fy5cv1+++/67HHHnNCpQAAAAAKIw9nF3Crdu3aqV27dlmuCwgI0Jo1a2yWzZgxQw0bNtQff/yhChUq5EeJAAAAAAqxAhWQcisxMVEWi0XFixfPtk1KSopSUlKs75OSkvKhMgAAAACuqEANscuN69ev69VXX9VTTz0lf3//bNtNnjxZAQEB1ldoaGg+VgkAAADAlbhkQLp586a6desmwzA0a9as27YdM2aMEhMTra/jx4/nU5UAAAAAXI3LDbHLCEfHjh3Tf/7zn9vePZIkb29veXt751N1AAAAAFyZSwWkjHAUHx+vdevWqVSpUs4uCQAAAEAhUqAC0pUrV3Tw4EHr+yNHjmjnzp0qWbKkypYtqyeffFLbt2/XV199pbS0NJ05c0aSVLJkSXl5eTmrbAAAAACFRIEKSFu3blWLFi2s70eOHClJ6t27tyZMmKAvv/xSklS/fn2b7datW6fIyMj8KhMAAABAIVWgAlJkZKQMw8h2/e3WAQAAAIC9XHIWOwAAAADICwQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBWogLRx40Z16tRJISEhslgsWrlypc16wzAUHR2tsmXLytfXV61atVJ8fLxzigUAAABQ6BSogJScnKx69epp5syZWa6fMmWK3n33Xb3//vv6+eefVbRoUUVFRen69ev5XCkAAACAwsjD2QXcql27dmrXrl2W6wzDUFxcnMaNG6fHH39ckvTpp5+qTJkyWrlypXr06JGfpQIAAAAohArUHaTbOXLkiM6cOaNWrVpZlwUEBKhRo0b66aefst0uJSVFSUlJNi8AAAAAyIrLBKQzZ85IksqUKWOzvEyZMtZ1WZk8ebICAgKsr9DQ0DytEwAAAIDrcpmAdLfGjBmjxMRE6+v48ePOLgkAAABAAeUyASk4OFiSdPbsWZvlZ8+eta7Lire3t/z9/W1eAAAAAJAVlwlIlStXVnBwsNauXWtdlpSUpJ9//lmNGzd2YmUAAAAACosCNYvdlStXdPDgQev7I0eOaOfOnSpZsqQqVKigESNGaOLEiapWrZoqV66s8ePHKyQkRJ07d3Ze0QAAAAAKjQIVkLZu3aoWLVpY348cOVKS1Lt3b82bN0+vvPKKkpOT1b9/f126dElNmzbV6tWr5ePj46ySAQAAABQiFsMwDGcXkZ+SkpIUEBCgxMREnkcCColKo792dgkF0tE3Oji7BAAACow7zQEu8wwSAAAAAOQ1AhIAAAAAmAhIAAAAAGAiIAEAAACAiYAEAAAAACYCEgAAAACYCEgAAAAAYCIgAQAAAICJgAQAAAAAJgISAAAAAJgISAAAAABgIiABAAAAgImABAAAAAAmAhIAAAAAmAhIAAAAAGAiIAEAAACAiYAEAAAAACYCEgAAAACYCEgAAAAAYCIgAQAAAICJgAQAAAAAJgISAAAAAJjsCkinT592VB0AAAAA4HR2BaTQ0FC1adNG8+fPV3JysqNqAgAAAACnsCsgvfbaazp16pR69+6tMmXKqGfPnlq9erXS09MdVR8AAAAA5Bu7AtLYsWO1Z88ebdu2TQMHDtT69evVvn17hYSE6KWXXtLWrVsdVScAAAAA5DmHTNLwwAMPaOrUqTp+/LjWrFmjDh06aO7cuWrUqJFq1aqlSZMm6Y8//nDEoQAAAAAgzzh0FjuLxaJmzZqpffv2evjhh2UYhuLj4zVhwgRVqVJFXbt2ZWIHAAAAAAWWwwLSunXr1K9fP5UpU0bdunXTmTNnNHXqVJ04cUKnT5/WG2+8obVr16pXr16OOiQAAAAAOJSHPRvv2rVLCxYs0KJFi3Tq1CkFBwerX79+evbZZ1WnTh2btqNGjZKPj49GjRplV8EAAAAAkFfsCkgPPPCAfH191blzZz377LNq3bq13NyyvylVu3ZtNW7c2J5DAgAAAECesSsgffzxx3ryySfl5+d3R+1btGihFi1a2HNIAAAAAMgzdgWkPn36OKgMAAAAAHA+uyZpePfddxUVFZXt+nbt2mnWrFn2HAIAAAAA8o1dAemjjz5SrVq1sl1fq1YtzZkzx55DAAAAAEC+sSsgHTp0SDVr1sx2fY0aNXTo0CF7DgEAAAAA+caugOTl5aUzZ85ku/706dO3ndUOAAAAAAoSu9LLww8/rHnz5uny5cuZ1iUmJmru3Ll6+OGH7TkEAAAAAOQbu2axi4mJUUREhOrXr68RI0aodu3akqQ9e/YoLi5Op0+f1sKFCx1SKAAAAADkNbsCUqNGjfTvf/9bAwYM0PDhw2WxWCRJhmGocuXK+vLLL/lhWAAAAAAuw66AJEmtW7fWwYMHtWPHDuuEDFWrVtWDDz5oDUwAAAAA4ArsDkiS5ObmpvDwcIWHhztid9lKS0vThAkT9Nlnn+nMmTMKCQlRnz59NG7cOMIYAAAAALs5JCDt27dPhw8f1sWLF2UYRqb1zz77rCMOozfffFOzZs3SJ598otq1a2vr1q3q27evAgICNGzYMIccAwAAAMC9y66AdOjQIfXs2VO//PJLlsFIkiwWi8MC0ubNm/X444+rQ4cOkqRKlSpp0aJF+uWXXxyyfwAAAAD3NrsC0oABA7R7927FxcWpWbNmKlGihKPqytIjjzyiOXPm6L///a+qV6+uXbt2adOmTZo2bVq226SkpCglJcX6PikpKU9rBAAAAOC67ApIP/74o8aOHauhQ4c6qp7bGj16tJKSklSjRg25u7srLS1N//jHP/TMM89ku83kyZMVGxubL/UBea3S6K+dXQIAAEChZtcPxQYGBiogIMBRteRoyZIlWrBggRYuXKjt27frk08+0dSpU/XJJ59ku82YMWOUmJhofR0/fjzf6gUAAADgWuy6gzRw4EB99tlnGjx4sNzd3R1VU7ZefvlljR49Wj169JAk1alTR8eOHdPkyZPVu3fvLLfx9vaWt7d3ntcGAAAAwPXZFZCqV6+utLQ01atXT88995xCQ0OzDEpPPPGEPYexunr1qtzcbG96ubu7Kz093SH7BwAAAHBvsysgde/e3frPo0aNyrKNxWJRWlqaPYex6tSpk/7xj3+oQoUKql27tnbs2KFp06bpueeec8j+AQAAANzb7ApI69atc1Qdd2T69OkaP368Bg0apISEBIWEhGjAgAGKjo7O1zoAAAAAFE4WI7sfMCqkkpKSFBAQoMTERPn7+zu7HCBXmMUOuXH0jQ7OLgEAgALjTnOAXXeQMqSkpGj79u1KSEhQkyZNFBgY6IjdAgAAAEC+smuab0l69913VbZsWTVt2lRPPPGEfvvtN0nS+fPnFRgYqI8//tjuIgEAAAAgP9gVkObOnasRI0aobdu2+uijj3TraL3AwEC1bNlSn3/+ud1FAgAAAEB+sCsgvf3223r88ce1cOFCderUKdP68PBw7d27155DAAAAAEC+sSsgHTx4UO3atct2fcmSJXXhwgV7DgEAAAAA+caugFS8eHGdP38+2/X79u1TcHCwPYcAAAAAgHxjV0Bq37695syZo0uXLmVat3fvXn3wwQd67LHH7DkEAAAAAOQbuwLSxIkTlZaWprCwMI0bN04Wi0WffPKJevbsqYceekilS5fmR1wBAAAAuAy7AlJISIi2bdumtm3bavHixTIMQ/Pnz9e///1vPfXUU9qyZQu/iQQAAADAZdj9Q7GlS5fWhx9+qA8//FDnzp1Tenq6goKC5OZm908sAQAAAEC+sjsg3SooKMiRuwMAAACAfGVXQHrttddybGOxWDR+/Hh7DgMAAAAA+cKugDRhwoRs11ksFhmGQUACAAAA4DLselAoPT090ys1NVWHDh3SSy+9pIceekgJCQmOqhUAAAAA8pTDZ1Jwc3NT5cqVNXXqVFWrVk1Dhw519CEAAAAAIE/k6VRzzZs316pVq/LyEAAAAADgMHkakLZu3cp03wAAAABchl2TNHz66adZLr906ZI2btyo5cuXq1+/fvYcAgAAAADyjV0BqU+fPtmuCwwM1OjRoxUdHW3PIQAAAAAg39gVkI4cOZJpmcViUYkSJVSsWDF7dg0AAOB0lUZ/7ewSCqSjb3RwdglAnrErIFWsWNFRdQAAAACA0zGDAgAAAACY7LqD5ObmJovFkqttLBaLUlNT7TksAAAAAOQJuwJSdHS0Vq5cqb179yoqKkr333+/JOnAgQP67rvvFBYWps6dOzuiTgAAAADIc3YFpJCQECUkJGjPnj3WcJRh//79atmypUJCQvTCCy/YVSQAAAAA5Ae7nkF66623NGTIkEzhSJJq1qypIUOGaMqUKfYcAgAAAADyjV0B6cSJE/L09Mx2vaenp06cOGHPIQAAAAAg39gVkMLCwvTee+/p5MmTmdadOHFC7733nurUqWPPIQAAAAAg39j1DNI///lPRUVFqXr16urSpYvuu+8+SVJ8fLxWrlwpwzD02WefOaRQAAAAAMhrdgWkpk2b6ueff9b48eO1YsUKXbt2TZLk6+urqKgoxcbGcgcJAAAAgMuwKyBJfw6zW7FihdLT03Xu3DlJUlBQkNzc+A1aAAAAAK7F7oCUwc3NTT4+PvLz8yMcAQAAAHBJdieZrVu3qm3btipSpIhKlSqlDRs2SJLOnz+vxx9/XOvXr7f3EAAAAACQL+wKSJs3b1bTpk0VHx+vnj17Kj093bouMDBQiYmJmj17tt1FAgAAAEB+sCsgjR07VjVr1tS+ffs0adKkTOtbtGihn3/+2Z5DAAAAAEC+sSsg/frrr+rbt6+8vb1lsVgyrS9XrpzOnDljzyEAAAAAIN/YFZA8PT1thtX91cmTJ+Xn52fPIQAAAAAg39gVkB5++GEtW7Ysy3XJycmaO3euIiIi7DkEAAAAAOQbuwJSbGystm7dqg4dOuibb76RJO3atUsffvihwsPDde7cOY0fP94hhQIAAABAXrPrd5AaNWqkVatW6cUXX9Szzz4rSfr73/8uSapatapWrVqlunXr2l8lAAAAAOSDuw5IhmHo8uXLeuSRR/T7779r586dio+PV3p6uqpWrarw8PAsJ24AAAAAgILqrofY3bhxQyVLltS7774rSapfv766du2q7t2766GHHsqzcHTy5En17NlTpUqVkq+vr+rUqaOtW7fmybEAAAAA3Fvu+g6St7e3goOD5e3t7ch6buvixYtq0qSJWrRooW+++UZBQUGKj49XiRIl8q0GAAAAAIWXXc8g9enTR59++qlefPFFeXl5OaqmbL355psKDQ3V3LlzrcsqV66c58cFAAAAcG+wKyDVqVNHK1euVO3atdWnTx9VqlRJvr6+mdo98cQT9hzG6ssvv1RUVJS6du2qDRs2qFy5cho0aJBeeOEFh+wfAAAAwL3NroD01FNPWf85u+m8LRaL0tLS7DmM1eHDhzVr1iyNHDlSY8eO1a+//qphw4bJy8tLvXv3znKblJQUpaSkWN8nJSU5pBYAAAAAhU+uA9LYsWPVo0cP1a1bV+vWrcuLmrKVnp6uhx56SJMmTZIkPfDAA9qzZ4/ef//9bAPS5MmTFRsbm59lAkCBUGn0184uAS7k6BsdnF0CABQIuQ5Ib7zxhsLCwlS3bl1FRETowoULKl26tNasWaOWLVvmRY1WZcuWVa1atWyW1axZU//617+y3WbMmDEaOXKk9X1SUpJCQ0PzrEYAAAAArsuuIXYZDMNwxG5y1KRJE/3+++82y/773/+qYsWK2W7j7e2drzPtAQAAAHBdd/07SM7w0ksvacuWLZo0aZIOHjyohQsXas6cORo8eLCzSwMAAABQCLhUQGrQoIFWrFihRYsWKSwsTK+//rri4uL0zDPPOLs0AAAAAIXAXQ2xO3r0qLZv3y5JSkxMlCTFx8erePHiWbZ/8MEH7666LHTs2FEdO3Z02P4AAAAAIMNdBaTx48dnmtZ70KBBmdoZhuHQab4BAAAAIC/lOiDNnTs3L+oAAAAAAKfLdUDK7veGAAAAAMDVudQkDQAAAACQlwhIAAAAAGAiIAEAAACAiYAEAAAAACYCEgAAAACYCEgAAAAAYCIgAQAAAICJgAQAAAAAJgISAAAAAJgISAAAAABgIiABAAAAgImABAAAAAAmAhIAAAAAmAhIAAAAAGAiIAEAAACAiYAEAAAAACYCEgAAAACYPJxdAAAAcL5Ko792dgkAUCBwBwkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwuHZDeeOMNWSwWjRgxwtmlAAAAACgEXDYg/frrr5o9e7bq1q3r7FIAAAAAFBIuGZCuXLmiZ555Rh988IFKlCjh7HIAAAAAFBIuGZAGDx6sDh06qFWrVjm2TUlJUVJSks0LAAAAALLi4ewCcuvzzz/X9u3b9euvv95R+8mTJys2NjaPq7p7lUZ/7ewSCqSjb3RwdgkAACAbfH/JGt9fCgeXuoN0/PhxDR8+XAsWLJCPj88dbTNmzBglJiZaX8ePH8/jKgEAAAC4Kpe6g7Rt2zYlJCTowQcftC5LS0vTxo0bNWPGDKWkpMjd3d1mG29vb3l7e+d3qQAAAABckEsFpEcffVS7d++2Wda3b1/VqFFDr776aqZwBAAAAAC54VIBqVixYgoLC7NZVrRoUZUqVSrTcgAAAADILZd6BgkAAAAA8pJL3UHKyvr1651dAgAAAIBCgjtIAAAAAGAiIAEAAACAiYAEAAAAACYCEgAAAACYCEgAAAAAYCIgAQAAAICJgAQAAAAAJgISAAAAAJgISAAAAABgIiABAAAAgImABAAAAAAmAhIAAAAAmAhIAAAAAGAiIAEAAACAiYAEAAAAACYCEgAAAACYCEgAAAAAYCIgAQAAAICJgAQAAAAAJg9nFwBkpdLor51dAgAAQK7w/SVrR9/o4OwScoU7SAAAAABgIiABAAAAgImABAAAAAAmAhIAAAAAmAhIAAAAAGAiIAEAAACAiYAEAAAAACYCEgAAAACYCEgAAAAAYCIgAQAAAICJgAQAAAAAJgISAAAAAJgISAAAAABgIiABAAAAgImABAAAAAAmAhIAAAAAmAhIAAAAAGAiIAEAAACAiYAEAAAAACYCEgAAAACYCEgAAAAAYHKpgDR58mQ1aNBAxYoVU+nSpdW5c2f9/vvvzi4LAAAAQCHhUgFpw4YNGjx4sLZs2aI1a9bo5s2batOmjZKTk51dGgAAAIBCwMPZBeTG6tWrbd7PmzdPpUuX1rZt29S8eXMnVQUAAACgsHCpgPRXiYmJkqSSJUtm2yYlJUUpKSnW90lJSXleFwAAAADX5FJD7G6Vnp6uESNGqEmTJgoLC8u23eTJkxUQEGB9hYaG5mOVAAAAAFyJywakwYMHa8+ePfr8889v227MmDFKTEy0vo4fP55PFQIAAABwNS45xG7IkCH66quvtHHjRpUvX/62bb29veXt7Z1PlQEAAABwZS4VkAzD0NChQ7VixQqtX79elStXdnZJAAAAAAoRlwpIgwcP1sKFC/XFF1+oWLFiOnPmjCQpICBAvr6+Tq4OAAAAgKtzqWeQZs2apcTEREVGRqps2bLW1+LFi51dGgAAAIBCwKXuIBmG4ewSAAAAABRiLnUHCQAAAADyEgEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAAATAQkAAAAATAQkAAAAADARkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJAAAAAAwEZAAAAAAwERAAgAAAACTSwakmTNnqlKlSvLx8VGjRo30yy+/OLskAAAAAIWAywWkxYsXa+TIkYqJidH27dtVr149RUVFKSEhwdmlAQAAAHBxLheQpk2bphdeeEF9+/ZVrVq19P7776tIkSL6+OOPnV0aAAAAABfnUgHpxo0b2rZtm1q1amVd5ubmplatWumnn35yYmUAAAAACgMPZxeQG+fPn1daWprKlCljs7xMmTI6cOBAltukpKQoJSXF+j4xMVGSlJSUlHeF5kJ6ylVnlwAAAADkmYLyvTujDsMwbtvOpQLS3Zg8ebJiY2MzLQ8NDXVCNQAAAMC9JSDO2RXYunz5sgICArJd71IBKTAwUO7u7jp79qzN8rNnzyo4ODjLbcaMGaORI0da36enp+t///ufSpUqJYvFkqf1OlpSUpJCQ0N1/Phx+fv7O7ucQo/+zj/0df6hr/MX/Z1/6Ov8Q1/nH/rasQzD0OXLlxUSEnLbdi4VkLy8vBQeHq61a9eqc+fOkv4MPGvXrtWQIUOy3Mbb21ve3t42y4oXL57HleYtf39//iXJR/R3/qGv8w99nb/o7/xDX+cf+jr/0NeOc7s7RxlcKiBJ0siRI9W7d2899NBDatiwoeLi4pScnKy+ffs6uzQAAAAALs7lAlL37t117tw5RUdH68yZM6pfv75Wr16daeIGAAAAAMgtlwtIkjRkyJBsh9QVZt7e3oqJick0ZBB5g/7OP/R1/qGv8xf9nX/o6/xDX+cf+to5LEZO89wBAAAAwD3CpX4oFgAAAADyEgEJAAAAAEwEJAAAAAAwEZAAAAAAwERAcrKZM2eqUqVK8vHxUaNGjfTLL7/ctn1cXJzuv/9++fr6KjQ0VC+99JKuX79uXT9hwgRZLBabV40aNfL6NFxCbvr65s2beu2111S1alX5+PioXr16Wr16tV37vJc4uq+5rrO2ceNGderUSSEhIbJYLFq5cmWO26xfv14PPvigvL29dd9992nevHmZ2nBdZy0v+ptrO2u57evTp0/r6aefVvXq1eXm5qYRI0Zk2W7p0qWqUaOGfHx8VKdOHa1atcrxxbuYvOjrefPmZbqufXx88uYEXEhu+3r58uVq3bq1goKC5O/vr8aNG+vbb7/N1I6/2Y5HQHKixYsXa+TIkYqJidH27dtVr149RUVFKSEhIcv2Cxcu1OjRoxUTE6P9+/fro48+0uLFizV27FibdrVr19bp06etr02bNuXH6RRoue3rcePGafbs2Zo+fbr27dungQMHqkuXLtqxY8dd7/NekRd9LXFdZyU5OVn16tXTzJkz76j9kSNH1KFDB7Vo0UI7d+7UiBEj1K9fP5v/w+W6zl5e9LfEtZ2V3PZ1SkqKgoKCNG7cONWrVy/LNps3b9ZTTz2l559/Xjt27FDnzp3VuXNn7dmzx5Glu5y86GtJ8vf3t7mujx075qiSXVZu+3rjxo1q3bq1Vq1apW3btqlFixbq1KkT30XygwGnadiwoTF48GDr+7S0NCMkJMSYPHlylu0HDx5stGzZ0mbZyJEjjSZNmljfx8TEGPXq1cuTel1Zbvu6bNmyxowZM2yWPfHEE8Yzzzxz1/u8V+RFX3Nd50ySsWLFitu2eeWVV4zatWvbLOvevbsRFRVlfc91fWcc1d9c2zm7k76+VUREhDF8+PBMy7t162Z06NDBZlmjRo2MAQMG2Flh4eGovp47d64REBDgsLoKo9z2dYZatWoZsbGx1vf8zc4b3EFykhs3bmjbtm1q1aqVdZmbm5tatWqln376KcttHnnkEW3bts166/Tw4cNatWqV2rdvb9MuPj5eISEhqlKlip555hn98ccfeXciLuBu+jolJSXTcABfX1/rf9m9m33eC/KirzNwXdvvp59+svlsJCkqKsr62XBdO1ZO/Z2Bazt/3OnnAce4cuWKKlasqNDQUD3++OPau3evs0tyeenp6bp8+bJKliwpib/ZeYmA5CTnz59XWlqaypQpY7O8TJkyOnPmTJbbPP3003rttdfUtGlTeXp6qmrVqoqMjLQZYteoUSPNmzdPq1ev1qxZs3TkyBE1a9ZMly9fztPzKcjupq+joqI0bdo0xcfHKz09XWvWrNHy5ct1+vTpu97nvSAv+lriunaUM2fOZPnZJCUl6dq1a1zXDpZTf0tc2/kpu8+Da9vx7r//fn388cf64osv9Nlnnyk9PV2PPPKITpw44ezSXNrUqVN15coVdevWTRLfRfISAcmFrF+/XpMmTdJ7772n7du3a/ny5fr666/1+uuvW9u0a9dOXbt2Vd26dRUVFaVVq1bp0qVLWrJkiRMrdz3vvPOOqlWrpho1asjLy0tDhgxR37595ebGvzKOdid9zXWNwoprG4VR48aN9eyzz6p+/fqKiIjQ8uXLFRQUpNmzZzu7NJe1cOFCxcbGasmSJSpdurSzyyn0+LbnJIGBgXJ3d9fZs2dtlp89e1bBwcFZbjN+/Hj16tVL/fr1U506ddSlSxdNmjRJkydPVnp6epbbFC9eXNWrV9fBgwcdfg6u4m76OigoSCtXrlRycrKOHTumAwcOyM/PT1WqVLnrfd4L8qKvs8J1fXeCg4Oz/Gz8/f3l6+vLde1gOfV3Vri28052nwfXdt7z9PTUAw88wHV9lz7//HP169dPS5YssRlOx9/svENAchIvLy+Fh4dr7dq11mXp6elau3atGjdunOU2V69ezXQHw93dXZJkGEaW21y5ckWHDh1S2bJlHVS567mbvs7g4+OjcuXKKTU1Vf/617/0+OOP273Pwiwv+jorXNd3p3HjxjafjSStWbPG+tlwXTtWTv2dFa7tvHM3nwccIy0tTbt37+a6vguLFi1S3759tWjRInXo0MFmHX+z85CzZ4m4l33++eeGt7e3MW/ePGPfvn1G//79jeLFixtnzpwxDMMwevXqZYwePdraPiYmxihWrJixaNEi4/Dhw8Z3331nVK1a1ejWrZu1zd///ndj/fr1xpEjR4wff/zRaNWqlREYGGgkJCTk+/kVJLnt6y1bthj/+te/jEOHDhkbN240WrZsaVSuXNm4ePHiHe/zXpUXfc11nbXLly8bO3bsMHbs2GFIMqZNm2bs2LHDOHbsmGEYhjF69GijV69e1vaHDx82ihQpYrz88svG/v37jZkzZxru7u7G6tWrrW24rrOXF/3NtZ213Pa1YRjW9uHh4cbTTz9t7Nixw9i7d691/Y8//mh4eHgYU6dONfbv32/ExMQYnp6exu7du/P13AqavOjr2NhY49tvvzUOHTpkbNu2zejRo4fh4+Nj0+ZelNu+XrBggeHh4WHMnDnTOH36tPV16dIlaxv+ZucNApKTTZ8+3ahQoYLh5eVlNGzY0NiyZYt1XUREhNG7d2/r+5s3bxoTJkwwqlatavj4+BihoaHGoEGDbL5Idu/e3Shbtqzh5eVllCtXzujevbtx8ODBfDyjgis3fb1+/XqjZs2ahre3t1GqVCmjV69exsmTJ3O1z3uZo/ua6zpr69atMyRlemX0b+/evY2IiIhM29SvX9/w8vIyqlSpYsydOzfTfrmus5YX/c21nbW76eus2lesWNGmzZIlS4zq1asbXl5eRu3atY2vv/46f06oAMuLvh4xYoT1b0iZMmWM9u3bG9u3b8+/kyqgctvXERERt22fgb/ZjmcxjGzGZgEAAADAPYZnkAAAAADAREACAAAAABMBCQAAAABMBCQAAAAAMBGQAAAAAMBEQAIAAAAAEwEJAAAAAEwEJACAVaVKldSnTx/r+/Xr18tisWj9+vUOO4bFYtGECRMctj/kj8jISEVGRjq7DADIcwQkACgg5s2bJ4vFYn35+PioevXqGjJkiM6ePevs8nJl1apVBS4EXb16VRMmTHBo2MtLzqh33759mjBhgo4ePZpvxwSAgsbD2QUAAGy99tprqly5sq5fv65NmzZp1qxZWrVqlfbs2aMiRYrkay3NmzfXtWvX5OXllavtVq1apZkzZ2YZkq5duyYPj/z/v5+rV68qNjZWklziTogz6t23b59iY2MVGRmpSpUq2az77rvv8qUGAHA2AhIAFDDt2rXTQw89JEnq16+fSpUqpWnTpumLL77QU089leU2ycnJKlq0qMNrcXNzk4+Pj0P36ej9OVte9b0jOLK23IZkAHBVDLEDgAKuZcuWkqQjR45Ikvr06SM/Pz8dOnRI7du3V7FixfTMM89IktLT0xUXF6fatWvLx8dHZcqU0YABA3Tx4kWbfRqGoYkTJ6p8+fIqUqSIWrRoob1792Y6dnbPIP38889q3769SpQooaJFi6pu3bp65513rPXNnDlTkmyGDGbI6hmkHTt2qF27dvL395efn58effRRbdmyxaZNxhDEH3/8USNHjlRQUJCKFi2qLl266Ny5czZtExMTdeDAASUmJkqSjh49qqCgIElSbGystaaMOn777Tf16dNHVapUkY+Pj4KDg/Xcc8/pwoULNvudMGGCLBaL9u3bp6efflolSpRQ06ZNrX0/YcIEhYSEWPt03759mZ7rkqRLly5pxIgRCg0Nlbe3t+677z69+eabSk9Pv6N6s5LRPxs2bNCgQYNUunRplS9fXpJ07NgxDRo0SPfff798fX1VqlQpde3a1WYo3bx589S1a1dJUosWLazHzPjss3oGKSEhQc8//7zKlCkjHx8f1atXT5988km2NQKAK+AOEgAUcIcOHZIklSpVyrosNTVVUVFRatq0qaZOnWodejdgwADNmzdPffv21bBhw3TkyBHNmDFDO3bs0I8//ihPT09JUnR0tCZOnKj27durffv22r59u9q0aaMbN27kWM+aNWvUsWNHlS1bVsOHD1dwcLD279+vr776SsOHD9eAAQN06tQprVmzRvPnz89xf3v37lWzZs3k7++vV155RZ6enpo9e7YiIyO1YcMGNWrUyKb90KFDVaJECcXExOjo0aOKi4vTkCFDtHjxYmubFStWqG/fvpo7d6769OmjoKAgzZo1Sy+++KK6dOmiJ554QpJUt25d6zkdPnxYffv2VXBwsPbu3as5c+Zo79692rJli03Ak6SuXbuqWrVqmjRpkgzDkCSNGTNGU6ZMUadOnRQVFaVdu3YpKipK169ft9n26tWrioiI0MmTJzVgwABVqFBBmzdv1pgxY3T69GnFxcXlWO/tDBo0SEFBQYqOjlZycrIk6ddff9XmzZvVo0cPlS9fXkePHtWsWbMUGRmpffv2qUiRImrevLmGDRumd999V2PHjlXNmjUlyfq/f3Xt2jVFRkbq4MGDGjJkiCpXrqylS5eqT58+unTpkoYPH55jrQBQIBkAgAJh7ty5hiTj+++/N86dO2ccP37c+Pzzz41SpUoZvr6+xokTJwzDMIzevXsbkozRo0fbbP/DDz8YkowFCxbYLF+9erXN8oSEBMPLy8vo0KGDkZ6ebm03duxYQ5LRu3dv67J169YZkox169YZhmEYqampRuXKlY2KFSsaFy9etDnOrfsaPHiwkd3/xUgyYmJirO87d+5seHl5GYcOHbIuO3XqlFGsWDGjefPmmfqnVatWNsd66aWXDHd3d+PSpUuZ2s6dO9e67Ny5c5mOneHq1auZli1atMiQZGzcuNG6LCYmxpBkPPXUUzZtz5w5Y3h4eBidO3e2WT5hwoRMffr6668bRYsWNf773//atB09erTh7u5u/PHHHznWm5WMc27atKmRmpqa4/n99NNPhiTj008/tS5bunSpzed9q4iICCMiIsL6Pi4uzpBkfPbZZ9ZlN27cMBo3bmz4+fkZSUlJd1Q3ABQ0DLEDgAKmVatWCgoKUmhoqHr06CE/Pz+tWLFC5cqVs2n34osv2rxfunSpAgIC1Lp1a50/f976Cg8Pl5+fn9atWydJ+v7773Xjxg0NHTrU5s7IiBEjcqxtx44dOnLkiEaMGKHixYvbrPvrXZY7kZaWpu+++06dO3dWlSpVrMvLli2rp59+Wps2bVJSUpLNNv3797c5VrNmzZSWlqZjx45Zl/Xp00eGYWQa2pYdX19f6z9fv35d58+f18MPPyxJ2r59e6b2AwcOtHm/du1apaamatCgQTbLhw4dmmnbpUuXqlmzZipRooTN59SqVSulpaVp48aNd1Rzdl544QW5u7vbLLv1/G7evKkLFy7ovvvuU/HixbM8vzuxatUqBQcH2zwX5+npqWHDhunKlSvasGHD3Z0AADgZQ+wAoICZOXOmqlevLg8PD5UpU0b333+/3Nxs/3uWh4eH9fmSDPHx8UpMTFTp0qWz3G9CQoIkWYNEtWrVbNYHBQWpRIkSt60tY7hfWFjYnZ/QbZw7d05Xr17V/fffn2ldzZo1lZ6eruPHj6t27drW5RUqVLBpl1HzX5+zyo3//e9/io2N1eeff27tpwwZzzHdqnLlyjbvM/r0vvvus1lesmTJTH0aHx+v3377zfqM0V/99fi59dfapD+Hw02ePFlz587VyZMnrcMCpazP704cO3ZM1apVy3RtZgzJuzWwAoArISABQAHTsGFD6yx22fH29s70xTQ9PV2lS5fWggULstwmuy/kruavd0cy3PqlP7e6deumzZs36+WXX1b9+vXl5+en9PR0tW3b1jpxwq1uvSOTW+np6WrdurVeeeWVLNdXr179rvctZV3b0KFDNXfuXI0YMUKNGzdWQECALBaLevTokeX5AcC9jIAEAIVE1apV9f3336tJkya3/QJfsWJFSX/eybh1WNu5c+dyvAtTtWpVSdKePXvUqlWrbNvd6XC7oKAgFSlSRL///numdQcOHJCbm5tCQ0PvaF85ya6mixcvau3atYqNjVV0dLR1eXx8/B3vO6NPDx48aHMH58KFC5n6tGrVqrpy5cpt++929d6NZcuWqXfv3nr77bety65fv65Lly7d9TErVqyo3377Tenp6TZh/cCBA9b1AOCKeAYJAAqJbt26KS0tTa+//nqmdampqdYvw61atZKnp6emT59uc9clLi4ux2M8+OCDqly5suLi4jJ9ub51Xxm/vfPXNn/l7u6uNm3a6IsvvrCZcvrs2bNauHChmjZtKn9//xzr+qu/TvMtyTrT319ryrgj9dc7UHfSHxkeffRReXh4aNasWTbLZ8yYkaltt27d9NNPP+nbb7/NtO7SpUtKTU29bb13w93dPdP5TZ8+XWlpaTbL7vRzk6T27dvrzJkzNrMHpqamavr06fLz81NERITddQOAM3AHCQAKiYiICA0YMECTJ0/Wzp071aZNG3l6eio+Pl5Lly7VO++8oyeffFJBQUEaNWqUJk+erI4dO6p9+/basWOHvvnmGwUGBt72GG5ubpo1a5Y6deqk+vXrq2/fvipbtqwOHDigvXv3Wr/0h4eHS5KGDRumqKgoubu7q0ePHlnuc+LEiVqzZo2aNm2qQYMGycPDQ7Nnz1ZKSoqmTJlyV33x12m+pT+HntWqVUuLFy9W9erVVbJkSYWFhSksLEzNmzfXlClTdPPmTZUrV07fffed9Xen7kSZMmU0fPhwvf3223rsscfUtm1b7dq1y9qnt96Zefnll/Xll1+qY8eO6tOnj8LDw5WcnKzdu3dr2bJlOnr0qAIDA29bb2517NhR8+fPV0BAgGrVqqWffvpJ33//vc3U8ZJUv359ubu7680331RiYqK8vb3VsmXLLJ9r69+/v2bPnq0+ffpo27ZtqlSpkpYtW6Yff/xRcXFxKlasWK7rBIACwYkz6AEAbpExTfOvv/5623a9e/c2ihYtmu36OXPmGOHh4Yavr69RrFgxo06dOsYrr7xinDp1ytomLS3NiI2NNcqWLWv4+voakZGRxp49e4yKFSvedprvDJs2bTJat25tFCtWzChatKhRt25dY/r06db1qampxtChQ42goCDDYrHYTPmtLKau3r59uxEVFWX4+fkZRYoUMVq0aGFs3rz5jvonqxqzmubbMAxj8+bNRnh4uOHl5WVTx4kTJ4wuXboYxYsXNwICAoyuXbsap06dylRrxjTf586dy9Tvqampxvjx443g4GDD19fXaNmypbF//36jVKlSxsCBA23aXr582RgzZoxx3333GV5eXkZgYKDxyCOPGFOnTjVu3LiRY71Zud31c/HiRaNv375GYGCg4efnZ0RFRRkHDhzI9HkbhmF88MEHRpUqVQx3d3ebfv3rNN+GYRhnz5617tfLy8uoU6dOpj4HAFdjMQw7nmoFAADZunTpkkqUKKGJEyfq//2//+fscgAAd4BnkAAAcIBr165lWpbxHFNkZGT+FgMAuGs8gwQAgAMsXrxY8+bNU/v27eXn56dNmzZp0aJFatOmjZo0aeLs8gAAd4iABACAA9StW1ceHh6aMmWKkpKSrBM3TJw40dmlAQBygWeQAAAAAMDEM0gAAAAAYCIgAQAAAICJgAQAAAAAJgISAAAAAJgISAAAAABgIiABAAAAgImABAAAAAAmAhIAAAAAmAhIAAAAAGD6/40cIEkAZBQ2AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from torch import concat\n",
    "from numpy import diff\n",
    "\n",
    "model.eval()\n",
    "\n",
    "predictions = []\n",
    "targets = []\n",
    "\n",
    "for batch, target in valid_loader:\n",
    "    with no_grad():\n",
    "        predictions.append(model(batch))\n",
    "        targets.append(target)\n",
    "\n",
    "predictions = concat(predictions).flatten()\n",
    "targets = concat(targets).flatten()\n",
    "\n",
    "ratios = predictions / targets\n",
    "\n",
    "fig, axis = plt.subplots(1, 1, figsize=(10, 5))\n",
    "\n",
    "axis.hist(\n",
    "    ratios,\n",
    "    bins=10,\n",
    "    label=rf\"Mean {ratios.mean():.3f}; Std dev. {predictions.std():.3f}\",\n",
    ")\n",
    "axis.set_xlabel(\"Prediction:target ratio\", fontsize=12)\n",
    "axis.set_ylabel(\"Frequency\", fontsize=12)\n",
    "\n",
    "axis.legend(fontsize=12)\n",
    "\n",
    "axis.set_aspect(0.5 * diff(axis.get_xlim()) / diff(axis.get_ylim()))\n",
    "\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}