char-rnn-generation/char-rnn-generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://i.imgur.com/eBRPvWB.png)\n",
    "\n",
    "# Practical PyTorch: Generating Shakespeare with a Character-Level RNN\n",
    "\n",
    "[In the RNN classification tutorial](https://github.com/spro/practical-pytorch/blob/master/char-rnn-classification/char-rnn-classification.ipynb) we used a RNN to classify text one character at a time. This time we'll generate text one character at a time.\n",
    "\n",
    "```\n",
    "> python generate.py -n 500\n",
    "\n",
    "PAOLTREDN:\n",
    "Let, yil exter shis owrach we so sain, fleas,\n",
    "Be wast the shall deas, puty sonse my sheete.\n",
    "\n",
    "BAUFIO:\n",
    "Sirh carrow out with the knonuot my comest sifard queences\n",
    "O all a man unterd.\n",
    "\n",
    "PROMENSJO:\n",
    "Ay, I to Heron, I sack, againous; bepear, Butch,\n",
    "An as shalp will of that seal think.\n",
    "\n",
    "NUKINUS:\n",
    "And house it to thee word off hee:\n",
    "And thou charrota the son hange of that shall denthand\n",
    "For the say hor you are of I folles muth me?\n",
    "```\n",
    "\n",
    "This one might make you question the series title &mdash; \"is that really practical?\" However, these sorts of generative models form the basis of machine translation, image captioning, question answering and more. See the [Sequence to Sequence Translation tutorial](https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb) for more on that topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommended Reading\n",
    "\n",
    "I assume you have at least installed PyTorch, know Python, and understand Tensors:\n",
    "\n",
    "* http://pytorch.org/ For installation instructions\n",
    "* [Deep Learning with PyTorch: A 60-minute Blitz](https://github.com/pytorch/tutorials/blob/master/Deep%20Learning%20with%20PyTorch.ipynb) to get started with PyTorch in general\n",
    "* [jcjohnson's PyTorch examples](https://github.com/jcjohnson/pytorch-examples) for an in depth overview\n",
    "* [Introduction to PyTorch for former Torchies](https://github.com/pytorch/tutorials/blob/master/Introduction%20to%20PyTorch%20for%20former%20Torchies.ipynb) if you are former Lua Torch user\n",
    "\n",
    "It would also be useful to know about RNNs and how they work:\n",
    "\n",
    "* [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) shows a bunch of real life examples\n",
    "* [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is about LSTMs specifically but also informative about RNNs in general\n",
    "\n",
    "Also see these related tutorials from the series:\n",
    "\n",
    "* [Classifying Names with a Character-Level RNN](https://github.com/spro/practical-pytorch/blob/master/char-rnn-classification/char-rnn-classification.ipynb) uses an RNN for classification\n",
    "* [Generating Names with a Conditional Character-Level RNN](https://github.com/spro/practical-pytorch/blob/master/conditional-char-rnn/conditional-char-rnn.ipynb) builds on this model to add a category as input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare data\n",
    "\n",
    "The file we are using is a plain text file. We turn any potential unicode characters into plain ASCII by using the `unidecode` package (which you can install via `pip` or `conda`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file_len = 1115394\n"
     ]
    }
   ],
   "source": [
    "import unidecode\n",
    "import string\n",
    "import random\n",
    "import re\n",
    "\n",
    "all_characters = string.printable\n",
    "n_characters = len(all_characters)\n",
    "\n",
    "file = unidecode.unidecode(open('../data/shakespeare.txt').read())\n",
    "file_len = len(file)\n",
    "print('file_len =', file_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make inputs out of this big string of data, we will be splitting it into chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " will continue that I broach'd in jest.\n",
      "I can, Petruchio, help thee to a wife\n",
      "With wealth enough and young and beauteous,\n",
      "Brought up as best becomes a gentlewoman:\n",
      "Her only fault, and that is faults en\n"
     ]
    }
   ],
   "source": [
    "chunk_len = 200\n",
    "\n",
    "def random_chunk():\n",
    "    start_index = random.randint(0, file_len - chunk_len)\n",
    "    end_index = start_index + chunk_len + 1\n",
    "    return file[start_index:end_index]\n",
    "\n",
    "print(random_chunk())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the Model\n",
    "\n",
    "This model will take as input the character for step $t_{-1}$ and is expected to output the next character $t$. There are three layers - one linear layer that encodes the input character into an internal state, one GRU layer (which may itself have multiple layers) that operates on that internal state and a hidden state, and a decoder layer that outputs the probability distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size, n_layers=1):\n",
    "        super(RNN, self).__init__()\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        \n",
    "        self.encoder = nn.Embedding(input_size, hidden_size)\n",
    "        self.gru = nn.GRU(hidden_size, hidden_size, n_layers)\n",
    "        self.decoder = nn.Linear(hidden_size, output_size)\n",
    "    \n",
    "    def forward(self, input, hidden):\n",
    "        input = self.encoder(input.view(1, -1))\n",
    "        output, hidden = self.gru(input.view(1, 1, -1), hidden)\n",
    "        output = self.decoder(output.view(1, -1))\n",
    "        return output, hidden\n",
    "\n",
    "    def init_hidden(self):\n",
    "        return Variable(torch.zeros(self.n_layers, 1, self.hidden_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inputs and Targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each chunk will be turned into a tensor, specifically a `LongTensor` (used for integer values), by looping through the characters of the string and looking up the index of each character in `all_characters`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 10\n",
      " 11\n",
      " 12\n",
      " 39\n",
      " 40\n",
      " 41\n",
      "[torch.LongTensor of size 6]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Turn string into list of longs\n",
    "def char_tensor(string):\n",
    "    tensor = torch.zeros(len(string)).long()\n",
    "    for c in range(len(string)):\n",
    "        tensor[c] = all_characters.index(string[c])\n",
    "    return Variable(tensor)\n",
    "\n",
    "print(char_tensor('abcDEF'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can assemble a pair of input and target tensors for training, from a random chunk. The input will be all characters *up to the last*, and the target will be all characters *from the first*. So if our chunk is \"abc\" the input will correspond to \"ab\" while the target is \"bc\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def random_training_set():    \n",
    "    chunk = random_chunk()\n",
    "    inp = char_tensor(chunk[:-1])\n",
    "    target = char_tensor(chunk[1:])\n",
    "    return inp, target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating\n",
    "\n",
    "To evaluate the network we will feed one character at a time, use the outputs of the network as a probability distribution for the next character, and repeat. To start generation we pass a priming string to start building up the hidden state, from which we then generate one character at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def evaluate(prime_str='A', predict_len=100, temperature=0.8):\n",
    "    hidden = decoder.init_hidden()\n",
    "    prime_input = char_tensor(prime_str)\n",
    "    predicted = prime_str\n",
    "\n",
    "    # Use priming string to \"build up\" hidden state\n",
    "    for p in range(len(prime_str) - 1):\n",
    "        _, hidden = decoder(prime_input[p], hidden)\n",
    "    inp = prime_input[-1]\n",
    "    \n",
    "    for p in range(predict_len):\n",
    "        output, hidden = decoder(inp, hidden)\n",
    "        \n",
    "        # Sample from the network as a multinomial distribution\n",
    "        output_dist = output.data.view(-1).div(temperature).exp()\n",
    "        top_i = torch.multinomial(output_dist, 1)[0]\n",
    "        \n",
    "        # Add predicted character to string and use as next input\n",
    "        predicted_char = all_characters[top_i]\n",
    "        predicted += predicted_char\n",
    "        inp = char_tensor(predicted_char)\n",
    "\n",
    "    return predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A helper to print the amount of time passed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import time, math\n",
    "\n",
    "def time_since(since):\n",
    "    s = time.time() - since\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main training function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(inp, target):\n",
    "    hidden = decoder.init_hidden()\n",
    "    decoder.zero_grad()\n",
    "    loss = 0\n",
    "\n",
    "    for c in range(chunk_len):\n",
    "        output, hidden = decoder(inp[c], hidden)\n",
    "        loss += criterion(output, target[c])\n",
    "\n",
    "    loss.backward()\n",
    "    decoder_optimizer.step()\n",
    "\n",
    "    return loss.data[0] / chunk_len"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define the training parameters, instantiate the model, and start training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0m 19s (100 5%) 2.1267]\n",
      "Wh! 'Lod at the to Cell I dy\n",
      "Whapesfe show dous that,\n",
      "But thes lo he ther, letrst surave and and cod a \n",
      "\n",
      "[0m 38s (200 10%) 1.9876]\n",
      "Whan the she ciching, doove whath that he gone prie hasigrow nice knotat by wiith haye! ha coll, and i \n",
      "\n",
      "[0m 59s (300 15%) 2.0772]\n",
      "Whurgre of nowif for of agand witeling in fromound be noyed th well and fort and withen a custrone fri \n",
      "\n",
      "[1m 19s (400 20%) 1.9062]\n",
      "Why sleemer chome, I\n",
      "tence lord thou let not mories, Wherly me cloonger on wit, me cre wort if thing i \n",
      "\n",
      "[1m 39s (500 25%) 1.9632]\n",
      "Whank of winded than inderreast, hids for hink marry, I son will now my be tor think that I be uncient \n",
      "\n",
      "[2m 0s (600 30%) 1.9364]\n",
      "What to youre\n",
      "Good the dorsentemat.\n",
      "What the not what a meifery part is be of look\n",
      "Whait of the hall w \n",
      "\n",
      "[2m 20s (700 35%) 1.8673]\n",
      "Whes Bester,\n",
      "Bars, and and most man\n",
      "ingeld my tiement make I lesiefoden as do you same to muse woke o' \n",
      "\n",
      "[2m 40s (800 40%) 2.1523]\n",
      "Whe my bone a me but mast at the face.\n",
      "Whe he frend him cope a be to with I comes or he God his for ma \n",
      "\n",
      "[3m 1s (900 45%) 1.8042]\n",
      "Whis our namure.\n",
      "\n",
      "TRANIO:\n",
      "May platis the lord,\n",
      "I wis he we but he hards paron's we for the surven neav \n",
      "\n",
      "[3m 21s (1000 50%) 1.9770]\n",
      "Whis, is at ell demes sy host is in\n",
      "The revention eart-aly, his the couth stare.\n",
      "The streath, the so h \n",
      "\n",
      "[3m 42s (1100 55%) 1.9771]\n",
      "Which the called these what mace all bries,\n",
      "Gow the from ceart repise--tring be of the\n",
      "Hee he that, of \n",
      "\n",
      "[4m 3s (1200 60%) 1.7054]\n",
      "What that hays how the frow he dresers gard.\n",
      "\n",
      "BAPTISTA:\n",
      "That was on a prain their with to goe, all me\n",
      " \n",
      "\n",
      "[4m 23s (1300 65%) 1.6584]\n",
      "Whe time, like\n",
      "Those paurstriet.\n",
      "\n",
      "SICINIUS:\n",
      "Glow a and elfers; rother's Rome servest enon't is may thu \n",
      "\n",
      "[4m 44s (1400 70%) 1.7370]\n",
      "When him these;\n",
      "There and of Have the in of the do best veath and hever the chaw, not pites with at my \n",
      "\n",
      "[5m 6s (1500 75%) 1.6769]\n",
      "Wher he have live the courtas,\n",
      "I here that whils him I shee my like deated,\n",
      "To countert a hardor of so \n",
      "\n",
      "[5m 26s (1600 80%) 1.7480]\n",
      "Wh for the grone them with are\n",
      "Belent dis are couch of my to tell ding.\n",
      "\n",
      "Sir:\n",
      "What the deatred thou as \n",
      "\n",
      "[5m 48s (1700 85%) 1.7725]\n",
      "Why.\n",
      "\n",
      "CUMETEL:\n",
      "I carcithy place, did the forling like grease in ratenforer;\n",
      "Which ot chatuse, be thy p \n",
      "\n",
      "[6m 8s (1800 90%) 1.6781]\n",
      "What feath wifiten,\n",
      "Thou kind Maner'd my king: I'll thou\n",
      "Reven's my streathence,\n",
      "By civery sow'd king' \n",
      "\n",
      "[6m 28s (1900 95%) 1.5265]\n",
      "What so srome the and any strand?\n",
      "\n",
      "BAPTISTA:\n",
      "Not bother hear are a common int.\n",
      "\n",
      "QUEEN MIRGANSIO:\n",
      "I say \n",
      "\n",
      "[6m 49s (2000 100%) 1.5479]\n",
      "Why, ruse the tort,\n",
      "And whese a to the vill bear not tell not the the borwading.\n",
      "\n",
      "JULIET:\n",
      "In be our no \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 2000\n",
    "print_every = 100\n",
    "plot_every = 10\n",
    "hidden_size = 100\n",
    "n_layers = 1\n",
    "lr = 0.005\n",
    "\n",
    "decoder = RNN(n_characters, hidden_size, n_characters, n_layers)\n",
    "decoder_optimizer = torch.optim.Adam(decoder.parameters(), lr=lr)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "start = time.time()\n",
    "all_losses = []\n",
    "loss_avg = 0\n",
    "\n",
    "for epoch in range(1, n_epochs + 1):\n",
    "    loss = train(*random_training_set())       \n",
    "    loss_avg += loss\n",
    "\n",
    "    if epoch % print_every == 0:\n",
    "        print('[%s (%d %d%%) %.4f]' % (time_since(start), epoch, epoch / n_epochs * 100, loss))\n",
    "        print(evaluate('Wh', 100), '\\n')\n",
    "\n",
    "    if epoch % plot_every == 0:\n",
    "        all_losses.append(loss_avg / plot_every)\n",
    "        loss_avg = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting the Training Losses\n",
    "\n",
    "Plotting the historical loss from all_losses shows the network learning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x11079f780>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAFkCAYAAACjCwibAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xl8VNX5x/HvYd8XUVlEq4JawAVZBAQFQUDBam1FjRtg\nfwqIRdGq1brUDXfUqohb645Wq+KGgKggIqJBtFpQFEQQZVMCsggh5/fHk+tMhklyJ5lkbpLP+/Wa\n12Tu3OXM3MncZ855zjnOey8AAIDiVMt0AQAAQMVA0AAAAEIhaAAAAKEQNAAAgFAIGgAAQCgEDQAA\nIBSCBgAAEApBAwAACIWgAQAAhELQAAAAQilV0OCc+6tzLs85N76Y9fo457Kdc1udc18654aW5rgA\nAKD8lThocM51lXSupE+KWW9vSa9KmiHpEEl3S3rYOde/pMcGAADlr0RBg3OugaQnJf2fpPXFrD5K\n0hLv/aXe+y+89/dJel7S2JIcGwAAZEZJaxruk/SK9/6tEOt2l/RmwrKpknqU8NgAACADaqS6gXPu\nVEkdJXUJuUkLSasSlq2S1Mg5V9t7/0uSYzSTNFDSN5K2plpGAACqsDqS9pY01Xu/Lp07TilocM61\nlnSXpKO999vTWZAEAyU9VYb7BwCgsjtd0tPp3GGqNQ2dJe0mab5zzuUvqy7pSOfc+ZJqe+99wjY/\nSGqesKy5pA3JahnyfSNJTz75pNq1a5diERFFY8eO1Z133pnpYiBNOJ+VC+ezclm4cKHOOOMMKf9a\nmk6pBg1vSjooYdmjkhZKujlJwCBJ70s6NmHZgPzlhdkqSe3atVOnTp1SLCKiqHHjxpzLSoTzWblw\nPiuttDfvpxQ0eO83Sfpf/DLn3CZJ67z3C/Mfj5O0h/c+GIthoqTRzrlbJP1TUj9JJ0kaVMqyAwCA\ncpSOESETaxdaStrz1ye9/0bSYElHS1og62r5J+99Yo8KAAAQYSn3nkjkve+b8Hh4knVmyfIhAABA\nBcXcEygXWVlZmS4C0ojzWblwPhEWQQPKBV9KlQvns3LhfCIsggYAABAKQQMAAAiFoAEAAIRC0AAA\nAEIhaAAAAKEQNAAAgFAIGgAAQCgEDQAAIBSCBgAAEApBAwAACIWgAQAAhELQAAAAQiFoAAAAoRA0\nAACAUAgaAABAKAQNAAAgFIIGAAAQCkEDAAAIhaABAACEQtAAAABCIWgAAAChEDQAAIBQCBoAAEAo\nBA0AACAUggYAABAKQQMAAAiFoAEAAIRC0AAAAEIhaAAAAKEQNAAAgFAIGgAAQCgEDQAAIBSCBgAA\nEApBAwAACIWgAQAAhELQAAAAQiFoAAAAoRA0AACAUAgaAABAKAQNAAAgFIIGAAAQCkEDAAAIhaAB\nAACEQtAAAABCIWgAAAChEDQAAIBQCBoAAEAoBA0AACAUggYAABBKSkGDc26kc+4T51xO/m2Oc+6Y\nItbv7ZzLS7jtcM7tXvqiAwCA8lQjxfWXS7pM0mJJTtIwSZOdcx299wsL2cZL2l/Sxl8XeL869aIC\nAIBMSilo8N6/lrDoSufcKEndJRUWNEjSGu/9hlQLBwAAoqPEOQ3OuWrOuVMl1ZP0flGrSlrgnFvp\nnJvmnDu8pMcEAACZk2rzhJxzB8qChDqyJocTvfeLCln9e0kjJH0kqbakcyS945w7zHu/oGRFBgAA\nmZBy0CBpkaRDJDWWdJKkx51zRyYLHLz3X0r6Mm7RXOdcG0ljJQ0t7kAXXjhWTZo0LrAsKytLWVlZ\nJSg2AACVy6RJkzRp0qQCy3JycsrseM57X7odODdd0lfe+1Eh179VUk/vfc8i1ukkKXv27Gz17Nmp\nVOUDAKAqmT9/vjp37ixJnb3389O573SM01BN1vQQVkdZs0Wxtm8vUXkAAEAZSKl5wjk3TtIUSd9K\naijpdEm9JQ3If/4mSa2890PzH18gaamkz2U5EOdIOkpS/zDHI2gAACA6Us1p2F3SY5JaSsqR9Kmk\nAd77t/KfbyFpz7j1a0m6Q1IrSZvz1+/nvZ8V5mDbtqVYOgAAUGZSHafh/4p5fnjC49sk3VaCckmS\ncnNLuiUAAEi3SM89QU0DAADRQdAAAABCiXTQQPMEAADREemggZoGAACig6ABAACEEumggXEaAACI\nDoIGAAAQSqSDBponAACIjkgHDdQ0AAAQHQQNAAAgFIIGAAAQSqSDBnIaAACIjkgHDdQ0AAAQHQQN\nAAAglEgHDTRPAAAQHZEOGqhpAAAgOggaAABAKJEOGmieAAAgOiIdNFDTAABAdBA0AACAUCIdNNA8\nAQBAdEQ6aKCmAQCA6CBoAAAAoUQ6aKB5AgCA6Ih00EBNAwAA0RHpoIGaBgAAoiPSQQM1DQAARAdB\nAwAACCXSQQPNEwAAREekg4bc3EyXAAAABCIdNFDTAABAdBA0AACAUCIdNOTmSt5nuhQAAECKeNAg\n0YMCAICoiHzQ8MsvmS4BAACQKkDQQF4DAADREPmggZoGAACiIfJBAzUNAABEQ+SDBmoaAACIBoIG\nAAAQSuSDBponAACIhsgHDdQ0AAAQDQQNAAAglMgHDTRPAAAQDZEPGqhpAAAgGiIfNFDTAABANEQ+\naKCmAQCAaCBoAAAAoUQ6aKheneYJAACiItJBQ82a1DQAABAVkQ4aatUiaAAAICpSChqccyOdc584\n53Lyb3Occ8cUs00f51y2c26rc+5L59zQsMerWZPmCQAAoiLVmoblki6T1ElSZ0lvSZrsnGuXbGXn\n3N6SXpU0Q9Ihku6W9LBzrn+Yg9E8AQBAdNRIZWXv/WsJi650zo2S1F3SwiSbjJK0xHt/af7jL5xz\nvSSNlTS9uOMRNAAAEB0lzmlwzlVzzp0qqZ6k9wtZrbukNxOWTZXUI8wxatWieQIAgKhIqaZBkpxz\nB8qChDqSNko60Xu/qJDVW0halbBslaRGzrna3vsi6xFIhAQAIDpSDhokLZLlJzSWdJKkx51zRxYR\nOJTYihVj9frrjXX88bFlWVlZysrKSvehAACocCZNmqRJkyYVWJaTk1Nmx3Pe+9LtwLnpkr7y3o9K\n8txMSdne+4vilg2TdKf3vmkR++wkKbtz52wdcEAnPfVUqYoIAECVMX/+fHXu3FmSOnvv56dz3+kY\np6GapNqFPPe+pH4Jywao8ByIAmieAAAgOlJqnnDOjZM0RdK3khpKOl1Sb1kgIOfcTZJaee+DsRgm\nShrtnLtF0j9lAcRJkgaFOV7NmtLWramUEAAAlJVUcxp2l/SYpJaSciR9KmmA9/6t/OdbSNozWNl7\n/41zbrCkOyWNkbRC0p+894k9KpJq2FAqw6YZAACQglTHafi/Yp4fnmTZLNlAUClr1EhaurQkWwIA\ngHSL9NwTjRpJP/6Y6VIAAAAp4kFD48bSTz9JpezgAQAA0iDSQUPDhtZ7YsuWTJcEAABEOmho3Nju\naaIAACDzIh00NGpk9wQNAABkXqSDhqCm4aefMlsOAAAQ8aChYUO7p6YBAIDMI2gAAAChRDpoqFHD\nmigIGgAAyLxIBw2StMsu5DQAABAFkQ8amjalpgEAgCiIfNCwyy4EDQAARAFBAwAACCXyQUPTpuQ0\nAAAQBZEPGqhpAAAgGggaAABAKBUiaNiwQcrNzXRJAACo2iIfNDRtavfr12e2HAAAVHWRDxp22cXu\naaIAACCzCBoAAEAokQ8aguYJul0CAJBZkQ8aqGkAACAaIh801K0r1a5N0AAAQKZFPmhwjrEaAACI\ngsgHDRJDSQMAEAUVImigpgEAgMwjaAAAAKEQNAAAgFAqRNBATgMAAJlXIYIGahoAAMi8ChU0eJ/p\nkgAAUHVViKBh111tauycnEyXBACAqqtCBA0tWtj9Dz9kthwAAFRlBA0AACAUggYAABBKhQgaGjaU\n6tWTvv8+0yUBAKDqqhBBg3NW20BNAwAAmVMhggaJoAEAgEyrMEFDy5YEDQAAZFKFCRpatCCnAQCA\nTKpQQQM1DQAAZE6FCRpatpTWrpW2b890SQAAqJoqTNDQooXNPbF6daZLAgBA1VShggaJJgoAADKl\nwgQNLVvaPUEDAACZUWGCht12s0Ge6EEBAEBmVJigoWZNmyKbmgYAADKjwgQNEgM8AQCQSRUqaIgf\nq2H6dOnddzNbHgAAqpIKFzR8/72UlyedfbZ0ww2ZLhEAAFVHhQoaguaJWbOkFSuk777LdIkAAKg6\nUgoanHOXO+fmOec2OOdWOededM7tX8w2vZ1zeQm3Hc653VMtbNA88eST9njFilT3AAAASirVmoYj\nJN0jqZukoyXVlDTNOVe3mO28pP0ktci/tfTepzy2Y4sW0ubN0jPPSG3bSjk50saNqe4FAACUREpB\ng/d+kPf+Ce/9Qu/9fyUNk7SXpM4hNl/jvV8d3EpQ1l8HeNq0SbrkEvubJgoAAMpHaXMamshqEX4s\nZj0naYFzbqVzbppz7vCSHCwYSrpzZ+noo+1vmigAACgfJQ4anHNO0l2SZnvv/1fEqt9LGiHpj5L+\nIGm5pHeccx1TPWarVlL16tKZZ9rfEkEDAADlpUYptp0gqb2knkWt5L3/UtKXcYvmOufaSBoraWhR\n244dO1aNGzcusOzaa7M0enSWatSwoaUJGgAAVdWkSZM0adKkAstycnLK7HjOe5/6Rs7dK+l3ko7w\n3n9bgu1vldTTe5804HDOdZKUnZ2drU6dOhW6n06dpMMOkyZOTLUEAABUTvPnz1fnzp0lqbP3fn46\n951yTUN+wHCCpN4lCRjydZQ1W5RK69bUNAAAUF5SChqccxMkZUk6XtIm51zz/KdyvPdb89cZJ2kP\n7/3Q/McXSFoq6XNJdSSdI+koSf1LW/jWraU5c0q7FwAAEEaqNQ0jZb0l3klYPlzS4/l/t5S0Z9xz\ntSTdIamVpM2SPpXUz3s/K9XCJqKmAQCA8pNS0OC9L7a3hfd+eMLj2yTdlmK5QmndWlq3TtqyRapb\n3PBSAACgVCrU3BOJWre2ewZ4AgCg7FWKoKGwJoq335YGDZJK0EEEAAAkqNBBwx572H1hQcMzz0hT\npkhr15ZfmQAAqKwqdNBQv77UtGksaPjgA2n79tjzs2fb/VdflX/ZAACobCp00CDFelC89ZbUvbv0\n6KO2fN066X/5g1sTNAAAUHqVImj45hvpwgvt8Qsv2H0wfkPt2tLixRkpGgAAlUpp5p6IhNatpUce\nkfLypLPOkiZNknJyrGmiVSupTRtqGgAASIcKX9Owxx4WMAwfLl1/veU0TJkivfee1KuXtN9+BA0A\nAKRDhQ8aDjlE2nVXadw4aa+9bBKrZ56RPvzQgoa2ba15gm6XAACUToUPGn7/e+n776UWLezxiSdK\nkydL27bFgob166Uff7Tnn37aaiIAAEBqKnzQIEk14jIzTjzR7hs0kA46yIIGyZoo8vIsYTIrywIN\nAAAQXqUIGuK1b295DD16WDARBA2LF0vZ2dKaNdLWrdKYMZktJwAAFU2F7z2RyDnp2WdjE1g1bCg1\nb241DUuWSI0bS/feK515pvTyy9Lxx2e2vAAAVBSVrqZBkg49VPrtb2OP27a1oGHKFKl/f+n006XB\ng622gQRJAADCqZRBQ6L99rMhpj/4QDr2WKuNuOACadkyaeHCTJcOAICKoUoEDUFNg/fSMcfYssMP\nt5yHmTMzWzYAACqKKhM0SDamQ6tW9nf9+lKXLgQNAACEVSWChv32s/tBgwou793bggbyGgAAKF6V\nCBratZOOPNISIOP17i398ENsQqu8vPIvGwAAFUWVCBrq1rUahQ4dCi7v2VOqVs2e27ZN6tZNuuyy\nzJQRAICoq3TjNKSiUSObq2LmTGnlSumjj2y46VtuyXTJAACIniodNEjWRPHPf0obN1qPijlzrKdF\nkDwJAABMlWieKErv3tJPP0kHHGAjRNaoIU2dmulSAQAQPVU+aOjTx26PPy41a2Z5DgQNAADsrMoH\nDQ0bSm+/bbkNkjRwoD3eti2z5QIAIGqqfNCQaOBA6eefLbcBAADEEDQk6NhR2m03migAAEhE0JCg\nWjVpwABp8mSrcQAAAIagIYmRI6Vvv5U6d5YWLMh0aQAAiAaChiR69ZLmz5fq1ZO6d7dBnwAAqOoI\nGgqx//7S++/bZFeXXsqkVgAAEDQUoU4dadw464I5fXqmSwMAQGYRNBTjuONswKe//pVZMAEAVRtB\nQzGck26+Wfr4Y2nixEyXBgCAzCFoCKFXL+nss6XRo6WTTpJ++CHTJQIAoPwRNIT08MPSpEnSrFk2\nG2Zubvr2vXq1tGhR+vYHAEBZIGgIyTnp1FNtpMilS6VXX7Xl3kvnnCM98UTJ933JJdKRRzLfBQAg\n2ggaUnTooVK3brH8hhkzrBbiiiuk7dtT319envTGG9KaNdIrr6S3rAAApBNBQwmMHGk1Dl9/Lf3t\nb1KbNtKKFdLzz6e+r08+seaJJk0s+AAAIKoIGkrg5JPtIn/aadK8eVbr0L+/NH58wUGgfvlFuv56\n6bbbpHXrku9r6lSpfn3pxhvt7+XLy+c1AACQKoKGEqhXTxo61AKG3r2lfv2kiy6y4aZnz7Z1Vq2S\n+vaVbrhBuuoqaY89pOHDdx6Seto0qU8f6ayzLHj417/K/eUAABAKQUMJjRpltQ033WRJkgMHSu3b\nW1LkccfZFNtffy3NnGlNF9deayNLdu0q9eghffedzaI5e7Zt26CBdMop0iOPSDt2ZPrVAQCwM4KG\nEjrgAOmnnywAkCxwuPVWqWVLqXp16fjjpQ8/tAmvdt1VuuwyCyImT7Yg4oQTpClTLHly4EDbx9Ch\nNrtmdnbmXhcAAIWpkekCVCaDB9utMEEwsddeNjT1sGHSb35jk2JJ1iujdm2bKOuww8qlyAAAhEZN\nQwZ07Cg9+aS0ebPVMjhny2vVkrp0kebMia27dauUk5N8P9u2WbIlAADlgaAhQ0480XIcbryx4PLD\nDy8YNIwZIw0YkHwf555rtRTLlpVdOQEACBA0ZFCfPpbvEO/wwy3nYflyq0l47jnrcbFxY8H11qyR\nnn7a5sE4+mjmwwAAlD2ChogJEivnzJHeektav95GjZw3r+B6jz1mzRrvvWfNHD17SmeeKV14ofXM\nAAAg3QgaIqZ5cxth8v33bYTJtm2ta+f778fW8V568EHpj3+0LpxvvmnDWy9bJj36qM1lAQBAuhE0\nRNDhh9v4Di++KA0ZYr0q4oOGmTOlxYstp0GS2rWzAGPWLBsP4rnnqG0AAKQfQUME9eghLVgg/fij\ndNJJ9nju3NgQ1Q89ZAmQvXvvvO3w4VLdutL996e/XPfeK51xhk0R/tNP6d8/ACDaUgoanHOXO+fm\nOec2OOdWOededM7tH2K7Ps65bOfcVufcl865oSUvcuV3+OF2v88+1uzQo4cFEF9+aU0Qzz0njRgR\n66oZr1EjCxwmTpS2bElfmXJzpb//3WbkPO006cADrTtoSXz3nY2e+fnn6SsfAKDspVrTcISkeyR1\nk3S0pJqSpjnn6ha2gXNub0mvSpoh6RBJd0t62DnXvwTlrRIOPFBq2tQmxnLOmiecsyaKG2+0HIcR\nIwrf/s9/tiDjoossgOjfv/AJs8KaM8f28dpr0rvvSitXWu1HSdx2m7RwoXU5BQBUHCmNCOm9HxT/\n2Dk3TNJqSZ0lzS5ks1GSlnjvL81//IVzrpeksZKmp1TaKqJ6dRuCumVLe9y4sdShg3WxfPtt6eab\nba6KwrRtK/3+91bbcPDBNjT1iBFWQ5GsdiLeo4/a/bBhBZe/9JKVp2tXe9ysmTRjhnUbTcXq1ZbE\nKUn//W9q2wIAMqu0OQ1NJHlJPxaxTndJbyYsmyqpRymPXam1aWOzaQZ69JCmT7dxHUaNKn77p56S\n1q6VPvnEciD+8x/p8ceL3mbzZmnsWOmKK6ybZ8B7CxpOOEGqVs1uRx1lQUOq7rrLtj/2WIIGAKho\nShw0OOecpLskzfbe/6+IVVtIWpWwbJWkRs652iU9flUTjN9w+eUFg4nC1K1rtQGSJVMOGyadf75N\nwX3EEZafkOjpp21ciO+/L9j08N//SkuXWu1FoF8/Gztiw4bwr2H9eum++yzo6dPH9hsfnKTb/PkW\nPAEA0qM0E1ZNkNReUs80lWUnY8eOVePGjQssy8rKUlZWVlkdMrJOPNEu3EE3y1TdfbcFA199JdWo\nId1wgwUQ++5rz3tvvSMGD7YRKF94IZaQ+dJLlmB51FGx/fXrZ1N4v/uubTNunDVfDB8eW2fBAmse\nqZYfmj72mCVPXnSR1YD8/LMldu6zT2ybvDybl+PUU20ujuJs3mz5G61bF1y+Y4e9vi+/tBEzmzcv\nfl8//mivswbTuAGoICZNmqRJkyYVWJZT2IRF6eC9T/km6V5JyyTtFWLdmZLGJywbJumnIrbpJMln\nZ2d7pN/mzd43b+79n/4UW/buu95L3k+d6v2IEd7vvbf3eXn23KGHen/qqQX3kZfnfevW3o8d6/2b\nb9q2u+3m/S+/2PPZ2bbsqadi2wwe7P3RR9vfK1bY85MnF9zv1Km2/Nlnw72WCy7wftdd7TXFe/xx\n20/Nmt6PG1f8fnbs8L5VK+/Hjw93XACIquzsbC9LHejkS3CNL+qWcvOEc+5eSSdIOsp7/22ITd6X\n1C9h2YD85ciAunWlSy+1X/7ffGPL7r1X2n9/+1X+xz/a8gULpCeekD7+2HpyxHPOahumTLEky3bt\nbD6M116L7U+SXn/d7rdvt0Gp+va1x61aWQ+RTz8tuN/nnrP72YWl1cbZsUN65hnL3Qi2k2zmz6uv\nlv7wB+n006UHHrB1i/LJJ9Yj5MMPiz8uAFRVqY7TMEHS6ZJOk7TJOdc8/1Ynbp1xzrnH4jabKGlf\n59wtzrkDnHPnSTpJ0vg0lB8lNGKEXbRHj7ZZNJ991mbUrFbN8g2aNpWuuUY65xxrcojPZwj06yct\nWmQTbL30kvWseOQR65o5aZK0++7S1KnW5JCdbc0RQdDgnDVdxCdD5ubaKJjVqoULGmbNklatkn7z\nm4KDWT34oPUYueEGy59YtszGlyjKO+/YPWNHoKTOO88+c0BllmpNw0hJjSS9I2ll3C3+d2hLSXsG\nD7z330gaLBvXYYGsq+WfvPeJPSpQjurXt9qG11+3X+qTJsV6ZdSsKR1/vPTKK1KnTnZBTtZVs18/\n6x561VVWS3H22VbzcOONFijcf7/tOzvbJt9q2FDq3Dm2/UEHFQwa3nnHAo4RI+yXf2KS5datts9g\nxs9nn5X23lsaP94SNz/+2C76V19t+Qzt2lkgE7yGogRjRixaZMFLuq1bZwFORfT995kuQcXw+uvS\nhAllm9wLZFy62zvScRM5DeVixw7vP/00lrsQ74MPvO/Xz/vvvy96H19/Hdt+/Xrv69SxXIIzzvB+\n2zbvGzXy/rrrbF/HHVdw2wce8L56de+3bLHHQS7FokW2jzfeKLj+tdfa8lNP9X77dstluPRS+3uP\nPbw/4QTLszj4YCtL4KGHvHfO+2++Sf4acnO9b9zY+z59bP9ffJF8vbw870eN8v7MM4t+T5IZNMj7\nNm12zr2IuoUL7b2bNy/TJYm2X37xvlo1+/y8916mS4OqLlI5Dag8qlWzX/vJahEOO8xmz2zRouh9\n7LtvbPvGja17p2TNHjVr2miUL71kU3gHTROBgw+2XIOFC+3+hRds+/33l3bbrWATxZIl0k03Sb16\nWR7DsGFWi3Hyydbb4dxzpcmTreZjyhQrS+DUUy2P48knk7+GBQuknBwrs1R4E8Udd1iNxZNPpvbr\n++ef7b38+mvp+uvDbxcFc+ZYz5o5czJdkrKRl2dNWaX17bexGoYXXij9/oCoImhAWl11lXW/7NbN\nHh9zjI2XsHXrzkFDhw52P3eu9M9/WiLlkCEWhPTqZd05AxdeaIHEG29YnsVTT9kAWJ062fMjR0qn\nnGLPt2pV8DgNGlhS5BNPxCb9ivfOOxZU/O530i67JA8apk6VLrvMmnCqV7dZRQOPP26BT2GmT5e2\nbbNA57bbih/Uat06m5sj/hiZkp1t9/PnZ7YcZeWxx6Tf/rb087QsWWL3AwZY0JDscwZUCumuukjH\nTTRPVBrLl1uVbbNm1hySaN997XnJ+x49Yk0dd9xhTR2//OL900/b888/b89t2uR9r17e3313+HIE\nXTk/+GDn5+K7gvbq5X1WVsHnp0zxvmFD74891poyBg3yvmdPe+6dd2y/DRp4/9JLyY999tnet2vn\n/datdt+9e/ImocDVV9s+d93V+zVrwr/GstCtm5XlwAMzW46ycvLJRTdJhXX//d7XqOH966/b/ubP\nT0/5gJKgeQIVVuvW0iGHWNJktSSftnvuse6Zn39uTRhBU8cRR1jtxJgxNh13VpbVFkg2Iua779pz\nYfXrZ4NPPfFEweW5ubavYA6NDh0K1jRMmGCDVx15pCVeVq9uzR3vvWdV0pdfbjORDhxoPUyuusq6\nlwby8qwb6nHHSbVrWxPL3Lk2yFYyGzdK//iHdRXdscMGwiqNtWutlqQkcnMtIbVdO6tJSeesqVHg\nfSwBtrRNFEuWWC+eo4+2CeVookClle4oJB03UdNQqSxf7v3atalts22b9/Xq2a+2P//ZfuGX1sUX\n26/39eu9v+su7wcMsOTE+OS1f/zD+1q1LLny5ZftuQsuKHj8nBzva9e25E7J+2nTrBblhhsssbNL\nF+//9z9bd948W2fmTHu8bp3fadCreLfeagNSLV/u/b/+ZeteeaX311zj/Z13pv6a//pXS2T8/PPY\nso0bY4NwFeXTT+34d9xReC1NYVautBqcn35Kvczl5b//jdVyPfJI6fb1xz9637+//X3WWd63b1/6\n8kXdF1/Y/0aYzxLKV1nWNGQ8QEhaKIIGeBvJ8Z57iq7KT8Unn9gnvn59u7gPHuz9RRfZBSM4xowZ\nts6iRd537ep9797J93XiibZe374Fy/fBB94fcIA1rdx5p13wmzSxICTQpo192SYKRur8v/+zx3l5\ndjGqUcOCHcn7BQsKrh+/32Q6d7btgiaXDRvs+H/+c9HbeW9Bi3PWRFKjhlXBF2bMGO9ffDH2+K67\n7Lhvvrk1ix3RAAAbbklEQVTzugsXWk+XlSuLL0NZuvtuCxB32cX7v/+9dPvq2NF6/3hvAaGU+aal\nspSbG2u6evnlTJcGiQgagDQ54wzvzznHuoom88MP9l8xapT/dVjtZP7zH7ugJvv1vWmTBQWSdcNL\nzJHIyrL8jXg//WQX0urVvV+8uOBzeXlW87LbbhbkBMt69PD+sMNiXVYTrV1rZezVy+4XLvT+3HOt\nXJ06Jd8m3vnnWwDkvfeHHGLvWzKrV9v+u3ePLQtqYR56qOC6ubm2XtAtN5NOOMGCwm7dLO+kpPLy\nrGvxLbfY4y+/9Em7DJe1lSvtsxXf3bis3H23nfPmzXceYh6ZR9AAlJO8PPvlKdmv9MJqOfLyvF+2\nrOh9zZhh83Yk/tq+806ridi2zR5/8oklhDZpUvSvtgsusC/p7dvtgiRZkHHWWcnL+e9/2zpffx0b\nv0Kyi2TNmpaYGUiWpNqjRyzgGT7c3o9knnjC/1rNv2SJNd/UqGGPr7ii4Lp33WUXmxEj7PlZswo+\nv3699889l7w88e6/3/ulS4tepyjB2BzXXuv9SSfFEmFLYu1aey3PPWeP8/Js39dfX/J9lsSYMb5c\nfvkvW2a1deed5/1NN3lft641eSE6SIQEyolzsa6gf/tb8jEsgvX22qvoffXta10V+yXMvNK1qyV5\nfvaZXWrPOsuSO+fPt26fhRk61EaVnDrVurV27WrdPR9/3GYxTTR9unUn3HdfS9j89FPrAnvnnZas\n+dlntt6339rsnvHdKnNzbfyKYATPTp2sq+i2bTsfZ8oUS5asW1f697+tfLm5Nnvp0qWx9ZYula64\nwsbDmDDBxgIZPTo2AueKFZYAO2SIjWtRmDVrrOvrww/Hlq1YYYmx8UmoRQnG5ujb185jaRIhg+6W\nwYyxzklduthssWH88kvqo5C+/badywcftMerVsX+DrrJlpUrr7RxUG66yZKCt2yxMVIk6X//i32u\nKpIVK+x/JZiLB4VjEmAgQffuNoT1CSeUzf4PPdR6YcybF+uh8NprBacIT6ZjRxuM65JLrDfDiy9a\nj43sbBsS/OSTC45R8eab1mtDkv70J2n9ehvqu3FjO/78+RYUTJkibdpk98G4F4sW2cUgPmjYts0u\nCh07xo6xY4eNjTFqlLR4sQ28ddBBduvUSfrii9i6d95pxx43znrS3HefBQ5du9q4HC++aOXae28b\nP2HAgOTvQzDQ1Mcfx5Y9/7z1xBk0yC6mxXnrLQvUDjvM3r9vv7UArrAgsSiJQYMUC+jCOPpoG+js\nvvuKXzcnxyaUmzHDzvWbb9p4JdOm2WBqHTumN2hYvNjO4eDB9t5s2WLn6dJLLdBs1Ejq2VN6+mnr\nPXLMMdKuu1rvoIo0xfxzz9nrnDfPPn8oHDUNQIKgW2SyLqLpUK+eXVTnzZMeekjac0/rslkc56xW\nYuFCG/zp+ONt+TXXSHXqFLzofP21/bLv398e165tv/JbtLAagfbtYxeX6dPtPn4wreC5Qw+1+0MO\nseMnXpA+/FD68Ue7WJ96qv2C/89/LFhJrGlYsEDq3dvmIJHs1/h//iMdeKAFLPvua+/7yJHWZTEn\nJ/n78N57dh9fMzJvnt3/+9/Jt9m6teDj2bOlww+XatWymoatW617akksWWITvDVpElvWpYvNmrpy\nZdHb5uZa2R95RFq9uvhj3X23vf6XXrKJ2Pr1s2BxwgSrtenXL3wNR1Hefts+o/vvb7VfwYRuU6fa\nCKdDhsTWzcqy5cccI+23n5XrlVdKX4bCrFljQVk6g6MXX7T7IAAsTm6utHlz+o5foaS7vSMdN5HT\ngErunHOsF0ODBtadMqyVK609+dlnCy6/8ELLxdi0yR7ff7/lO+TkJN/P0KHWNTQ31/umTW3wrQYN\nYr0xzjrL+w4dCm5z2GGWd/GPf8TWu+oq2z431xIyGza0dvU5c7x/7DH7e9Mma+dv2tS6pRZnxQpL\nIE1MogwcfrjlDEixuVHatrVusE2a7NwF8IUXrFzx7e4HHBDrwfLRR7avxK+bHTuKn3vFe+vtkpjv\nsWyZ7XPy5J33GZ+oGMyzIhXfg2PjRjvH558fW7ZuneXD1K1rCakvvGD7+u67wvfz1luW8FuUQYO8\n/+1vrUfMgQd6/7vf2fKsLO8POqjguqtWWS+Uo46yc92zp83jUpwdO6xLcHG5QYn+8hf/6xw0JTV8\nuHUl9t7KH8wbEvRcKs6NN1pycFSRCAlUMg89ZP99zqX+pZls0qslS+yL7/777Yv7iCNio1YmE4xH\n8d57Vo5bbrH7jz6yi26TJtZdNN6aNdb7wjkLKF580QKP+C/voUO93313CyJmzbJ9fv65BQLJLqKF\nGTgwefm3bLFyjx1r+3v99djYF5deavevvVZwm8GDY6/Newt4atb0/t577fGqVfZ8fJdR7y2wq1+/\n+N4Ifft6P2RIwWV5edbb5aqrCi4fM8YmVwsSV59/3o49ZIitX1hPGO+9v/12SzBN/LwsX+79++/b\n30GwUlgy5Hff2T4uv7zo17TXXvZ+em9dkp2zhN0GDZIneH71VSyx9tlnrQyffFL0MYKA6brril4v\n3sqVlkTctq2dwzBBXaLVq+1/pV4929/DD9vjPn3sXIYxdGjBoDVqSIQEKpmuXe3+mGOKT6hMVLfu\nzsv22Uc68UTp1ltjVbd//Wvh+whyFO6+2+bmGD3amjhmzbL2/vXrY5OPBXbdVXrgAatOb9HCjvfR\nR9Kxx8bWueMOq8quXj2Wo7F0aSw57sADw73GYcOsGn7x4oLLP/rIyn3GGZYfMX9+rDr+7LMtmS2+\niWLdutiImIsW2f2331rC5H772ePddrPmm/hkyMcft6ajTZuKTsqUrEo7Pp9Bsqacrl2t+SYwc6aN\n9vndd7HX9fnnUrNmluexdq3NqZLM1q323p555s6fl9atLQ9Hsqau3XYrvOr+gQesav3TTwt/PRs2\n2HsRnKvTTrNzf/LJOzdNBNq0sfdQss9F69b2WosSnLeg6SOMcePs8z99uuVMPPJI+G0Dr71mdTs1\na0rXXWdNPT172nw5X38dbh9BU1LQVJaqn36yZtBNm0q2fSYRNAAZ0KGD9RS49NL07XPsWLtAV6tm\nF6sgCTKZjh1tveeftyG069e3L81337U8gzZtLDkvmS5d7EL61luWAPn738eea9bMelJIlqhXq1Ys\naKhfP3yS2Qkn2ORhF19syZaB996zIOfggy3f4uOP7bU2bmxBwJAhdhH45Rdb//nn7QLRuHEsKTO4\nYAdBQ9ATJggaPvvM8iqGDbPXMmVK4eXcvNm2a9Mm+fv00Ud2/M2bLRk1SCz94AO7//xz+yy0bWs5\nKrffnrwHyJNPWg+JogLB4LV07pw8r2HbNgsaatUqetK0YBj1IGioU8fO8xdf2Pt+wAFFl6FmTem8\n8ywAWr8+tnz9ertYBoIyzpkTO19FWbbMyn/JJfY5Ou00aeLE1HueTJ5sQdaVV1pgOG2aBTpt2kjL\nlyfvIZQoCBriZ+KdO9f2tXJl8ROW3Xqr5RidfLKVf+tW+8z95S+pvZZ4K1bY52fjxpLvI5R0V12k\n4yaaJ4ASmTMnltdQnPbtrYo1mPjryitt5Mldd/X+ssvSU5799rMBqYYNs5yIVLz2mlUbB9Xk3nt/\n3HGxMRUuusja8084wQaT8j42NPRjj9njI4+04cKPOirWhHDPPdbEET80eL9+seePOMLa8TdtsqHH\nW7VKPg7GkiU2SFbt2t5/9tnOz7/yipXlkkuszHXq2MBP++/v/ejRtk779jbegffef/yxvd6grT3e\nwIHhx5L429+8b9Fi5+XBSJWXX273heW7PPiglSO+GeyHH6z848aFK8OKFdakEZwH7+0cDBgQe9yz\np+VNxA+zXpS//MVyOoLclOzs5M1KRdm82Zolbr7ZmoJat/a/ji8SjAb75ZfF72fPPW3drl3t8YYN\nsWHvJcsJKUxOjuXk9OtnTUVnnBEbXXO33cK/lkRXX23NRxs2kNMAoAyccYZ9AwTzZEybFvvS+/DD\n9BxjwAAbcrtz55KNujh+vJXn9tvtC3+XXWKJo8GgUk2bxtro8/Ls4l+zpuV3OOf9o496P3KkDW7l\nveUV/Pa3BY8zbJiNVPn557bPINH0zTf9TsN3e28jhTZpYkFLYTNarl1r+R0tWtggX08+acvPOsve\nj19+sYvGfffFtvnzn+2Lf/ny2LJNmywwGT8+3HtWWDJk9+52ofr4Y19gvpVEY8ZYYJNoyZLU5pk4\n4ojYxXPxYjtmjRp20dy+3S6yt95q5+/aa4ve1/btNrBZ4vDnPXtaYmbYcgWBXPCZnzzZ8nS8t8HC\nJJvVtih5eXY+OnSwZOOff47lfcyebfk2iYOnxQvml1mxwj6bkvctW8aCueKSVJPZts32EQxlTtAA\nIO2eecZ6IgS/ojdutC/B3/wmffN9jBhhX+p164a/6MXLy7MLhRTrmTF9uj0XXOAlu1AGtm3z/g9/\nsOW1a9tF6q677Jfyjh02xflxxxU8ztVXW43CRRdZTUvwhb91qyVD3nRTbN0HHrD3adCgkk3Idd99\ndvH88EMr4zvvxJ5bv94ujvGJla++austXBhu/8mSIWfOtGUvvWSvqXp17ydOTL59374250lp3XOP\nvc4ff7Taj7p1rQzPPx+rEXrnHaspOuqogtv+/LPV8nz1lT0O3oPES8KCBXaMxKTdwpxzjtV+Jft8\n5+buHMQls369lSWYwn7GDAuQggnLgknq4oeY37LF9r91q13c4wPoGTMsofKLLwp+vlMRBIpBcEvQ\nAKBcHHdceoc/vvlm+wVW0i/DwBdfWJPJgAGx5pfc3NiFKP6XufcWOJx9tl14vLdfj5L9mmzb1n4N\nxnv4YStns2axbQLHH2/NHFu2xOYUOf/84icLK0zQxTPYV+LEVkEzwowZ9nj0aO/33jt8IJeXZxem\n3/3OgqRNm+xC2aNHrEmmXbtYE0mi3XdPrRtwYVautPf0wQetx8jIkXbc4cNjk6Hl5MSGVQ96juTl\neX/yyf7XIc9zc22o74MPTv4eXHedBUHF1Y7t2GG1PonnN17btrH5XQoTzC0yY4bVNp11lj1++ml7\nfutWa/66557Ycffe24LPAw+MzQOTKPg8lyS47t+/4NwvBA0AKqSg+11ZdU/r3t0ukMVdUIOq55df\ntgvMhAkFn49vmgmqrgMTJ9o27dsXvBiU1LZtdpFs1sxqFRLl5Vn+R7du9ve++9oEaql4+WXLS7j4\nYguQ6tSxLo6Bk0+2QChR0P00mEejtHr3tiBEsl/gf/mLXbhHjYo1EQXNJUGNy7hx/tfciyCnpVat\nwi+m27ZZbsnee9t8K7m5Vrvx/PMFc02CmoG5cwsv78CBVvNRlNmz/a9diYPuvI0aFcwB6drV+zPP\ntL+DIHH0aKsBS5yPJV6XLhZUpSJo+nn00dgyggYAFVJQVdusWfqaPOI98EC4fv47dtiFc/Ron7TW\nIxgzoFevnbddtsyCho4drVo9HXr2tOMVNi5AkEtx222xYCdV99wTC4Ruv73gc9ddZ/khiefkrbd8\nSk0hxZkwwfZ34IF2rLff9r/moQSznO7YYY/79LFaHedi41tccon/NRdi1arCj7N4cWxm1ZYt7XxJ\nljtwxx3W5CBZzVdRzjsvNnjVxo3J34egKWDNGmu2kmJ5EYHRo2NB0fXXW1ARTFBXlKImhivMxRfb\n+xcftBA0AKiQ1qyxb5kwIwSWtYMPtnwNyftvvin43ObNltGeONJm4IsvUksCLM5FF1k5xowpfJ2+\nfe2Xdq1a1sZfElddZYmo8T1FvLceB8mSJYNBv0ra9JLohx/swn3XXfZ427bYaJ5Brx3vremieXOr\nZr/xxtgsp1u2WMBxyinhjvfBB5YDM3Gi5UNcfHEscLrgguID19tvt2aEvDzL66hTZ+dAceJEOy87\ndlhQ7FzB/AXvY6Ohrl9veUN/+EO48o8fb8dMPF+FWbPGEmcTezsRNACokPLyLIExfujjTAnayWvX\nTj71dlGjMaZb0GzzwAOFrzN3rq1Tmmm7C/PVV7bvN96wv0eNssDo3HPTPzzy4sUFL4JDhvgie28k\n2rq18J4IYUybZsOXFzfduvexYCqYer5hQwta4n/FX3ddwWallSt33s/Chf7XZp5q1SyvI4yghikI\nUnv0sB4f8ce65BLrVum9NeHUr2+jXMZjREgAFZJzNgDP6NGZLklsUKI2bZJPRlanTvmV5cgjbfTG\nI44ofJ1u3aTrry/dgD+F2WcfG2xr+nSb1Oz++21SshdfDD9qZ1ht29oIoYEhQ2xyr/jZUotSu3Zs\ntMmS6N/fprkPMwFdMEjXyJH298yZNhhY/CBsq1dLu+8ee9yy5c772X9/mwH0ppukvLxwM69KNkmY\nZINvPfOM9P77NrBX4JFHpNtus5lOv//eZnY9/3wbBbS8EDQAKFOnnWbDO2daUIZgJMhMatHCRpIM\nRs8szJVXhpsBNVXVqtlIlHfcEZvyfNQoG8q6S5f0Hy/ekCE2amK9emV7nJIIhj7/5hubPfbQQ+0i\nfe+99h5JOwcNyVSrZu/j/Pn2Pu+5Z7jj77673T791M5NtWoW2AWjok6ZYvubOdOGgvfeRk0tTwQN\nAKqEKAUNUdCtmw3VPW2aBS/jx9vF8rzzyv7YyeZPiYIGDaTmza1W6rTTbNnQoXYfTMUeJmiQpMMO\ns/uwtQyBgw6yGoVPP5Wuvtqmns/Otvu5c6ULLogNK17etQySVKN8DwcAmbH//jYvQvv2mS5JNNx8\ns03Y1KRJbFmqk6dVRjfdZEFD0KTSqJHVFARzcqxebU05xSlp0HDwwdKMGXaMK66wGoepU20Ol7w8\nmyCudWsLLtq2TW3f6UDQAKBKaNDAfrEVN+FSVVGvXjSbCDJt+PCdl3XoEAsaVq0KV9MweLD0z39K\nffumdvwgr+Hiiy3I7dfPaoO+/tqea93ans9Ukx/NEwCqjIMOslkegVS0b29BQ26uTbceJmioVcsC\nkDAJmPGOP95yWU45xR4PHGgJka++WnAa+kwhaAAAoAgdOljzQDB9epigoaSaNbNeM0FwO3CgJUKu\nW0fQAABA5HXoYD0VZs2yx2UZNCTaZx9L3m3YUOrZs/yOWxhyGgAAKEKQPPv223bfvHn5Hn/MGEvA\nrFmzfI+bDEEDAABFaNjQelAEQUN51jRI1rUyKmieAACgGB06SMuXW4+T+vUzXZrMIWgAAKAYHTrY\nfXnXMkQNQQMAAMUgaDAEDQAAFCNIhiRoAAAARSJoMAQNAAAUo2FDm9grmD67qqLLJQAAIbz3XtXu\nOSERNAAAEErTppkuQebRPAEAAEIhaAAAAKEQNAAAgFAIGgAAQCgEDQAAIBSCBgAAEApBAwAACIWg\nAQAAhELQAAAAQiFoQLmYNGlSpouANOJ8Vi6cT4SVctDgnDvCOfeyc+4751yec+74Ytbvnb9e/G2H\nc66KzxVWtfClVLlwPisXzifCKklNQ31JCySdJ8mH3MZL2k9Si/xbS+/96hIcGwAAZEjKE1Z579+Q\n9IYkOedcCpuu8d5vSPV4AAAgGsorp8FJWuCcW+mcm+acO7ycjgsAANKkPKbG/l7SCEkfSaot6RxJ\n7zjnDvPeLyhkmzqStHDhwnIoHspDTk6O5s+fn+liIE04n5UL57Nyibt21kn3vp33YdMSkmzsXJ6k\n33vvX05xu3ckLfPeDy3k+dMkPVXiggEAgNO990+nc4flUdOQzDxJPYt4fqqk0yV9I2lreRQIAIBK\noo6kvWXX0rTKVNDQUdZskZT3fp2ktEZHAABUIXPKYqcpBw3OufqS2sqSGyVpX+fcIZJ+9N4vd87d\nJKlV0PTgnLtA0lJJn8uin3MkHSWpfxrKDwAAyklJahq6SHpbNvaCl3RH/vLHJJ0tG4dhz7j1a+Wv\n00rSZkmfSurnvZ9VwjIDAIAMKFUiJAAAqDqYewIAAIRC0AAAAEKJXNDgnBvtnFvqnNvinJvrnOua\n6TKheM65a5JMTPa/hHWuyx8VdLNzbrpzrm2myouCwkxEV9z5c87Vds7d55xb65zb6Jx7nonpMqe4\nc+qc+1eS/9nXE9bhnEaAc+5y59w859wG59wq59yLzrn9k6xX5v+jkQoanHOnyJImr5F0qKRPJE11\nzu2a0YIhrM8kNVdsYrJewRPOucsknS/pXEmHSdokO7e1MlBO7KzIiehCnr+7JA2W9EdJR8qSn/9T\ntsVGEcJMLjhFBf9nsxKe55xGwxGS7pHUTdLRkmpKmuacqxusUG7/o977yNwkzZV0d9xjJ2mFpEsz\nXTZuxZ67ayTNL+L5lZLGxj1uJGmLpJMzXXZuO52rPEnHp3L+8h//IunEuHUOyN/XYZl+TVX9Vsg5\n/ZekF4rYhnMa0ZukXfPPQ6+4ZeXyPxqZmgbnXE1JnSXNCJZ5e1VvSuqRqXIhJfvlV4V+7Zx70jm3\npyQ55/aR/YqJP7cbJH0gzm3khTx/XWRduOPX+ULSt+IcR1mf/OruRc65Cc65XeKe6yzOaVQ1kdUe\n/SiV7/9oZIIGWeRUXdKqhOWrZG8Gom2upGGSBkoaKWkfSbPyBwNrIfuAc24rpjDnr7mkbflfVIWt\ng2iZIuksSX0lXSqpt6TXnXPBwH0txDmNnPzzc5ek2d77IG+s3P5HMzWMNCoZ7338GOefOefmSVom\n6WRJizJTKgCF8d7/O+7h5865/0r6WlIf2QB+iKYJktqr6PmbykyUahrWStohi4biNZf0Q/kXB6Xh\nvc+R9KVsyPEfZPkpnNuKKcz5+0FSLedcoyLWQYR575fKvoeDjHvOacQ45+6VNEhSH+99/PxN5fY/\nGpmgwXu/XVK2pH7BsvxqmH4qo4k3UHaccw1kXz4r87+MflDBc9tIlgnMuY24kOcvW1JuwjoHSNpL\n0vvlVliUmHOutaRmik0myDmNkPyA4QRJR3nvv41/rjz/R6PWPDFe0qPOuWzZ9NljJdWT9GgmC4Xi\nOeduk/SKrEliD0nXStou6Zn8Ve6SdKVz7ivZlOfXy3rGTC73wmInxU1Ep2LOn/d+g3PuEUnjnXM/\nSdoo6R+S3vPezyvXFwNJRZ/T/Ns1su52P+Svd4usdnCqxDmNEufcBFl32OMlbXLOBTUKOd77rfl/\nl8//aKa7jiTpSnJe/gveIot+umS6TNxCnbdJ+R/QLbJs3Kcl7ZOwzt9l3YI2y76Y2ma63Nx+PTe9\nZV2vdiTc/hn2/EmqLetLvjb/C+k5Sbtn+rVV1VtR51Q24/AbsoBhq6Qlku6XtBvnNHq3Qs7jDkln\nJaxX5v+jTFgFAABCiUxOAwAAiDaCBgAAEApBAwAACIWgAQAAhELQAAAAQiFoAAAAoRA0AACAUAga\nAABAKAQNAAAgFIIGAAAQCkEDAAAI5f8BEXt19l83XNQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x106e39da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(all_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating at different \"temperatures\"\n",
    "\n",
    "In the `evaluate` function above, every time a prediction is made the outputs are divided by the \"temperature\" argument passed. Using a higher number makes all actions more equally likely, and thus gives us \"more random\" outputs. Using a lower value (less than 1) makes high probabilities contribute more. As we turn the temperature towards zero we are choosing only the most likely outputs.\n",
    "\n",
    "We can see the effects of this by adjusting the `temperature` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thoo head strant me reporce\n",
      "O and hears of thou provand of treech.\n",
      "\n",
      "LUCI death in that to tellon is head thing come thou that to not him with your firsure but,\n",
      "They here thyse of yet in thou thy meat to\n"
     ]
    }
   ],
   "source": [
    "print(evaluate('Th', 200, temperature=0.8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lower temperatures are less varied, choosing only the more probable outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This commanderence the forself to the the to the the the to the to the the formands\n",
      "What to the strange the boy the the have the the to the to to the formands\n",
      "That the the the the the the sorn the to th\n"
     ]
    }
   ],
   "source": [
    "print(evaluate('Th', 200, temperature=0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Higher temperatures more varied, choosing less probable outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "That,\n",
      "henct wto Haste's, norsee'd stave brYiry's is dsem.\n",
      "Hell hurss Heamous halloR:\n",
      "Tht a readerty the!\n",
      "\n",
      "KuWhrate.\n",
      "\n",
      "VLOMAY, mere's no, toojecur' kong.\n",
      "\n",
      "DUKE VIx whJos ivistomzliben\n",
      "The vrieglad bloot, \n"
     ]
    }
   ],
   "source": [
    "print(evaluate('Th', 200, temperature=1.4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Exercises\n",
    "\n",
    "* Train with your own dataset, e.g.\n",
    "    * Text from another author\n",
    "    * Blog posts\n",
    "    * Code\n",
    "* Increase number of layers and network size to get better results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next**: [Generating Names with a Conditional Character-Level RNN](https://github.com/spro/practical-pytorch/blob/master/conditional-char-rnn/conditional-char-rnn.ipynb)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}