Multi_Agent_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi Agent Collaboration and Competition Project\n",
    "\n",
    "---\n",
    "\n",
    "I wanted to do this whole thing on my own computer.  However, I'm still struggling to get the Unity ML-Agents environments to work properly.  Because I'm on a time schedule, I need to bang out the project.  So I'm using the Udacity Workspace to do this project.  It sucks, but I'm working to resolve all of this.  \n",
    "\n",
    "I'm using my last project, the DDPG_Project, as the foundation of this project and building from there.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We need to install a bunch of packages.  Run the next two code cells to do the install.  This line will take a few minutes to run and will only work in the Udacity Workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mtensorflow 1.7.1 has requirement numpy>=1.13.3, but you'll have numpy 1.12.1 which is incompatible.\u001b[0m\r\n",
      "\u001b[31mipython 6.5.0 has requirement prompt-toolkit<2.0.0,>=1.0.15, but you'll have prompt-toolkit 2.0.9 which is incompatible.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip -q install ./python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import torch\n",
    "import workspace_utils\n",
    "from ma_ddpg_agent import Agent\n",
    "from collections import deque\n",
    "from unityagents import UnityEnvironment\n",
    "from workspace_utils import active_session\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to install the environment.  Because I am in the Udacity Workspace, the environment already exists.  All I need to do it tap into it via the proper file path.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='/data/Tennis_Linux_NoVis/Tennis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, two agents control rackets to bounce a ball over a net. If an agent hits the ball over the net, it receives a reward of +0.1. If an agent lets a ball hit the ground or hits the ball out of bounds, it receives a reward of -0.01. Thus, the goal of each agent is to keep the ball in play.\n",
    "\n",
    "The observation space consists of 8 variables corresponding to the position and velocity of the ball and racket. Each agent receives its own, local observation. Two continuous actions are available, corresponding to movement toward (or away from) the net, and jumping.\n",
    "\n",
    "The task is episodic, and in order to solve the environment, your agents must get an average score of +0.5 (over 100 consecutive episodes, after taking the maximum over both agents). Specifically,\n",
    "\n",
    "- After each episode, we add up the rewards that each agent received (without discounting), to get a score for each agent. This yields 2 (potentially different) scores. We then take the maximum of these 2 scores.\n",
    "\n",
    "- This yields a single score for each episode.\n",
    "\n",
    "The environment is considered solved, when the average (over 100 episodes) of those scores is at least +0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Size of each action: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.          0.\n",
      "  0.          0.         -6.65278625 -1.5        -0.          0.\n",
      "  6.83172083  6.         -0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents \n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you can see how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Because I'm in the Udacity Workspace, you will not be able to watch the Agents while they are training.  Suck ... I know.\n",
    "\n",
    "This code takes random Actions, so it's performance will be less than stellar.  However, it should prove that everything is set up properly and working as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: -0.004999999888241291\n",
      "Total score (averaged over agents) this episode: 0.04500000085681677\n",
      "Total score (averaged over agents) this episode: -0.004999999888241291\n",
      "Total score (averaged over agents) this episode: 0.09500000160187483\n",
      "Total score (averaged over agents) this episode: -0.004999999888241291\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):                                         # play game for 5 episodes\n",
    "    env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "    states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "    scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "    while True:\n",
    "        actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "        actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "        env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "        next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "        rewards = env_info.rewards                         # get reward (for each agent)\n",
    "        dones = env_info.local_done                        # see if episode finished\n",
    "        scores += env_info.rewards                         # update the score (for each agent)\n",
    "        states = next_states                               # roll over states to next time step\n",
    "        if np.any(dones):                                  # exit loop if episode finished\n",
    "            break\n",
    "    print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Now to train the Agent\n",
    "\n",
    "A few **important notes**:\n",
    "- When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```\n",
    "- You can see the list of files in the workspace by clicking on **_Jupyter_** in the top left corner of the notebook.\n",
    "- **_After training the agents_**, you can download the saved model weights to watch the agents on your own machine! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But first, we get the actions for each of the two Agents and combine them into a single array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_actions(states, add_noise):\n",
    "    action_0 = agent_0.act(states, add_noise)\n",
    "    action_1 = agent_1.act(states, add_noise)\n",
    "    action_jackson = np.concatenate((action_0, action_1), axis=0).flatten()\n",
    "    \n",
    "    return action_jackson"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with the Deep Deterministic Policy Gradient (DDPG) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ma_ddpg(n_episodes=5000, train_mode=True):\n",
    "    \"\"\"Multi-Agent Deep Deterministic Policy Gradient (MADDPG)\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes            : maximum number of training episodes\n",
    "        train_mode            : if 'True' set environment to training mode\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    scored = []\n",
    "    mov_avg = []                                                    # list containing the moving averages\n",
    "    mean_scores = []\n",
    "    mean_score = 0.0\n",
    "    solved = False\n",
    "    scores_window = deque(maxlen=100)                              # last 100 scores\n",
    "\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=train_mode)[brain_name]    # reset the environment\n",
    "        states = np.reshape(env_info.vector_observations, (1,48))  # get the current state (for each agent) and combine\n",
    "        scores = np.zeros(num_agents)                              # initialize the score (for each agent)\n",
    "        agent_0.reset()\n",
    "        agent_1.reset()\n",
    "        \n",
    "        while True:                                                # run the entire match, not just 1000 timesteps\n",
    "            actions = get_actions(states, add_noise=True)          # select a new action (for each agent)\n",
    "            env_info = env.step(actions)[brain_name]               # send all actions to the environment\n",
    "            next_states = np.reshape(env_info.vector_observations, (1, 48)) # get next new state (for each agent)\n",
    "            rewards = env_info.rewards                             # get new reward (for each agent)\n",
    "            dones = env_info.local_done                             # see if episode finished\n",
    "            \n",
    "            agent_0.step(states, actions, rewards[0], next_states, dones, 0) # agent 1 learns here\n",
    "            agent_1.step(states, actions, rewards[1], next_states, dones, 1) # agent 2 learns here\n",
    "            \n",
    "            scores += np.max(rewards)                              # update the new score (for each agent)\n",
    "            states = next_states                                   # roll over states to next time step\n",
    "            if np.any(dones):                                       # exit loop if episode finished\n",
    "                break\n",
    "\n",
    "        scores_window.append(scores)                            # save the most recent new score\n",
    "        average = np.mean(scores_window)                        # calculate the average\n",
    "        if solved==False:\n",
    "            scored.append(scores)                               # save the most recent new score to the total\n",
    "            mov_avg.append(average)                             # save the most recent moving average\n",
    "                \n",
    "        print('\\rEpisode {}\\tAverage Score: {:.3f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "            \n",
    "        if i_episode % 10 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.3f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if (np.mean(scores_window) >= 0.5) & (solved==False):\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.3f}'\n",
    "                    .format(i_episode, np.mean(scores_window)))\n",
    "            solved = True\n",
    "        \n",
    "        if (np.mean(scores_window) >= 0.5) & (solved==True):\n",
    "            print('\\nEnvironment solved with BUFFER in {:d} episodes!\\tAverage Score: {:.2f}'\n",
    "                    .format(i_episode, np.mean(scores_window)))  # to ensure that my final plot stays > 30\n",
    "        \n",
    "            torch.save(agent_0.actor_local.state_dict(), 'checkpoint_actor_0.pth')   # save the actor weights\n",
    "            torch.save(agent_0.critic_local.state_dict(), 'checkpoint_critic_0.pth') # save the critic weights\n",
    "            torch.save(agent_1.actor_local.state_dict(), 'checkpoint_actor_1.pth')   # save the actor weights\n",
    "            torch.save(agent_1.critic_local.state_dict(), 'checkpoint_critic_1.pth') # save the critic weights\n",
    "            \n",
    "            break\n",
    "        \n",
    "    return scored, mov_avg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we initialize the Agents and run the ma_ddpg function and see how well we learn ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 10\tAverage Score: 0.000\n",
      "Episode 20\tAverage Score: 0.000\n",
      "Episode 30\tAverage Score: 0.003\n",
      "Episode 40\tAverage Score: 0.010\n",
      "Episode 50\tAverage Score: 0.008\n",
      "Episode 60\tAverage Score: 0.008\n",
      "Episode 70\tAverage Score: 0.010\n",
      "Episode 80\tAverage Score: 0.010\n",
      "Episode 90\tAverage Score: 0.019\n",
      "Episode 100\tAverage Score: 0.019\n",
      "Episode 110\tAverage Score: 0.023\n",
      "Episode 120\tAverage Score: 0.037\n",
      "Episode 130\tAverage Score: 0.040\n",
      "Episode 140\tAverage Score: 0.044\n",
      "Episode 150\tAverage Score: 0.048\n",
      "Episode 160\tAverage Score: 0.051\n",
      "Episode 170\tAverage Score: 0.057\n",
      "Episode 180\tAverage Score: 0.062\n",
      "Episode 190\tAverage Score: 0.061\n",
      "Episode 200\tAverage Score: 0.068\n",
      "Episode 210\tAverage Score: 0.069\n",
      "Episode 220\tAverage Score: 0.058\n",
      "Episode 230\tAverage Score: 0.064\n",
      "Episode 240\tAverage Score: 0.062\n",
      "Episode 250\tAverage Score: 0.062\n",
      "Episode 260\tAverage Score: 0.072\n",
      "Episode 270\tAverage Score: 0.079\n",
      "Episode 280\tAverage Score: 0.082\n",
      "Episode 290\tAverage Score: 0.082\n",
      "Episode 300\tAverage Score: 0.086\n",
      "Episode 310\tAverage Score: 0.101\n",
      "Episode 320\tAverage Score: 0.105\n",
      "Episode 330\tAverage Score: 0.103\n",
      "Episode 340\tAverage Score: 0.110\n",
      "Episode 350\tAverage Score: 0.117\n",
      "Episode 360\tAverage Score: 0.110\n",
      "Episode 370\tAverage Score: 0.111\n",
      "Episode 380\tAverage Score: 0.112\n",
      "Episode 390\tAverage Score: 0.116\n",
      "Episode 400\tAverage Score: 0.117\n",
      "Episode 410\tAverage Score: 0.113\n",
      "Episode 420\tAverage Score: 0.115\n",
      "Episode 430\tAverage Score: 0.125\n",
      "Episode 440\tAverage Score: 0.126\n",
      "Episode 450\tAverage Score: 0.127\n",
      "Episode 460\tAverage Score: 0.132\n",
      "Episode 470\tAverage Score: 0.127\n",
      "Episode 480\tAverage Score: 0.129\n",
      "Episode 490\tAverage Score: 0.129\n",
      "Episode 500\tAverage Score: 0.127\n",
      "Episode 510\tAverage Score: 0.118\n",
      "Episode 520\tAverage Score: 0.112\n",
      "Episode 530\tAverage Score: 0.107\n",
      "Episode 540\tAverage Score: 0.099\n",
      "Episode 550\tAverage Score: 0.094\n",
      "Episode 560\tAverage Score: 0.084\n",
      "Episode 570\tAverage Score: 0.079\n",
      "Episode 580\tAverage Score: 0.074\n",
      "Episode 590\tAverage Score: 0.072\n",
      "Episode 600\tAverage Score: 0.068\n",
      "Episode 610\tAverage Score: 0.070\n",
      "Episode 620\tAverage Score: 0.076\n",
      "Episode 630\tAverage Score: 0.072\n",
      "Episode 640\tAverage Score: 0.079\n",
      "Episode 650\tAverage Score: 0.080\n",
      "Episode 660\tAverage Score: 0.085\n",
      "Episode 670\tAverage Score: 0.091\n",
      "Episode 680\tAverage Score: 0.095\n",
      "Episode 690\tAverage Score: 0.091\n",
      "Episode 700\tAverage Score: 0.092\n",
      "Episode 710\tAverage Score: 0.092\n",
      "Episode 720\tAverage Score: 0.093\n",
      "Episode 730\tAverage Score: 0.096\n",
      "Episode 740\tAverage Score: 0.094\n",
      "Episode 750\tAverage Score: 0.098\n",
      "Episode 760\tAverage Score: 0.103\n",
      "Episode 770\tAverage Score: 0.106\n",
      "Episode 780\tAverage Score: 0.102\n",
      "Episode 790\tAverage Score: 0.109\n",
      "Episode 800\tAverage Score: 0.110\n",
      "Episode 810\tAverage Score: 0.110\n",
      "Episode 820\tAverage Score: 0.105\n",
      "Episode 830\tAverage Score: 0.103\n",
      "Episode 840\tAverage Score: 0.100\n",
      "Episode 850\tAverage Score: 0.101\n",
      "Episode 860\tAverage Score: 0.101\n",
      "Episode 870\tAverage Score: 0.096\n",
      "Episode 880\tAverage Score: 0.095\n",
      "Episode 890\tAverage Score: 0.086\n",
      "Episode 900\tAverage Score: 0.084\n",
      "Episode 910\tAverage Score: 0.085\n",
      "Episode 920\tAverage Score: 0.093\n",
      "Episode 930\tAverage Score: 0.096\n",
      "Episode 940\tAverage Score: 0.097\n",
      "Episode 950\tAverage Score: 0.084\n",
      "Episode 960\tAverage Score: 0.075\n",
      "Episode 970\tAverage Score: 0.071\n",
      "Episode 980\tAverage Score: 0.075\n",
      "Episode 990\tAverage Score: 0.078\n",
      "Episode 1000\tAverage Score: 0.076\n",
      "Episode 1010\tAverage Score: 0.075\n",
      "Episode 1020\tAverage Score: 0.072\n",
      "Episode 1030\tAverage Score: 0.067\n",
      "Episode 1040\tAverage Score: 0.074\n",
      "Episode 1050\tAverage Score: 0.084\n",
      "Episode 1060\tAverage Score: 0.091\n",
      "Episode 1070\tAverage Score: 0.095\n",
      "Episode 1080\tAverage Score: 0.094\n",
      "Episode 1090\tAverage Score: 0.101\n",
      "Episode 1100\tAverage Score: 0.100\n",
      "Episode 1110\tAverage Score: 0.100\n",
      "Episode 1120\tAverage Score: 0.096\n",
      "Episode 1130\tAverage Score: 0.092\n",
      "Episode 1140\tAverage Score: 0.080\n",
      "Episode 1150\tAverage Score: 0.079\n",
      "Episode 1160\tAverage Score: 0.080\n",
      "Episode 1170\tAverage Score: 0.087\n",
      "Episode 1180\tAverage Score: 0.099\n",
      "Episode 1190\tAverage Score: 0.100\n",
      "Episode 1200\tAverage Score: 0.122\n",
      "Episode 1210\tAverage Score: 0.126\n",
      "Episode 1220\tAverage Score: 0.133\n",
      "Episode 1230\tAverage Score: 0.144\n",
      "Episode 1240\tAverage Score: 0.154\n",
      "Episode 1250\tAverage Score: 0.156\n",
      "Episode 1260\tAverage Score: 0.163\n",
      "Episode 1270\tAverage Score: 0.163\n",
      "Episode 1280\tAverage Score: 0.157\n",
      "Episode 1290\tAverage Score: 0.162\n",
      "Episode 1300\tAverage Score: 0.149\n",
      "Episode 1310\tAverage Score: 0.158\n",
      "Episode 1320\tAverage Score: 0.159\n",
      "Episode 1330\tAverage Score: 0.155\n",
      "Episode 1340\tAverage Score: 0.161\n",
      "Episode 1350\tAverage Score: 0.167\n",
      "Episode 1360\tAverage Score: 0.173\n",
      "Episode 1370\tAverage Score: 0.185\n",
      "Episode 1380\tAverage Score: 0.182\n",
      "Episode 1390\tAverage Score: 0.191\n",
      "Episode 1400\tAverage Score: 0.196\n",
      "Episode 1410\tAverage Score: 0.202\n",
      "Episode 1420\tAverage Score: 0.215\n",
      "Episode 1430\tAverage Score: 0.235\n",
      "Episode 1440\tAverage Score: 0.253\n",
      "Episode 1450\tAverage Score: 0.267\n",
      "Episode 1460\tAverage Score: 0.266\n",
      "Episode 1470\tAverage Score: 0.272\n",
      "Episode 1480\tAverage Score: 0.290\n",
      "Episode 1490\tAverage Score: 0.295\n",
      "Episode 1500\tAverage Score: 0.308\n",
      "Episode 1510\tAverage Score: 0.350\n",
      "Episode 1520\tAverage Score: 0.363\n",
      "Episode 1530\tAverage Score: 0.373\n",
      "Episode 1540\tAverage Score: 0.378\n",
      "Episode 1550\tAverage Score: 0.373\n",
      "Episode 1560\tAverage Score: 0.373\n",
      "Episode 1570\tAverage Score: 0.405\n",
      "Episode 1580\tAverage Score: 0.487\n",
      "Episode 1581\tAverage Score: 0.509\n",
      "Environment solved in 1581 episodes!\tAverage Score: 0.509\n",
      "\n",
      "Environment solved with BUFFER in 1581 episodes!\tAverage Score: 0.51\n"
     ]
    }
   ],
   "source": [
    "agent_0 = Agent(state_size, action_size, random_seed=0, num_agents=1)\n",
    "agent_1 = Agent(state_size, action_size, random_seed=0, num_agents=1)\n",
    "with active_session():\n",
    "    scores, avgs = ma_ddpg()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Now Let's Plot Our Results From Training\n",
    "\n",
    "There are two lines in the plot below.  The Blue line shows what the scores where during training.  The Yellow line shows the average score over 100 episodes.\n",
    "\n",
    "We are considered \"trained\" when we can average a score of 0.5 for every Agent over 100 episodes.  And that happened at episode 1581.\n",
    "\n",
    "This time I didn't bother to train a little bit longer in an attempt to ensure that my saved weights will give me a good chance of getting a score of 0.5 or better in the future.  I had a REALLY hard time getting to 0.5, and it took a LONG time.  So I'm going to take my win and move on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XecVNXd+PHP2V5hd9mFXVh6D7CNBcGCBBRswR5Ao7FEYzcaY0JMMc/vMcUU8zwm1mg0CS5EiOUxRlEQ1ESR3qvs0suyBba3Ob8/ztydsjOzM8vO7izzfb9e85qZW8/cmfudc0+7SmuNEEKIs19EdydACCFE15CAL4QQYUICvhBChAkJ+EIIESYk4AshRJiQgC+EEGFCAr4QQoQJCfhCCBEmJOALIUSYiOruBDhLT0/XQ4YM6e5kCCFEj7Fu3bqTWusMf5YNqYA/ZMgQ1q5d293JEEKIHkMptd/fZaVIRwghwoQEfCGECBMS8IUQIkwEvQxfKRUJrAUOa62vCHT9pqYmDh06RH19fecnTnSpuLg4srOziY6O7u6kCBGWuqLS9kFgB9CrIysfOnSI5ORkhgwZglKqc1MmuozWmrKyMg4dOsTQoUO7OzlChKWgFukopbKBy4E/dXQb9fX19OnTR4J9D6eUok+fPnKlJkQ3CnYZ/u+BRwHbmWxEgv3ZQb5HIbpX0AK+UuoK4ITWel07y92plFqrlFpbWloarOQIIUTYC2YO/zxgjlKqBFgEzFBK/c19Ia31C1rrQq11YUaGX53FhBBCdEDQAr7WeoHWOltrPQSYB6zQWn8jWPsLJqUUN910U+v75uZmMjIyuOIK10ZHV155JVOnTm13e48//jgDBgwgLy+PkSNHcs0117B9+/bW+dOnT2f06NHk5OQwZswY7rvvPiorK1vnR0ZGkpeXx/jx47n++uupra0F4Pjx49xwww0MGzaMiRMnMnXqVN544402+7fZbDzwwAOMHz+eCRMmMGnSJIqLiwM+LkKInkXa4fshMTGRrVu3UldXB8AHH3zAgAEDXJaprKxk/fr1VFZW+hU8H3roITZu3MiePXuYO3cuM2bMwLlIa+HChWzevJnNmzcTGxvLlVde2TovPj6ejRs3snXrVmJiYnjuuefQWnPVVVcxbdo09u3bx7p161i0aBGHDh1qs+/Fixdz5MgRNm/ezJYtW3jjjTdISUnp6OEBzJ+gECK0dUnA11qv7Egb/FBy6aWX8s9//hOAoqIi5s+f7zJ/6dKlfO1rX2PevHksWrQooG3PnTuXWbNm8dprr7WZFxMTw5NPPsmBAwfYtGlTm/kXXHABe/fuZcWKFcTExHDXXXe1zhs8eDD3339/m3WOHj1KVlYWERHm68/OziY1NRWA9957j4KCAnJzc5k5cyYA5eXlXHXVVeTk5DBlyhQ2b94MmCuVO++8k1mzZnHzzTfT0tLC9773PSZNmkROTg7PP/98QMdBCBFcPS6Hr1TnP/xhBfL6+no2b97MOeec4zLf+hOYP38+RUVFAX+ugoICdu7c6XFeZGQkubm5beY3Nzfzr3/9iwkTJrBt2zYKCgr82tfXv/51/u///o+8vDy++93vsmHDBgBKS0u54447WLp0KZs2beL1118H4Kc//Sn5+fls3ryZn//859x8882t21q3bh1vvfUWr732Gi+99BK9e/dmzZo1rFmzhhdffFGKioQIIT0u4Gvd+Q9/5OTkUFJSQlFREZdddpnLvOPHj7N3717OP/98Ro0aRVRUFFu3bg3wc/lOiPP8uro68vLyKCwsZNCgQdx+++1tlr/33nvJzc1l0qRJbeZlZ2eza9cufvGLXxAREcHMmTNZvnw5n3/+OdOmTWvtGJWWlgbAp59+2lqHMWPGDMrKyjh16hQAc+bMIT4+HoBly5bxl7/8hby8PM455xzKysrYs2dPQMdBCBE8ITU8cqibM2cOjzzyCCtXrqSsrKx1+uLFi6moqGgNlKdPn2bRokX893//t9/b3rBhA4WFhR7ntbS0sGXLFsaOHQs4yvCdjRs3jqVLl7a+/+Mf/8jJkye9bjM2NpZLL72USy+9lH79+vHmm29y8cUXe2wr7+nPyFouMTHRZbmnn36a2bNnt/NphRDdocfl8LvTbbfdxk9+8hMmTJjgMr2oqIj33nuPkpISSkpKWitM/bV06VKWLVvWpl4AzFhCCxYsYODAgeTk5HjdxowZM6ivr+fZZ59tnWa13nG3fv16jhw5ApgWO5s3b2bw4MFMnTqVVatWtRbDlJeXAzBt2jQWLlwIwMqVK0lPT6dXr7YjZcyePZtnn32WpqYmAHbv3k1NTY0/h0AI0QUkhx+A7OxsHnzwQZdpJSUlHDhwgClTprROGzp0KL169WL16tVtyvotTz31FH/729+oqalh/PjxrFixAud+CDfeeCOxsbE0NDRw0UUX8dZbb/lMm1KKN998k4ceeognn3ySjIwMEhMT+dWvftVm2RMnTnDHHXfQ0NAAwOTJk7nvvvuIi4vjhRde4JprrsFms9G3b18++OADHn/8cW699VZycnJISEjg1Vdf9ZiGb33rW5SUlFBQUIDWmoyMDN58802f6RZCdMwLL8C3vx3YOqq9suOuVFhYqN3veLVjx47WogzR88n3KUTnuOMO+NOfANQ6rbXnsls3UqQjhBBhQop0guiJJ55obdpouf7663nssce6KUVCiLNFRwpnJOAH0WOPPSbBXQgRMqRIRwghwoQEfCGE6IFsHbjLiAR8IYQIExLwhRAiTEjA94OMhy+ECDUdaaUjAd8PMh5++2Q8fCFCnwR8P8l4+DIevhA9XY8L+DIevoOMhy+ECESPC/gyHr6Mhy+E6BjpaRsAGQ/fQcbDF6Ln6XE5/O4k4+HLePhChAoZSyfIZDx8GQ9fiJ5MxsMXXUq+TyE6xy23gMl7yXj4QghxVpMinRAj4+ELIUKJBPwgkvHwhRChRIp0hBAiTEjAF0KIMCEBXwgheiAZLTNInnjiCcaNG0dOTg55eXmsXr26S/e/YcMGlFK8//77XbpfIcTZRSpt2/HZZ5/xzjvvsH79emJjYzl58iSNjY1ntM3m5maiovw/9EVFRZx//vkUFRV1yrAFWmu01q2jZQohwoOc8e04evQo6enpxMbGApCenk7//v0BWLNmDeeeey65ublMnjyZqqoq6uvrufXWW5kwYQL5+fl89NFHALzyyitcf/31fO1rX2PWrFkA/PrXv24dSvinP/2px/1rrVmyZAmvvPIKy5Yto76+HoDvf//7PPPMM63LPf744/z2t7/1ut2SkhLGjh3LPffcQ0FBAQcPHuTuu++msLCQcePGuez/3XffZcyYMZx//vk88MADrTd6qamp4bbbbmPSpEnk5+e32/tXCBFirNxeKDwmTpyo3W3fvt3pXTB371lVVZXOzc3VI0eO1HfffbdeuXKl1lrrhoYGPXToUP3FF19orbU+deqUbmpq0r/5zW/0LbfcorXWeseOHXrgwIG6rq5O//nPf9YDBgzQZWVlWmut33//fX3HHXdom82mW1pa9OWXX65XrVrVZv+ffPKJnjFjhtZa6/nz5+ulS5dqrbVev369njZtWutyY8eO1fv37/e63eLiYq2U0p999lnrOlZampub9YUXXqg3bdqk6+rqdHZ2tt63b5/WWut58+bpyy+/XGut9YIFC/Rf//pXrbXWFRUVeuTIkbq6utrrsfPE9fsUQnTUTTe1jve7VvsZ6CSH346kpCTWrVvHCy+8QEZGBnPnzuWVV15h165dZGVltQ4/3KtXL6KiolyGEh4zZgyDBw9m9+7dAFx88cWtQw4vW7aMZcuWkZ+f3zoWvqehhIuKipg3bx5gxuS3xtrPz8/nxIkTHDlyhE2bNpGamsqgQYN8bnfw4MEuY/78/e9/p6CggPz8fLZt28b27dvZuXMnw4YNax3503lAt2XLlvHLX/6SvLw8pk+fTn19PQcOHOjU4y2E8I/NFvg6Uobvh8jISKZPn8706dOZMGECr776KgUFBX4PJWxxH0p4wYIFfPvb3/a6fEtLC0uXLuXtt9/miSeeQGtNWVkZVVVVJCcnc91117FkyRKOHTvW+qfgbbslJSUu+y8uLuY3v/kNa9asITU1lVtuuYX6+nqf6ddas3TpUkaPHu11GSFE6OphOXwdxIdnu3btcsl5b9y4kcGDBzNmzBiOHDnCmjVrAKiqqqK5udllKOHdu3dz4MABjwFy9uzZvPzyy1RXVwNw+PBhTpw44bLMhx9+SG5uLgcPHqSkpIT9+/dz7bXXto5Aad2Fa8mSJVx33XV+bxfMmP2JiYn07t2b48eP869//QswVyX79u2jpKQEMGP9O6f56aefbv1TsO6UJYToGSSH347q6mruv/9+KisriYqKYsSIEbzwwgvExMSwePFi7r//furq6oiPj+fDDz/knnvu4a677mLChAlERUXxyiuvtFb4Ops1axY7duxg6tSpgCk6+tvf/kbfvn1blykqKuLqq692We/aa6/l2Wef5aabbmLcuHFUVVUxYMAAsrKyfG43MjLSZTu5ubnk5+czbtw4hg0bxnnnnQeYm6s888wzXHLJJaSnpzN58uTWdX784x/zne98h5ycHLTWDBkyhHfeeacTjrIQoivI8MiijerqapKSktBac++99zJy5EgeeuihTtm2fJ9CdI4bb4TXXgMZHlmckRdffJG8vDzGjRvHqVOnfNYzCCF6DinSEW089NBDnZajF0KEjqDl8JVScUqpL5RSm5RS25RSP+votkKp2El0nHyPQnSeUBtLpwGYobXOBfKAS5RSU9pZp424uDjKysokWPRwVpPSuLi47k6KEGEraEU62kToavvbaPsj4KidnZ3NoUOHKC0t7czkiW4QFxdHdnZ2dydDiLAV1DJ8pVQksA4YAfxRax3wMJPR0dGtvT6FEEJ0XFBb6WitW7TWeUA2MFkpNd59GaXUnUqptUqptZKLF0KI4OmSZpla60pgJXCJh3kvaK0LtdaFGRkZXZEcIYTo8UKq0lYplaGUSrG/jgcuAnYGa39CCCF8C2YZfhbwqr0cPwL4u9Za+uELIUQ3CWYrnc1AfrC2L4QQIjAytIIQQvRAIVWGL4QQIrRIwBdCiDAhAV8IIXqgjtziUAK+EEKECQn4QggRJiTgCyFEmJCAL4QQYUICvhBChAkJ+EIIESYk4AshRA8kPW2FEEJ4JQFfCCHChAR8IYQIExLwhRCiB5IyfCGEEF5JwBdCiDAhAV8IIXogKdIRQgjhlQR8IYQIExLwhRAiTEjAF0KIMCEBXwgheiCptBVCCOGVBHwhhAgTEvCFECJMSMAXQogeSMrwhRBCeCUBXwghwoQEfCGE6IGkSEcIIYRXEvCFECJMSMAXQogwIQFfCCHChAR8IYTogYJaaauUOl8pdav9dYZSamjguxNCCNFd/Ar4SqmfAt8HFtgnRQN/C1aihBBCdD5/c/hXA3OAGgCt9REgOViJEkII0fn8DfiNWmsNaAClVGLwkiSEECIY/A34f1dKPQ+kKKXuAD4EXvS1glJqoFLqI6XUDqXUNqXUg2eaWCGEEB0X5c9CWuvfKKUuBk4Do4GfaK0/aGe1ZuC7Wuv1SqlkYJ1S6gOt9fYzS7IQQgibLfB12g34SqlI4H2t9UVAe0G+ldb6KHDU/rpKKbUDGABIwBdCiG7QbpGO1roFqFVK9e7oTpRSQ4B8YLWHeXcqpdYqpdaWlpZ2dBdCCCHa4VeRDlAPbFFKfYC9pQ6A1vqB9lZUSiUBS4HvaK1Pu8/XWr8AvABQWFjYga4EQggRfjrS8crfgP9P+yMgSqloTLBfqLX+R6DrCyGE6Dz+Vtq+qpSKAUbZJ+3SWjf5WkcppYCXgB1a69+dWTKFEEKcKX972k4H9gB/BJ4BdiulprWz2nnATcAMpdRG++OyM0msEEIII5hFOr8FZmmtdwEopUYBRcBE74nRnwIq8CQJIYQIBn87XkVbwR5Aa70bM56OEEKIHsLfHP5apdRLwF/t728E1gUnSUIIIYLB34B/N3Av8ACmmOZjTFm+EEKIHsLfgB8F/I/V2sbe+zY2aKkSQgjhVU4OpKQEvp6/ZfjLgXin9/GYAdSEEEJ0sS1bYNeu9pdz52/Aj9NaV1tv7K8TAt+dEEKI7uJvwK9RShVYb5RShUBdcJIkhBAiGPwtw/8O8LpS6gjmJij9gblBS5UQQohO5zOHr5SapJTK1FqvAcYAizHj3L8HFHdB+oQQQnSS9op0ngca7a+nAj/EDK9QgX2ESyGEED1De0U6kVrrcvvrucALWuulwFKl1MbgJk0IIYQ3HRlLp70cfqRSyvpTmAmscJrnb/m/EEKIENBe0C4CVimlTmJa5XwCoJQaAZwKctqEEOKsdvo09OrVdfvzGfC11k8opZYDWcAyrVsvIiKA+4OdOCGEOFutXQuTJnWsaKaj2i2W0Vp/7mHa7uAkRwghwsOxY12/T387XgkhhOjhJOALIUSYkIAvhBA9yOLF5jkYzTKFEEKEkHnzOr6uBHwhhAgTEvCFECJMSMAXQogwIQFfCCF6IKm0FUII4ZUEfCGECBMS8IUQogeSIh0hhBBeScAXQogeSHL4QgghvJKAL4QQYUICvhBC9EBSpCOEEMIrCfhCCNFJ0tJgwADHe6Xgm9/0f32lYOFCz9NvuOHM09fuLQ6FEEL4p6LCPJytWxfYNrZv9zx9/fqOpcmZ5PCFECKIAi1rj4z0Ps9m6/h2QQK+EEKElCgf5S7OAb8jJOALIYQX27bBsWPtL9fcDP/+t+d57eXEd+2Cjz92vI+ONs+HDpl5zk6edLyurm4/Xe6kDF8IIbwYPx769oXjx30vd9998Pzznue1F/DHjHFdzsrhjxkDNTWu60+Z4njd0uJ7u54ELYevlHpZKXVCKbU1WPsQQohgq6pqf5nKys7bn1WGX1fXdt6pU2e27WAW6bwCXBLE7QshRNAp1f4yvsrdveXwvZXHW0U6ntbrSEWts6AFfK31x0B5sLYvhBChIibG+zxvQdrbdF9/HmdKKm2FEOIMxcZ6n+cc2G02ePhhx2tPrBy+r+10VLcHfKXUnUqptUqptaWlpd2dHCGECJi3IO1u+3Z46infy/hqhx+yRTr+0lq/oLUu1FoXZmRkdHdyhBAiYL5y+N645/Cbm82zr4B/pro94AshRCjzJ1fdGWX4jY3t7y9kc/hKqSLgM2C0UuqQUur2YO1LCCG6U0cCvnsOv77e8/TOFMxWOvO11lla62itdbbW+qVg7UsIIbpToEU62dlt/wisgH/HHd7XC9kcvhBCnA38aYcf4SOSegrShw+3zck3NZlnqwdtZ7TKcScBXwghzlBnFMPU1p75NtojAV8IIc5QRypa3f8kamp870MpKdIRQohu5yuH797xytN0kBy+EEJ0GaVgzZqOretvzts54Lv/SXgaLK2zScAXQgi7zZs7tp6/OXyrc5X7dOjY+PaBkoAvhBB2HRljHvzP4ftazp8cvpThCyFEJwlGwPdWhu+t41UwScAPM5WV/t2yTYhwUV1tbicI3gN3czN8+aX3bfj6o/BWpHP4sOtyznfVcr+1oadtdYQE/DAzbBj079/dqRAidOTnw8CB5rW3wD13LowY4X0b/rbDdw7YP/6x67wf/cjx2rrtYWeTgB9mTp0KTg8+IXqqEyccr70FbusKwBt/i3Scc/jdQQK+EELYecvhtze8QkeaZXaEFOkIIUQn6Wilrb+BvKPb7ywS8HuwHTtgyZLuToUIBpsNfvvb7k5F51iyxHMl5Mcfw4oVXZ8eXzorh/+733me15EcfmcOlywBvwe74AK4/vruToUIhpUr4ZFHujsVneP662H69LbTp0+HmTO7OjW+eStjDzTgf/e7jtfOfyIdCd6dWe4vAb8H6+7LQxE8Z1vFuqffaih+RuuuU878GR7Z17novM2OnLNWwNdayvCFED2UP4G0qzU0dGw9X4HYGuceOj+Hf/75nwS0LQn4Qohu4eumId3FUw4f2v9z8hXInQP2meTw3UVGNvP++7MD2lYIHvLgGTXK9DQNRTNmwDvvdN72vvnNs6fSL5zMnw9/+EPw93PuuabStDtFRgZv2/n5sG1b+8s98QScPu14bwX88nITL8CMU79+vWOZhx6C3r1hwABza8NHH/XdeKKuznzWGTPOLIe/f7/rFchVV71JQkKAQ2xqrUPmMXHiRB1MoPV77wV1Fx0GWufnB7ZOSopZz9v24uLaTo+I8L6O6H6gdXKy1suXB/d7Aq1nzQre9t33lZHRdnpsbPA+I2h9993tLxcZaZWMm8e3vmWm/+MfrtOth7Vtfx6eln39df/Xtx6HD3uabtNr1kzUWqOBtdrPGBtWOXyA6OjuTkHXCcVKMRE6QrEMvbtZ5e2d2RTSmbciHV/FW56KdEaN2k1h4bqA9x92AT8qqrtTIIQIVd7K8DuLtz8SX3++7n8SmZlHueGG1zq0/7ALf8EsNww1ksPv+Wy20Kzc7AyheIVh5fCDde74yuG7z5swYTMjR+7BZrsCiAXgzjuf5/nn7+rw/s+6n9LPfgbx8W2nB9J5IToafv97/5Y9csT8cIuL2857+GFITvZ/v52tsTH4+9+2rWMn7tq1XXvCDx0Kl1zSdfs7E843qw7mYFvvv28qN7/xDejbNzj7sL5j9+96xIiuGf/dG6U8798K+MHq4+Ithx8ZCdnZB1m58kJqahJYvXoymzfnsnTpdQwfHofWip07R59RsIezMOC//rrnL9Ka5s8/d3MzvP22f/uzxrR2H9sa4K23uua2Zb4Ee/87dnRsPedWD12hpASWL+/afZ4JK9AHe3TFjRvhn/+E0tLg7sedr7Hlg80Kup5uGh7MMvz0dM9/JF/72tvs3ZvNwYODuPDCj0lIqGPy5LY31x09enebad/+9nMBpeGsK9LxVgZnfbn+ltH5m/u0Tshgl/2Fqt69zXOgRQ9xccFJz9lAKUdgaGyEhITuTU9P5S1z5ysWBDPga912u7ff/if+9Kc7At5WVVUSkyatYdeuMYD/uf6zLofvLfBa94v091LN34Bv/UCce9MFuo2ezKoTCfRKQgK+b12Vw+8qoXQuWL9VT71qnYcx6Gw2m9lufv56fvazn6C18hrs//CHe7nssn8yatQuli+f0Tr99devY8qUz+jVq8oe7ANz1uXwPQVecNzWz9cJVFkJKSne5584YXJbSUlt9+dpv85/PidPQkwM9OrlmFZeDmlpjjRVVbmuf/Kko1XRqVOmo4enVkbO6fZ0mVpb27aYq7bW5Mjj4szNHdLTHUG4udnMt9KVlmaei4tNWbgzK82lpa6frT1W81h/rgyc0+qNdSwD5f6dV1eb7ykmxnTISUgIrGVXZaX5TGlpjm23l7bGRvP9OB8/q4PgmQR8f46be9qdj0V9vfkNJiWZ6adPm23FxHj/TJWV5nNERLjeNKSpyRTjJCa2n3t2/z06c99vc7OZlpRk0muzOb6vigrXdaznmhoz3dPNgHydz+C5rs4b545fs2a9z9e//ga33/483/qW5+Xvv/9/+fDDi9i5c6zL9FmzlnfeFYe/Dfa74tEZHa/69m3bmePnP3d0WHjrLe/rOneMuOgi13mlpa4dKizvvmum/eMfnrfn3gHDff7y5VoXFjrmL17smJ+YaDqsWPOmT3dd3+p4BSYdWmvdq5fWSUntd+aIjTXHykrHuec6tjttmuuyLS1aV1S47sf9M37xhffj6smSJWa9U6faXzY+Xuu0NO/z9+1re2zdgelk42znTs/fyVe+4ng9eXL76XNf3/k7//TT9tOWn++6Tu/eju0cPBjY/p3FxDi+Y2/pBNffkbMBA9p+npwcx+slSzxvd/78tvtor4OS+zYyM9tOb2oy8+rqHNMuvNBMS0z0vO1Vq1zTX1Ki9ZYt5nV0dNvlzznHLPvcc/6n33oo1aIXLHhCl5Wl6v/8Z4res2e4/vjj8/XJk2m6vfA3c+YWH9ttb9/+d7w663L4nuzZ43jdXpGO9Q/ufgnqnvu2tJcjAN/l+8ePw969jvfOlaA1Na45dl8VXQcPmmfnbuK+NDS4VtTt3+947T52eW2tyQ2B96EpAh10ysq5WjlCX+rqHEVynpSVBbZvy8mTnqcfOOB4HUiOzpmVI3Pelje+9uHrd9WexsYzq4x1vqm2paTE8XrfPs/r7W5bt+g361yxrsidWd/XiRMwaJB5bf1WrVy7u507zbPzVbS1rKdjG3idnCY9/SQLFvyChx9+qnXq1KmfAzBihPeTdvv2sbz33iU8/vjjZGYGcHnsJCsLjh71f/mwCPjObe87WobvrQmZP824fJVvR0a6XuJ6KpLpCs7pdy9GqK52fAZvJ0JHA763P9JAdLQJnQ5iPwXrOPnzfbpfrjv/9kKtDN/X78TTMoHyda5YAf/kSUfAb+/4WJmB8nLzXFPj/c8BHN9bewE/MbGahx56ivvu+wP9+p3wvTDQ0hLB669fz549eaxbN4a3356D1o6yzMGD291Ep5CA74V7wPeWw7R+cL5yYr5O+shI1zR1V8D3NaKfcw7b2+cMtJWStT9/r0j82VagLYWcK0Y7uwe29T368336+uMJtYDvnFZv59KZlDf7E/Cdr+jaO5+tQG/dqPz0ad/nqqcr9qioJi68cBXTpn1MS0skeXkbufrqN33ud9OmHLZuHU9dXTzr1k1k8eK5VFSk0bev603TLbGxvj+HN4FWhveIgL9wIdx4o+d5b70FF17ou7LVOQg0NZlRJHfsgGuvNR1Ajh+H888381euNM/79pkvJi3NtO3fuNHztt1/IO+8A1OmmEoni6cchXVSrFjhOt99WecTzD0wuBevOFeS+UNr+POfzevycvP44IO2J9Ef/uBofuktsK9fDxMmmON64oQJoFdfbSr4Xn3VvP/3v82IgZMnm04/4Dngf/yxqXDbscP1CmDlSpMTGjrU/CZaWkxxWFaWmd/cDFu3mquNtWvNtK9+FcaPN69bWuDdd2HsWFOEZf2Jbd9u0mYFEuegozUUFZlRLIuKzN2b3P8cliwxaTjvPMc0q8LQGgHV+lOprjaf/drE1/KaAAAbRklEQVRr237u5583z1bxGZjf3k9+4nj/2Wfmtz52rNnm/ffDY4/BunVw5ZUmKH7+OVxxhSP927aZabffbqY5343Jnc0G//iH+Tye/mwaGx39Gazix3ffhcJCR+etQH+H//VfZr/R0a6B9oknzOe56y7TV+B//9dMf/BB871WVDiOszfW72zLFvP885+3LQJRysbYsTt46KGn6Nu3jJ07BzBlSikrVx4jN3cTKSmnaE9VVRINDbHMnbuYFStmAJ4jsadgD547i1o69UrU38L+rnh4q7T1VSkIpuLG4qnS9p57HBUcv/+9a4VHr16ulTrOjwkTtP5//893JdMrr5hpzz3n2MaUKa7bW7fO8fqjj8y8ujrPFTBWhZen9GRlOeZVVbnOe+45rYcNC7yiyflx7rntL/PUU22Pv/UYMsT1/R13eP4czg9PFX++Kp379HFUxLk/Tp1qW8EVHd02DX36OPYNpmLW03fs6f0Pf+j5N5iS4rrOF1+4rl9TY+bPmdP2N9ReJXtTk+u+EhLM60cfbZvGiRPbpn/8eMe0kpK223eutP3rXz2nwdv3aE3PzXVdpr3v3d9HWtpJfdllxXrgwP36rrue0QsXztd/+cs39JIl1+gVK6brH//4Z/rRR3+p77nnD/rRR3+pb7nlZd2v31E9duw2ffvtL+r58xfqW255Wd9000J97rmf6ssv/z/96KO/1K+9Nk//+99T9f79A7XuYMhqaVH60Ud/qePias/4c86cGfg6116r9Wuvac3ZVGlr5TJ8Xbr4KpMD1yId9xyBr7Ln06cdl4TeeLoEdM8FOxcHWa+9Xer76m7unPN2r1Crr2//OPgSE9NAY2MzvXo1U1ubQHNzNKBJTa0gIsJGWVkf4uPraGyMx1vuxf0z+XPvAU+X8L6KQWprvX/O+npzKjjzlEu1jrH1XQRSrOQtbe7T3SuE6+tNE09Pn9c9ze5OnoTMTMd76zN5SounY+O8T085Yuf9d3S4g44WRUZEOBcBaYYOLWbOnLeZOXM5eXkbGTjQ9+XCV7+6smM77oDi4nEsW3YepaUZ/OhHXyUi4jyefDLwDiWHD8Pw4eZYl5VBnz6BFSnGxpq4ZY3Bf8MN/q8b8gHfCrhn0lrBOeC3F8CdNTe3fwJY6XIOLO5DMDufDNYJ6a2s0ltdwfDhe8nL+xI4AQymqmoAeXkmUmmtSElpYMKECK66ah21tQkMGnSApqZoMjOPsWbNJBITaxg9ehfDhu1j9OhdpKZWYLNFUF6eRkZGKVlZbZtFNDdHEhVl/mVsNkVEhKa+vhfwFeACYDwXXDCElpZIysr6UFHh2hHEn3L9QP+kbDbfAT8Q1vfSfr2OBhQxMQ3ExDQBzZhTJxprUCt37i2HfKWtvYB/4oRrwD+TwdTc0+WekerK+yQXFq5h7tzXKShYS3r6SYYN20dS0hnkWs7QoUMDeOutK9m1azR9+pQxcuQe9u8fzH/+cy7JybMoK4tl2TKz7I9+1PH9pKc7/uSs7zKQQR1jYjp+K8aQD/hWs7Izqcx0PkECCfgtLe0HLSvQO58ovgK+dcI5B62IiBYuueQ9Zs5czqRJJZigbmPVqihiYhqZMmV1m/3m5cGGDa7TvvlN32n1JDvbwyBAdlawN2k0USku7jTwuf3hetek4uLhrF5dyNat4zlwYBC9euVg/hxivO7D0x+fr6s5rR3HMDn5NP37HyE1tYKsrKNERUWTmzuQxsYYEhJqycgopbo6GehNYWE9yclVDBx4kOjoCKCB0aPr+d73ahk+vJS0tBISEmpJTa1g4MCDQB/27y8nIaGWtLRytI6hweNZNpilSwtITKwD+rBoURPJyVWMHJnIq6/G09ISSW1tAnFxw4BULrooBq37A5OBpNbPZOnb9zgjRuxl2LB9TJiwherqJGJi+gJTgPFA1Bn1WvXUhDUiwsbw4fvIzd3EoEGnuPbaXlRVJZOefpLc3E3063eclpZGnn66D/X1cSil6d37FIcPDwDGUVAwnISETKA/3q7++vc/zEUXfUhu7iYyMkoZO3aHX+O519XFcfp0ComJVWzYkM+KFTM4diyTwYP309wcRUpKJYMH7ycqqpmamkT69TvOeef9m/r6ODZsKODo0UxqahLtmZqjlJX14csvh7N37wiKi4dy5Eh/GhpiKS4eSkWF995xV1xhAm1niIlpG/ADuU9HfHzHW7cp3V72ogsVFhbqtfbatmefNYFy4kRTQfPWWzBnjlnu7rtNJdq0aSY4FBaay5tzzjG5Ia3hO9+B554zFUkREY72w66XkA4vvwy33dZ2unX5ZElIqCEuLpKsrFiSkxVlZa7t/C3x8dDS0kBOzmZGjy5n1Kh/k5e3kePH+zF8+FEiIqqJj68jPr7Oaw67K9lsivr6uDa3TKuri6OlJbLDOa/m5mj27RvC4cMD2LlzDHv2jOTEib6UlAwhKakaMwrgGPr2jWTQoFLi4spITd2J1ori4qGkpZWTmXmM0aN30avXaWJjG8jKOsrYsTvo1asT2nR2E5tNUVubQG1tIlVVSWitSEsrJy3NVy1kAqtWFfLFF+fwxRfncOpUMvHxda1/br17t3DqVCQtLZEoFUlTk6KxMYaamkSiopopK+tHS0s0GzaM49xz/8OUKZ9z2WXvMmLEXpKTz3yUvWPHMrHZFF98MZnDh4eSnb2PadM+prY2gQEDjvi1jfLyVFavPoe3357DqlUXsnv3KFpaAsuXxsQ00NQUjVIRndZD9bLLTMOFoiLz3gqbHfnz1doU79XVmQr63r1h7lxYvNi/9ceMMf0LHGlQ67TWhf6sG7IB3zqQS5bAddeZg/H1r9M6b9Qo0+nCCvi5ufDSSx3bb0JCDX37lpGUdIqhQ4sZP34rw4btY8SIva05yIgIGy0tkfTt6+jJUlOTQFRUM7GxjZSVpXH0aBb79w8mLa28tQglNrbzRlX7/PNzqKhIZdCgA8TF1ZOWVklqajnFxUOoqEgFICqqmdOne2GzRTBtmrmj/dq1Ezl0KJvKyhQ+/fR8Nm/OobIyhf79T1BXF015eRolJUNIT4+mrKyRG288wMqVkTQ2xnDkyABAExnZgs0WQV7eRi6++AMGDjxEfv4uBg/e7vMq4WzS3BxJS0s0jY1RRETYSEzspja0Z5HKyrE8/PAjbNs2joEDh3LyZAZKKVaudB0mGuCqq+CWW0xrrd//HrKz4eKLTQaustKUZc+dC6mp8OSTJqC+/jp88onrPp95xuSQv/998/7mm02RyjXXmBZ81tV6Zia88Ya5ii8rMznzv/zFzLvnHvP8/vvwgx+YVnyjRrXtdHb//fD00+b18uUmnj3zjGm9VVlphqb+n/8xVxB33w2XX25akBUXw8yZ5vVnn5kr4bQ0mDrVpPvVV822IYQCvlLqEuB/gEjgT1rrX/pa3lPAf/FFuOMOeOUVR5FFoAFfKRszZy7nkkveo3fvU2RlHSU7+xApKZWkp58MiRO3sTGVhQuvZNmyWdTWJjBixD5SUspZs2YSmzblcuxYJo2NruXFu3fDyJHmB/vXv5ppDz8Mv/udeT1jhmn26c0Pf2iaqVkWLIBf/MJUCPfr5zu98fHmhDC9RDVZWUcZM2YnQ4cWM27cNgYP3k9+/gaGDSsO+Fj4q7ExmtLSDI4c6U91dRJKaQYMOElzcwsNDbFUVKQSF2eKcurr40hNTWD48P5AFIsWxVJWFse998YBfYBs+3MMOTn9iIurJz6+jlWrBhEdnU18fB2nTydTXKwYNsxqRlfLpElr6N//KL16JdLScpz6+jjy8ppZsyae/v0baGioJj39JAUFJygvryUz8xhDhpQwatRulyIzZwcPZtPQEEt1dRKbNuWSkFDL9devBg4G7ViWlqazcWMeR49m0a/fcSIjW4iIiOfLLzPZsCGf6uokevc+RVxcPXFx9fTufYqUlEr69j3B0KHFjBmzk+jo9joNfBU4hx/+sB8lJX157bUJwHhSUpTLuDY2mwnAt95qmg1bgd85VCkF3/ueCezOlDIZxNdfd53mzDl3XlBgmrR6c+GFptjSV5hcv96URGzYYO4v4L4vpUwu3p9GDB0RSMAPWhm+UioS+CNwMXAIWKOUeltrvd2/LWiio5uorFRcdNFKsrIqgGQgicmTYxgxogGIYfbsSsaPb2Dw4Aguv1yRn7+BSZPWEBdXT0xMI0OGlJCZeYy4uA7WcrhpbIzGZosIcHuTgFMUFV3MF18M59ChbCIibDQ2xhAbm8iiRZm8994YbrstsELCjAzz7HyTk/79Ha99DUcArhWB4GjP7jw4nDeJic4VkYqjR/tz9Gh/PvrIdbnU1HImTlxHUlI148Zto1+/44watZuMjFJstgiSkqrJyjpKU1M06ekD+PTT3hw61J9Ro3aTmFjDzp1jOHKkP8eOjWDPnizq6+MoL08DxvLJJ+nYbK61XRdeCKtWeU7zt79tivkAHnnEtJa49962y1ltti3NzVBVZQpZrWOelASlpQl8/PGFKGVaXVhDZJw4YfozZGc72qTPmweLFjm2GRPTQGxsA8nJVWRklJKcXAUMZ82aNBoa2rb8MAHnCFdfvZopUz4nN3cTkZEtNDTEU1sbR1lZHxobY+yB2kZsbANJSdXU18cxfPiXJCbWtBbdnDrVi8rKFLZtG8dLL93OJ59cQGlpBu7l755yrN7Ex9cyc+ZyMjJKiY1toH//I+TlbeTEib6sXHkJS5ZcQV2daWz+xz+allGvveb82Ryscm0r0+Ge07d4GxQukOGk28vv+tNYxCo2cj733IXKXcuCWWk7Gdirtd4HoJRaBFwJ+Aj424EcoBGtHQO6PPKI61Krneow33vP8dq6xAlUfX0sZWV9qKxM4fjxfmzdOp5du0azb98wSkqG0NQUzYABhzl6NIsvvxyOzRaJUjYyM48xa9YyKipS2bZtHJmZx+jf/wjHjpmKotraBHbuHIPW5kT6+c9NxyBnAwea545caFlj0FidosARkKD9gO9+QlkB35/RFb01MXRXUZHGhx9eDMCbb17tdTnrFm+zZ3uuoB82zHXslvHjPdfFuP+JOeuMXszWn2GgY9Q7d8QDaGyMpbExlqqqXvZiMxg3znvrC9OLuD9vvnm1y3F0r2PqTIGUT9fVJfDOO1/zOC8uzrWVkr8tgazfp7d6N2+9U311YnLXXhm/PwHfWsbXiKjhEPAH4HoNegg4x/cqdcAW34t0UGVlbz766KusX1/A7t2j2Lt3BOXlaZSXp3H6dC+8tS6wfPnlCJf3Wkdw9Gh/Xn31Fq/LgOvAYIMGmYCfnOyoZbeGG3YPCN7ExzsCufUj6tPHdR/Or731EAZHcExLM2WUw4Y5tus85IN7D0gwJ6M/A4P5y+qlmZnpeVCujAzX6dax7N3btWfqGB9DhDv/GQ4c6PkuZdA2wMTEtP0TTE93DDjXq5dJt5XDt47joEGOHP6oUd7TZRk50nVIXaXMtk+dMn8wngJwerr3z+HMOZccFWVeuwde9+958OC2A+l1RFqauRWopW9f1wHj+vf33Bdi3DjznJHhuYeqdZzdebtdY3R02+8xO9t7usG/89K6wvbVln7AgPa30xWCGfA9RdA2+Vil1J3AnWDKwTxpakqlvHwiWkcSFVVNZGQtLS2JREQ0ADaamtKw2WIAGzZbPFrnsG/fZJKSomlpGciRI/3Iy0umT59IJkwwrXvi482IfO7No4YMMZ1TbDaTm6uuNl3ZGxrMSXfkiGmqOWSICUDJyeZx5IiZHxtr1mlpMT+8nBzH53jjDccogCUl5gdr/TjPO89s7+BBE4x27jT7KC01aU1JMT+orCxTxu58Ajz4oDl28fGmpZIVGAcPNldDI0ea4JiebopiLEOHmi73EybApk2mPNNa98QJk8bUVBNUq6vNn1RysllmyhRYtsycRGlpJt0jRpigFxnpyAFXV7vmblpazHLl5eZPo6HBTBs92szftMmss3+/KUoZO9YEndGj4T//MUUnp0+bslJrHPQtW8xnr6w0Q2TMnm0CcHOzSVe/fiYN1j4APvrI+2iZx4+b79M6VocPu57M+/aZ47JsmTmmBQVm+7t2mRM7M9NUrGVlmRFOGxrMMldcYYpIrABu/TaqqsxnOuccM1xBRYVZJzvbpOE//3Ecw+Rkc3z69DGfffx4891GRJhHaalJm5Xrz8w00xMTzbysLMdv/sgR0zzYGpakTx8zJEVmpvlOcnPN95Gebsqp+/RxpKuy0hw/az8REWbb/fub6cXF5rtoaDDrOefwN292vTpct65t+fa+feb3uW+f52aI1nx3X3xh0u2+LJjfsfNvcf9+31eEYOoCnEez9WT8eMc+9u0zj5EjHWk+fLhj92oIhqBV2iqlpgKPa61n298vANBa/8LbOoWF4/TatUVOU4YDcZj/jhC5JhJCiBASEpW2wBpgpFJqKHAYmAe00wk4HlOGL4QQorMFLeBrrZuVUvcB72OaZb6std7WzmpCCCGCJKhDK2it3wXeDeY+hBBC+EcKxoUQIkxIwBdCiDAhAV8IIcKEBHwhhAgTEvCFECJMSMAXQogwIQFfCCHChAR8IYQIExLwhRAiTEjAF0KIMCEBXwghwoQEfCGECBMS8IUQIkwE7QYoHaGUqgI64aZqQZcOeLlfUkjpKemEnpNWSWfn6ylpDdV0DtZaZ7S/WJCHR+6AXf7euaU7KaXWSjo7V09Jq6Sz8/WUtPaUdPoiRTpCCBEmJOALIUSYCLWA/0J3J8BPks7O11PSKunsfD0lrT0lnV6FVKWtEEKI4Am1HL4QQoggCYmAr5S6RCm1Sym1Vyn1g25Oy0Cl1EdKqR1KqW1KqQft09OUUh8opfbYn1Pt05VS6n/tad+slCro4vRGKqU2KKXesb8fqpRabU/nYqVUjH16rP39Xvv8IV2czhSl1BKl1E77sZ0aisdUKfWQ/XvfqpQqUkrFhcoxVUq9rJQ6oZTa6jQt4GOolPqmffk9SqlvdlE6f23/7jcrpd5QSqU4zVtgT+cupdRsp+lBjwue0uo07xGllFZKpdvfd9sx7TRa6259AJHAl8AwIAbYBHylG9OTBRTYXycDu4GvAE8CP7BP/wHwK/vry4B/AQqYAqzu4vQ+DLwGvGN//3dgnv31c8Dd9tf3AM/ZX88DFndxOl8FvmV/HQOkhNoxBQYAxUC807G8JVSOKTANKAC2Ok0L6BgCacA++3Oq/XVqF6RzFhBlf/0rp3R+xX7OxwJD7bEgsqvigqe02qcPBN4H9gPp3X1MO+3zdnsCYCrwvtP7BcCC7k6XU3reAi7GdAjLsk/LwvQZAHgemO+0fOtyXZC2bGA5MAN4x/5DPOl0YrUeW/uPd6r9dZR9OdVF6exlD6TKbXpIHVNMwD9oP3Gj7Md0digdU2CIWyAN6BgC84Hnnaa7LBesdLrNuxpYaH/tcr5bx7Qr44KntAJLgFygBEfA79Zj2hmPUCjSsU4yyyH7tG5nv0TPB1YD/bTWRwHsz33ti3Vn+n8PPArY7O/7AJVa62YPaWlNp33+KfvyXWEYUAr82V789CelVCIhdky11oeB3wAHgKOYY7SO0DymlkCPYSicb7dhcsr4SE+3pVMpNQc4rLXe5DYr5NIaqFAI+MrDtG5vOqSUSgKWAt/RWp/2taiHaUFPv1LqCuCE1nqdn2npzuMchblsflZrnQ/UYIofvOmuY5oKXIkpWugPJAKX+khLSP527bylrVvTrJR6DGgGFlqTvKSnu34DCcBjwE88zfYwrduPaSBCIeAfwpSXWbKBI92UFgCUUtGYYL9Qa/0P++TjSqks+/ws4IR9enel/zxgjlKqBFiEKdb5PZCilLKGzHBOS2s67fN7A+VdkE5r34e01qvt75dg/gBC7ZheBBRrrUu11k3AP4BzCc1jagn0GHbb+WavzLwCuFHbyz5CMJ3DMX/4m+znVjawXimVGYJpDVgoBPw1wEh7S4gYTOXX292VGKWUAl4Cdmitf+c0623Aqn3/JqZs35p+s70GfwpwyrrEDiat9QKtdbbWegjmmK3QWt8IfARc5yWdVvqvsy/fJbkQrfUx4KBSarR90kxgOyF2TDFFOVOUUgn234GVzpA7pk4CPYbvA7OUUqn2K5pZ9mlBpZS6BPg+MEdrXeuW/nn2Fk9DgZHAF3RTXNBab9Fa99VaD7GfW4cwjTiOEWLHtEO6uxLBfn5chmkN8yXwWDen5XzM5dhmYKP9cRmmbHY5sMf+nGZfXgF/tKd9C1DYDWmejqOVzjDMCbMXeB2ItU+Ps7/fa58/rIvTmAestR/XNzGtGULumAI/A3YCW4G/YlqPhMQxBYowdQtNmEB0e0eOIaYMfa/9cWsXpXMvppzbOqeec1r+MXs6dwGXOk0PelzwlFa3+SU4Km277Zh21kN62gohRJgIhSIdIYQQXUACvhBChAkJ+EIIESYk4AshRJiQgC+EEGFCAr44KyilWpRSG50ePkdXVErdpZS6uRP2W2KNphjgerOVUo/b226/e6bpEMIfoXYTcyE6qk5rnefvwlrr54KZGD9cgOnQNQ34dzenRYQJCfjirGbvHr8Y+Kp90g1a671KqceBaq31b5RSDwB3YcZ42a61nqeUSgNexnS6qgXu1FpvVkr1wXTWycB0tlJO+/oG8ABmON/VwD1a6xa39MzFjPw4DDNuTz/gtFLqHK31nGAcAyEsUqQjzhbxbkU6c53mndZaTwb+gBlvyN0PgHytdQ4m8IPpcbvBPu2HwF/s038KfKrNIHBvA4MAlFJjgbnAefYrjRbgRvcdaa0X4xh/fQKmR2++BHvRFSSHL84Wvop0ipyen/IwfzOwUCn1JmbYBzBDbFwLoLVeoZTqo5TqjSmCucY+/Z9KqQr78jOBicAaMwwP8TgGMnM3EtM9HyBBa13lx+cT4oxJwBfhQHt5bbkcE8jnAD9WSo3D95C3nrahgFe11gt8JUQptRZIB6KUUtuBLKXURuB+rfUnvj+GEGdGinREOJjr9PyZ8wylVAQwUGv9EeZmMilAEvAx9iIZpdR04KQ290Vwnn4pZhA4MAOXXaeU6mufl6aUGuyeEK11IfBPTPn9k5hBwfIk2IuuIDl8cbaIt+eULe9pra2mmbFKqdWYDM58t/Uigb/Zi2sU8JTWutJeqftnpdRmTKWtNQTxz4AipdR6YBVmSGW01tuVUj8Cltn/RJqAezH3RHVXgKncvQf4nYf5QgSFjJYpzmr2VjqFWuuT3Z0WIbqbFOkIIUSYkBy+EEKECcnhCyFEmJCAL4QQYUICvhBChAkJ+EIIESYk4AshRJiQgC+EEGHi/wNGPE5oB5QL7QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fab1b9fd8d0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores, c='blue', linewidth=1, label=\"MA_DDPG Score\")\n",
    "plt.plot(np.arange(len(scores)), avgs, c='yellow', linewidth=3, label=\"Score Average\")\n",
    "plt.legend(loc='upper left')\n",
    "plt.margins(x=0.0, y=0.1)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we are finished, we can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}