42. Random Forest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aboriginal-connection",
   "metadata": {},
   "source": [
    "# Random Forest\n",
    "\n",
    "#### scikit-learn 中：随机森林分类器及回归器\n",
    "\n",
    "+ RandomForestClassifier()：分类器\n",
    "+ RandomForestRegressor()：回归器\n",
    "+ 特点：所有子模型在节点划分时，都是在随机的特征子集上寻找最优的花饭特征；\n",
    "+ 也就是在迭代寻找划分维度及其阈值时，不是对全部特征进行搜索，而是对部分特征进行搜索；\n",
    "+ 优点：这种方式增加了每一个子模型的随机性及差异性\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "adjusted-oxide",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "exact-jacksonville",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "X, y = datasets.make_moons(n_samples=500, noise=0.3, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "handy-protection",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABDj0lEQVR4nO2df5QcVZn3v0/39CQ9wc0kJG7IJAh7Tg68AiGRBOEkuwrIL4EwBh2QZd311TfwIiLoBsLqGwaWd8mPXUBUViJy1KMCUWAYfm1EYGWTXdwkhASykleEhWQGJIFMFKbJ9Ezf94/q6qmuvvfWvVXV3dXdz+ecnExXV1dVV3c/997nx/chIQQYhmGY1iFV7wtgGIZhagsbfoZhmBaDDT/DMEyLwYafYRimxWDDzzAM02K01fsCdEybNk0cccQR9b4MhmGYhmHr1q37hBDTdfsk2vAfccQR2LJlS70vg2EYpmEgoteC9mFXD8MwTIvBhp9hGKbFYMPPMAzTYrDhZxiGaTHY8DMMw7QYic7qYaLTt20AazfswuBQDjM7s1h+5lHont9V78tiGKaOsOFvYvq2DeC6B15ALj8GABgYyuG6B14AADb+DNPCsKuniVm7YVfJ6Lvk8mNYu2FXna6IYZgkwIa/iRkcylltZximNWDD38TM7MxabWcYpjVgw9/ELD/zKGQz6bJt2Uway888qk5XxDBMEuDgbhPjBnA5q4dhGC9s+Juc7vldbOgZhimDDT/DWMK1EUyjw4afYSzg2gimGeDgLsNYwLURTDPAM36GCcDr2hGKfbg2gmkk2PAziSUJvnS/a0cF10YwjQQbfiaRJMWXLnPt+OHaCKbRYMPPJBKdL93G8EddNehcOARwVg/TkLDhZxJJHDpDcawaZnZmMSA5Z1dnFptWnGp8LQyTJDirh0kkcegMxZGBw7IXTDPChp9JJHEY3DhWDd3zu3DBCV1IEwEA0kS44ASuhmYam1gMPxHdTURvEdGLiuc/TkQHiOj54r+VcZyXaV6653fh5qXHoaszC4LjWrl56XFWBjeOVUPftgHcv3UAY8JJ5BwTAvdvHUDftgHjYzBM0ojLx/8DAN8G8CPNPv8mhDg3pvMxLUBUnaHlZx5VkYppu2qIK8jMMEkiFsMvhHiGiI6I41gMExcm6qRBWT/czIZpRmqZ1XMyEW0HMAjgb4UQO2U7EdEyAMsA4PDDD6/h5THNiG7VYJL1o8rq4YItppGpVXD3OQAfEkIcD+BbAPpUOwoh1gkhFgghFkyfPr1Gl8fUgr5tA1i06ikcueJRLFr1VCg/eRzHcDHJ+uGsHqYZqcmMXwjxB8/fjxHRHUQ0TQixrxbnZ8IRp2RCHDn1fdsGsPzn25EfE6VjLP/5dqtjeDFx43AzG6YZqYnhJ6IZAH4vhBBEdCKclcbbtTg3E464JRPiCJLe8PDOktF3yY8J3PDwzlDXZOrG4WY2TLMRVzrnPQD+A8BRRLSHiL5ARJcR0WXFXT4N4MWij/92ABcJIVRCh0wCiFt+OI4g6f7hvNX2INiNw7QqcWX1fDbg+W/DSfdkGoS4s1mSGCRlNw7TqrBWDyMlbkMdR059ZzaDoVzl7L4zmwl1TUBrunGSIHfN1BeWbGCkxO0GiaMSt3fJMcikqGxbJkXoXXJMqGtqRdzYzUCxqYwbu+FK5NaCZ/yMlGq4QaLOrtk1Ex2uRGYANvwNRa2X6PV2g6jer64ga+2GXRgYyiFNhDEh0GV4n1rF/cGVyAzAhr9hSEpHqlph+379+7uiaib3qZXubRKD7EztYcNfZeKaSTbKEr1W79d/nvcOjipbJAbdp0a5t3EQR5CdaXzY8FeROGeSjbBEr9X7lZ0n7PGCztVscJyEAdjwV5U4Z5KNsESv1fs1aYAue12YczUj9Y7dMPWH0zmrSJwzySRWmfoF01Qz7zDv95Sj5QJ9pxw93fp4QfcpifeWYaoJz/irSJwzyWov0W198zJ3CwGQ6XCEeb9Pv7RXuV11X6d0ZNDR3mad1cPuD6bVYMNfReIOpFVriR7GNy9ztwigwviHfb+61dKtF86T3tfrzzsm9P1h9wfTSrDhryKNMpMM45tXGWYBpyo36vvVrZaStvphmEaDDX+VaYSZZFAsQmYIVYa5qzOLTStOjXxNQasl2X2Nw2C3Uk4/07qw4We0s2uVIbzghC7cv3WgavngtrP6MAZbNlC0Uk4/07pQkmXxFyxYILZs2VLvy2h6/EYTcIz4zUuPK0kg+OnyGMokuERUWUWqFYjqPevSRAmo+/tkmCCIaKsQYoFuH57xM9rZ9dX3PS99zeBQrsLd4qZ3ms7QgwYNG9eNbSqpamafIqCgmAt51SwBdv0wjQsbfgaAOhZhmpJq42ox2bdv2wCW/2w78gVPf92fyfvr9m0bsE4lVQ0IBQFk0lTR4tELu36YRocLuJgy/EVZpxw93ai4yaZVo8m+vf07S0bfJV8Q6O3fWXG83v6dUqNPgDLmoKstmNTeVuoboMJEJoJhkgob/ibDb7htGmzImnTcv3UAF5zQFdhAxaZKWbXvQFGHB4C005Zse9+2AeW+Amp3jC4IfSCXx6YVp+LVVeegSzFAUPHcDNOIsOFvIqJ2V1LNxJ9+aW/JEG5acarSJSRDtl03217+s+1WBlXX/F1nnLvndylbNnqvb/mZR0ln/iLg3AyTZNjwNxE27hYZUbR2bPRuZPu6uO4cUvhZ/Nt11yYAXHXf86WVj381dO7xhwVec/f8LqkbKejcDJNkOLjbREQRhQsTIPVik3ffPb8LW157Bz9+9nXpsVSuGwBws4/djB+TZORSYJhQCtp63VhPv7S3pO/jHSjda+9qMfVOpvnhGX8TYeNu8aMyoroAqRfT1Et31q0y+i4q33qXp6jMJsCaL4iKTB3XjeWuQPxdu1w3Eat3Ms1GLIafiO4moreI6EXF80REtxPRy0S0g4g+Esd5mXKiGCid9o5Jv1qT2IKpwZ7SkdG+lzB6/CoGh3KBLrLu+V24eelxgQFuJgQ71gO3Hgv0djr/71hf3+O0CHHN+H8A4CzN82cDmFP8twzAP8d0XsZDFAOlWhWkiQKDraaxBVuDrXovOteVaqWgYmZRUE6Gd3v3/K7AADdjyY71wMNXAgd2AxDO/w8sAx75avTjPHwlG38Nsfj4hRDPENERml3OB/Aj4ehDPEtEnUR0mBDijTjOz4wTVhROJooGOE3LgypVTWMLpsHQ/cN5XPfAC7h56XFSuYXJ2Yw0DtCZzWDTilOlcgwyvCsIWx8+K3jGwJM3Ann/fRfAlruBw08C5vaEP04+52w3PUaLUavgbheA3Z7He4rb2PAnBNdofW399pKv2yWXH8PX1m8vPTZV6vQbTtV+MnTVsUEZP95As+583tWQTd+EsAqecQ4WTTHwHNijeELYGW3VcZTHZ2oV3FWlQlfuSLSMiLYQ0Za9e+VdmJjq0D2/CwWFaN+YEFj+8+1Y/rPtFb580+peXRqnDNUKYWhYnvWzfzhfStUEgE0rTtUGiV1DaesiC5M2q4qDfKPvBeuCu6j1Golh8iz1czZGW3Uc3fFbnFrN+PcAmO15PAvAoGxHIcQ6AOsAR52z+pdWHRp1Rqablcv0a9zMGFfJU/d+/SmfQR+uLktJdY1+ITXTLmhBLjLv5xkmr181WPzk2ddLxzNdOTSNdPRpKx2fvuyO2hjt01Y6Pn2vuyeTdbYzUmpl+PsBXEFE9wL4KIADzezfj9rMo16DRt+2AQyPjFq/TqbUqcK737wbfqHM2de5WpafeRSuvu957cDhGkI3RhDlfprGDHQxAV3WlOy6VamwOveV7ByJnoDM7QFef9bx6XvvhK3Rdl1CT97orBQmz3Jez/59JbEYfiK6B8DHAUwjoj0ArgeQAQAhxHcBPAbgkwBeBjAM4PNxnDepRJmRRfUf2zYaV53XhrCFTL1LjilT4HSZ0pHR9s/tnt+FqxRy0V5cQxi1C5pJNlJQ2qxNfENlwIM+nxQRjlzxaMnAA4g0AakJ597iBHKjGu25PWzoLYgrq+ezAc8LAF+K41yNQJQK2jCDht8o+AuRgOAfuolxy6QJECgz1FGbx7vntp2RqqppvcRVWRv0uZkMsMvPPEo6yMmQXbfJ5+P/3Ce0pRrDJcRGu+awZEMVMM1ykRFm0NAZBdMfepBxc2fh7vnich3oZuM6N4Uq/dQlzspa3WydgIqUU9V13/DwTuxXBKaDrlv3+bgrPC+5/Jjy3rDGkIYd61vCZcSG3xAbX6lpQFF27JTkRwyE8x+bPu8eXzeDfj9fAFC75vFBLi//amFyNgMiJ+OnsyMDIYCr73seazfsijw46WIKNg1pVNlIQHBbR11ze1tDzhpDCtxCMDdI7BaCAU1n/FvO8IcJdtn63W1cGCo3jZeo/mO/77d7flfFfTjl6OkVzdO9+FcONvcxzL6y9+O/BtkgFDWwLsMVlfNm4AD2DWl0xltWqOa+H/d++EX0ggrQpnRk8H6+YDwBaXniKgRrgFVDSxn+sEYhjN/ddGasctOkiVAQInDQUBkFL37f75bX3ikz8q5S5UcOn4x//9072nTFvm0D6O3fWZaNE9Rm0aaFYlAAMyhzRbZiMnF39W0bKHPFdGYz6F0yHmC+qfs4LPjQ1MABTOequ/XCeZEKxQRQ+pz9cQXZcavhmmtq4igEa5BVQ0sZ/rDZNlGCtUGoe78KvLrqHOXrdEbB9fmqfL/3/Hq3dPuzr+zXpkhOzmaUhjmXH8PfPbCjVPmbJsJnPzobj2x/Q9lC0X/PTQKYQa4V2YoJ0LdK7Ns2gOU/315WpzCUy1cMUCbxCNX9m+kpGDM1xLL74Rp97woh6Lhs6A2ZPKuo9yPZbkqDyEe0lOEPa8CjBGuDCHtslVFwg7C6fG+VcVRtB5wZJBG0hnm4GAdwj6WTXpbl7wdl6Zi6VmSkVToPxWPIitPyBREqBVd33TYxEpvvq+q4ic7jTxpxFII1iHxES+nxh9Wrr6Yee9hjq4zC/uF8SVZBhcoI6rbfvPQ4bXBSxZLURmxsvxKvTLgYG9uvxJLURul+QXIDtr1+/egGNd0xwqbgepmYCfczi9JfAWgiaYdaMbcHOO92YPJsAOT8f97t+pm6Xw46O0W+X8LkI1rK8Ic1stXUY1cdG4BWw0X349flimczaXz2o7Ol90G1/Z96jkf3/C7rFc6S1EasytyFWal9SBEwK7UPqzJ3YUlqI6Z0lPe7DWoPadvr149OXlp3jCgpuC6u2qipwXWb1bixGy82E46orThbkrk9wNUvAr1Dzv9BRt8vBz3yLpDy9XJOoHxES7l6ohQMVTON0X9skyD08jOPMqpe9dJZTHn8ybOvo7MjgwltKRzI5cvuw9x3foFFr92Bw7APb2AaNn3ocnTPP6t0TplLI0WAbKy5pm09OmikbFsHjeCatvU49bwryraH1dgPyud30clLLz/zqAofv8t7B0fRt21A+9mbVOWGrdzWBXSDqGZsioHcnz82AmSnAu2TOKsnSdQqDz0KJkHo7vldFdk1OjqzGRwcHU/t2z+cRzaTxq0Xzisdc3P/nTjvtVXIFo11F/bhvNdWYXP/IVi45FLlwHm1YgCaSfsU29+u+AxUxjOo9aPsmk45eroygC0zvu5jWYHVUC4fmPllOviEdRvJArommMSPOAYQAZXfPrcfuPbV2l6LJS3l6mkUTGdq5x5/WIUrIJMmZFLlW1WBWf+yv+u5NSWjX3otjWD2c2tLj7vnV3aiUrlDBsU06fY36dCKbTI3HAH4y5MODzRE/mu6qfs4pby06t52z+/CtpVnSFcXFe4Rn1+3O72pzF2nipVUq3JbRZBrk2MAEWlgOWg2/AnEJKjXt20A928dKEsfJAAXLpyNtZ85viJmoArMugalb9sAZgj5DP2Diu0uKgPz70dcjmHRXrZ9WLRj4CPXlM7pxjHWbtiFC07oKrvuWy+ch5u6nXiHbU/VsIHRQMOraPPXtfuR0r4fmNjm6Bp5MPXNRw3oegmKTdnGALyfl2nvgMgkuZfuaSsd/72XBPrzZbScq6cRMJF8ULkEnn5pL27qrgw8B7UXXLthFxaIaZglcc+8RdMwQ3O9KhcQcBy+/srb+FrqPsyktzEoDsU/FS7ExQCGVx+NJcNvYoE4FGtSPegfWoz7tw7Ig+Y71mP0oS+jbex95/GB3c5jQOk7Vblfhkf0PvtA94giT3vm1jUYOHg7AMc9lEkRpnRkMDSct3Kh2Mp9BKFzbdqsLqpRER1I0ouhGlgOmg1/RKrhIzUJQtu6BGQGheD8gN0MkjWpHqzK3FUWkB0W7dh9wnKt4Xev2f++F616CgOji/AgFpW2LUltxLHP/R9kcRAgYBY5mT7IA/35xVIf/PDjK9HhGv0ibWPvO9sVPzL3GP44iJth493HS6DhVfh1D8PbZY/zBYGO9jZsW3mGdH8v/u/QBSd04emX9pYktr2z8DiNrE0NSVWavwRJGzRCMVSDKouyqycCMh/p1fc9j2/0vRD52DJfuhdbl4B32Q+gTOLBTRvsLyzGivwXsacwDQVB2FOYhr+ny7BwyaWh3oNsELqmbb1j9D24mT6q10zMvSk9vmq7S3d6E/5FXF5RQ6BzZwSm7ir8t4OiMm5h4pd3JS2836H7/nN3qZ2lX24jTveKTXpz7BlCCpdZmSunQYqhGhE2/BFQuVt+8uzrVfd/hqlJcAeTrs5shbSA+7i/sBiLR27Hnx38CU4Z/RY+ev5loa9RNgjpMn1UrxksVBpV3XYAJcMyA3sraggAvcHSDroSv24OE7BmtHLWZ+KX7+3fKZW0+Mmzr0tn2Ffd93xs/nWb+pQ4Yw8A9LN5lwYOniYddvVEQNdOL45mFzo3UpSaBONZmlrlwAiZ2+QNTEMXKo3/oDgUmRRJB6672i/BNfk7KlxQd7VfgnmqeyQxLO7Kon9kcXiDVVzWDz++EhNzb2KwcCi+k7oYj9PJ8MrkZVKE4ZHRClVUP6p0XJ1mUpB/3cb9aJreHHfswWg2z710qwYb/gjoCne8xrVaUtBhaxJM2wDmx8y0alTIBqfBD1+Dqa6Pv8iwaMea0R4cMrFNeq555yzDygdHcZW4txQkvg0XYeKxn1bfI4VhmUlvBxssj+95ODsDa/IX4ofvnugJWi/Cde/eVmYEMymUgrmTsxm8NzJaqgmoRiBU5V+vVhA2ykRDiokgWgMHT5MOCY2GSb1ZsGCB2LJlS70vQ0nftgFlgw63ylJWZJXNpAMlH9yAq580UUlCIcp1m/bXJUCrEhqGr/zddVjetr5kxNeM9qC/sFh7LtngqcpUShPhV5kvY1aqcmUxhA/gX8//tfr++TNJ4AxMK/JfRH9hMbKZNCZmUtJOWm6RleqzkxVhzb/xF9JjTWpPoyD0ongAcJunAA9Qf2/CFIBVFcl9Riar1sZJusZ9gq6PiLYKIRbo9uEZfwR0DTpOOXq6VsI4rBS0TnrA5rqB8tnbsGeG6qUa3Zq2/MnpWDy02OpcstWNqmJ4TAisGe3BP2bWoZ1Gy57rTL2P7vQmAIofZYCLyKSloU0g9PrzjqmQi8ikCf/3U079gk5lFUDFdyGRMg06o2hiLJOe1pn065PAhj8iqgYdQYqNYaWggXgaZsv0gWQGqBrdmuLyF+vuUX9hMa4XP8Kh9G75E4V8KR1Q6oLTuIhMrkd3XZ0dGSxa9ZTUVaKL5ehWaP7vgurck7Pqc1eVIKNoYhiTntaZ9OuTwIY/Bmxmoy6q2a1pV62qzOBUqT4xE5e/OEgjZ4rf6Lsc2KP0hX9s4gcxJf/7ipd40zX9ukdA+cAlu65MmvDu+2q/v+69u8+pRPm83wXpuVOE90ZGSy7HmhRfucRhFJOe1pn065PQdIY/KaJTutmoanYrU2bUHT8Im3uxdsMuaVqhKoAY9R6HFsvzuA26J89C18Iv46r/miNtuzioqETG5FnKgqTrxy6oKGIrCCcNdWP7lbgNF2HxkssBBHe88j7/3sHRiliPzcqte35XYPW16twyN14cq0Yj4jCKcXTGqiZJvz4JTWX461JWrkA1G3U7ZMmuR+Ue8s/8TVIFbe+FqW+4rvdY4jZY+ML12FQMCPqvbc1oD1Zn7ioXniumAw7+VO0iQt4pNOuifRBwZKeBYpVx+i60pY8H5vYEztK9zx+54lHpfkErN+8gOzmbQSZNZe442SQirnPHQhxGMelpnUm/PgmxFHAR0VlEtIuIXiaiFZLnP05EB4jo+eK/qtyRJDWekBXH3HbhPGxbeYa1doory0tw3AwgR3pAp6hoey9MC3RqeY/9omDDj6/UFv347/nWPzkdL55wU6mj0nD2MPSKS3HkTychpWnF6BaxDYhp8AmdOnpB3iIjQ8IUQPkrw4dyeUA4kwebhkCxF1/ZEIeQWZjOWLUk6dcnIfKMn4jSAL4D4HQAewBsJqJ+IcR/+Xb9NyHEuVHPpyNpGQ227gyVe8ibijf/xl9UNAyRLdtV79nV5vGvEkwDrrW6x7KVxcQJb8qLyjxug9I937EeePJa4DknY2TzR1bjc5s/FNiU3YuqyjiM7zZMQFs2yNpoAEU5d2zElYsfRhOnlimWDabZE4er50QALwshXgEAIroXwPkA/Ia/6lSzKXotUAUF3zvouHU6OzLSlEug0vDqYgx+94zrTsjlx5Au+slV3Z5qdY9lRm9QHKr02ZchcQkd+9z/weljX0A/ytNI0764QPn51DECW9z7+Pyj6/DFkR9jZuptvJ+dgY70jfCnlnoD/NLrUqhnmsQcvMJvvf07ccPDO60VRK2ph1EMk2KZoFz8ahOHq6cLgNeJt6e4zc/JRLSdiB4nomNUByOiZUS0hYi27N271+pCqtkUvRb4XRVTOjKAcJb4AlAafaDS8MruhRd3leB1JwDOTNi9Zyr1ylrcY5lxWzPaU6HvL3UbSDJJsjhYEoLzUhCiov+v7nw5TAjtu+1Ob0Iv3en0IIZAR+6NCmEy/+chw/9ZmzRU6Z7fVfrs3IFuKJcPdBk2LCZaQF5MROOaiDgMv2zx7Z9CPQfgQ0KI4wF8C0Cf6mBCiHVCiAVCiAXTp0+3uhAb0amk4hUI62hv0zZO9yIL8HnVOGUMDuWsffZW9zhCEw3ZCqK/sBhrMpcH+1ItcvFndmZx/XnHVDROcc/nVytdMfKF8LNAA2MUVP8hG2RNP8OgY8caq6l3AxXbbCLbgaLBicPVswfAbM/jWQAGvTsIIf7g+fsxIrqDiKYJEdDaKQSh0wQTgH+5bqKnAxQDvkCpQKezIwMhUGqkPkXhIprZmQ3lsze6xxGrGVV+6XnnLAPm36B/sSKTpADCKxMuxqCYhjWjPXgi/bGylY2s525/YTH6R8bdQ7qBNBADYxTUdF62EjN1CZnEYWKJ1SShktU2m6gBc/GjEMeMfzOAOUR0JBG1A7gIQL93ByKaQeSkURDRicXzBpdCJoxqtp6TLddNxDGzmTTOPf6wstfuH86X3EMDQzm8+/6oshVgp8LNAUK09xdxBhVp9SbLJAHQRoWSRPPq9u/jRwtfKzteR7t+HhTk0gr8fhjIDKtiJW6AX5a2q/qe+I9lEocxitUEzeaTMHs2ySbyvg9SmMIE5+JHIfKMXwgxSkRXANgAIA3gbiHETiK6rPj8dwF8GsD/JqJRADkAF4kkq8NJsMlfD1PgpNL2r8jhTxMmtbeVZvOnHD0d9/x6tzZLJV8Q6MxmMGlCW8U19fbvlL5GCGD5z7ZL358RMcygQq/e/JkklAJE+b3N4iAW/u5bAC41Fq3TDTxG3w+DfG/bDJy1G3ZJC/0I5e6/vm0DGB4Zlew5jlGsxmQ2n4TZc1A2kf99CMlnn/Bc/CjEUsAlhHgMwGO+bd/1/P1tAN+O41z1wrT1nM4AAOXuhM5sBr1LnGKuoBx+2SDinsskNfFALo/nr69MATyg0IMHnAGjt39nOONb72pGbyZJb6d8n6IhCvJ9A85n4P+c/dWxgd8PiZb/XeISzBtbhO7ia1zhv5/++nWMh3fUn6/ue+P/nvivL5tJYWImbZfVo5jNl7XBrPdn76LLJpK9DwCgNCAKTZ/V01SVu9XE1BeuGiBueHgn3j04WpaDP5TL46tF/RWTHH4/Nzy800ha2T2+arsuluCVGTBayZRS4najYr1SyxmUNzVPMuMHUDJEQX5t/0xYNrirqKh8HvNp+Y8AWV9q7X2bd8Mb08/lC8rVl+5746Ia2KZOmmAv1ayYtU8cfnO8iX0tK1nDpmDKBibAMfq9Q7FeYhLh1ouGmFY/qozI/uF8ReEVABQAXPfADus0yb5tA9r0TtPjBKV9es8XlDJYnhIHjDurYF/NGCUrxJ+aF7CM1/m1ZbEFkxWCi23l89oNu6TfE1c3yY/J9ybWojtNz+HS9dWqkjVsCuaO9VC2l2tSn74fNvyGmBrmMIVMuXwBAKyCmbq0uxQ5biST47hBVL80gYub426UMihdPgvnh3/1i3ZG3/YH7R0oHrxMvox3yU4tM0Sqz/a2C+dJA6qmBtOm8tmdteuOLXvOJAgeq2TDaSsrahvcDmpl1ze3x/nMe4fsPnsbwgaRn7wRcvcZNa1P3w+7egwxlRJWBecmtKWU/VXd48qMjAqdgbilZ56VX97dV6bHf/15x2jPV7Y9rqCerZSvSaDOw+jwftzUvxPn/PedWPi7b6H7wB6cccgM/H3uM7jn/ZMAABMz6jmR172yJLUR17Stx0zahzdpGta1XVLWptHUNUNwVlU615vKUAcFwU85erq0WVCooru5PVjTv9OpPvZ1UIuU6hqGsN835fOiaX36ftjwW2CSZaIaIAC1njpgv+xWNvvIZkIFY4MGNiOphuwUIPdO5cFtl89xFN9oaCOBv83/M9JbBVBU7uzIvYGV4rt4LzWK/sJi7B/OK7O23MH99LFflUk4z8Q+9NKd6L34GGDueAtJb2xElT4r4Nz75WceVTEAA1A2og+ib9sA7t86UGb0CcAFJ4Svd5l3zjKc/sDJyI1USfvH1G8fNoisfN3sym1NChv+KqAaILa89g5+/Ozr0tfYLrtVK4veJeNqGLZppbKuXG5RWKAk8I71wME/Vh403W6/fI6r+EbDJByEX6AzSyO4vu1HpYItlWa9+/ikh65AB0bKD+JbmfgDwbq4zOBQTlpM5s3+skWVJvz0S3ZyKF7iaqQjxSRdNGoCQZjgc5Pp+LDhryE3dTt9VONYdgf9+KLq5vtfP5TLI5MiTOnIyNP/nrzRaWvop/0Q+x+I7Q9TNVCEYCq9iyWpjY4uP9Qrse75XcBDweqdazfswuljv8I17Y47yK0ado/vxR38besXdAN8tdRUq1YhH+Tmq2jS7ql2mTzbzCDbKoYmoRI5Ztjw1xhVj96w7hmvoV+7YReuvu/5Uscnk7oDFaaSwO55/y23Wx4gzu23fl9GP0zvDCw7BUhl5AOPggIIaUmAjwilxupAwEpMMeC8iWl4tpjaeMIfnihzB82ifViVuQvIo8z4y1JGTb4jQQO866LzxiIGxTT849iF6NtmFwuqCaoB3N2uSiDITnWCyKbYKIY2YE/dINjw14G4Z0tR8spt93O3b+6/E13PrcESsQ8LxDQM0SGYCkmP27Dpcbofpn8GlnvHcSllpzoDzeRZwMh78ngDgIMijXvHTsHn0r+scPcA42JugSsxycpkWLTjH/KfwRMPvIAtr72Da9rWl7VxBIAOGsGK9vXYOvH0yKu1oMLC5WcehY0P3oEbqXzw+Qf6HlY+SAAuT5bxp7Q8OE/FrCuVWy/3jvO9qIYhTkIlcsxwOmcTECWv3Ha/mZ1ZbO6/E8du/QZmYl9J+2YSchgRvnlEtYp2ZDOwsRGgfdJ4+uDZqyu0WgSAIXwA1+QvxbpDvoSR9k7p4QfFoWbaQMV89TcxfVy9M/9F9BcWI5cfwz2/3q1s5nIY3i6psPqzuVTGXCavETRAd8/vwo2T7pcOPlfh3tp3pwuqz1BlZIkxZ39dJ+pqaQEFaSzVW4k0BDzjb2CCGnb4kc1gVS4FnWbM7IeWlfexBTCBxvB24RDkxETMpLfxfscMdJxdpaWwyQxM4i6i01aic24Pvunus2OtNJYw67ybsWmur6JVFdyb24OTfzpJao7GhFA2cyHNSkhlzIdy+fHq2CK6bKtxN9wb0nqlmfR2NF+/7J4A5vo4Ml+5asbv7q+jWjNwXcypQf3/bPgbFBNhsSkdGXS0VwqzqY4hcynIBoVC316pIZlK7+Ir+c+hv7AY2UIaN48dV9KgqcA0S0K2n2nWT5Af1zTIJ/txP/Ql4PFrgdx+/MfEafiHkc9UBGzTRFgz2lPm4wecZi5Zz0rIP/hOzmaUNR/+GI1qgD7l6Oml7YPt8sFnUBwKAUhbcQYiuyd9lztBkrGR8W1eI2jiKw+owdCicytGycrRfU9uPbYh/f9s+BsUk4Yd15+nTwEM8g+rYhFv0XTMQGU6IBHGA5f5xepAsuksSbXf8RcD238ajxaMSZBP5VoqxhBmYC9W+wK22UwaF5zQhfu3fgzIoxhYfRtv4FAMnnANFirSPQeGctKmMC7+GbpqgPZ+trLBx622dc9pk/GlvCey4LrXCJqs1CbPDpehpfv845iVq74nDer/Zx9/g6Jz7/j90yqd+LCpfrs/shw5fwvEIh00gm9m7sDG9iux4A9PyA9gWmqv2u+3v6iNFoyLwY84SyP4u/aflckm3NR9HG5eehy2/snp+POR2/Hn2QewufsZLFxyael10uypMaGU0JDFXrxd29x4gfczlHYSK8YiXKy7b9kYNndf5YxcjPvGT1sJpY6OH0rD6POvZn8Agx4LSYRn/A2G6xZQkSYqU1zUuXPCNk5fuORSbAaw4LlrpD9RIidz5Lb2O4DV9zqBVu+P0nSWpNuvlg28DesEZmAfXl11TuB+XlSDbEE4qwaVLr/fPXTK0dPx9Et7S487fV3X/J3EbK5Fik3thGsEZb5yF3cWft7t0AZwXTJZvbH3unZUxwszK/e7jOacEd/qs4bwjL+BMGnE7dfm17lzojROX7jkUlBAiTsBjjvEL7BmOktKymxK0dGrguyUsocmiqa6jlsq8TXZcX/87Otlj2Vd14Kwqh43NWxeI1im2inBnYUHSSd4Z/iPfBW4YSrQO9n5/5GvVor8KY9j+T2SiQdu/6njeqzV6jMm2PA3EKYNQ7zo3DmRm9PPOQNGy3L/stqkLZ7NftXGLzOcnTqeV+7l4B/LBjgTRVPd4OtmV7m9kddu2FWa6Qd9D/IFgUntbUjLChUkWFePz+1x7oOEUZEquZQqjODcHv3nd2CPYqAlYMEXgN4D42qfj3wV2PL98YCwGHMeP3yVgXZTCCVOlcto6w+cY1VTiTRm2NXTQNg2DAGC3Tmhi8l2rHdmOybLciAw1VKaZWFbWh8RbbWs37W0+sjKArFCviybwySGosueUrnpTGs23O5q/qpdr5pmpOrxs1dLC9jc+EFXZ7YyLRbQ+9YnzzL/3Lf+QH6M/HsGFx9CiVPlGhJjDZHC6YUNfwOhk+ztspSJjqykaKmIaZVqWQdBLGttI5UUhcc4mMZQVIOvasWQJjJqtzmzM4vTR3+Fa/KVkhFTM+3oXXFD4DG0eFtJDr9ZJtGs/Y7pfOtet1DQZx4p9TOEEqcurtEAKZxe2NXTQNg2DAHMGnWEwiYwZuOeCdtVKSJGjWa8GMQfosRQAPWKYUyIwK5p7nm+QvdIq3avydwnf6FtFercHnRc+xL6u3fiwo7v4eHiTF/7HVPdu+xUO8Mpc7cBTqtNXUwmrLswKNaT8BROLzzjbyDCyuFWRUlRp2l+2srwM/Y6CWJZp7YaKIhGlS/W9dN1c/VVWT3u4DJ55C1pGKYj92blxgj57lbfMdW9O3u12etdTvgbx6dfsf3zwOEnlYv4AeM6TmFXkO5rHrxM28O5EWDD32BUTQ7XFp3hi5JqWaeCGOvUVkM/dHd6E7on3AhM3ANMmAWkVwIwuzc6N53J92DRqqewQBwqrdqVGqlaDbpxxW7OvcX5f+sPHENMaWcwcLcHHS+MS9F9vlbN5KsEG34mHHH9eP0/vji6ePklm4HA2V6oWIhvgOvbNoC1xcY1MzuzuO3Dv8XCF64PXTEadcUwOJTDmpS8ardDZqTCDro28huPXzv++WanAkvXRRtUzr1l3NDbEKWat8ZJB9WAhEGQqF4sWLBAbNmypd6XwVSLiqYacHT1RaF8KZ3KAN13mP2wZMf0oin8se1Y5n+tf+DYNOFKdEln27PttONDsmjVUz4tfqdH7l3tl6D3G5LA7q3Hqt13quuV3W/ZPd6x3tHy8cs6pNuB879Te6MZ5r02CES0VQixQLdPLDN+IjoLwDcBpAHcJYRY5Xueis9/EsAwgL8RQjwXx7kbnSjGpuGvw1TvRZWLLptpBmUbaVwXUdxosuDwYQju0FVN3FVMf368ajebSePmc46TvyBMS0KVe+jBy5y/vbNj2Wc7NhLelRQl+6tBNXbiIrLhJ6I0gO8AOB3AHgCbiahfCPFfnt3OBjCn+O+jAP65+H9LE7U9ovc4UYx2XNdhjemPTGYcVEt1kxTTA3tiTxmVBYFVksymbivV52r6eVu7isK4MExz23WfdVjphCjCa2EbtVebGqUyR3b1ENHJAHqFEGcWH18HAEKImz373AngX4UQ9xQf7wLwcSHEG7pjN7urx12K++nqzJbp7eiQuRiymbRVymYc1xEK1XJbCjmVkaFe6yM7FRjNBbsnLJDdwyWpjVjd/n1kcdD6PKrP1VH8HIj0ecdK0Ofguk50+4VxrwQdL8hgmrqoaklM12Ti6okjj78LgPcT2FPcZrsPAICIlhHRFiLasndvpfRvMxFHI2zr/PMqXUcoVKX5MvwzsbBLcvd8Mas1ynL2n0h/DC9+5O+BybMhQHgT0/GV9z6PRY9NK9PrkaH6XO/59e7In3esBOa273ZqAkbeg9TcpNvDZcNoVxAGtR9+GY4kaOxUU0XURxyGX/ZL9S8jTPZxNgqxTgixQAixYPr06ZEvLsno2huaEofRjuM6QiH78S34n2b6PEFLcleyNzu1qCnj+XEbVN3aoiqUW7jkUvR9fAM+PHYvTnr/m3iosBgn/OEJLOz7CwhNkZSueEtG1QdpVWGX+xmqiqkAAKLYF7kNyEwa35ydGj6wG/T5mxjMuT3OSmPpOufxA8vq2zqxhnGHOIK7ewB4659nARgMsU/LEYecgmn+uc4vfMrR0/GTZ18vG4ljkXUwQZbz7y2+Ufk5dRK/gJMZ5HUNeXnyRnv/roHv1UR6YUlqY3l6pcI3rfpcVXINVR2kg/zpqtx2P2MjwCF/Clwdw0//tJWOodZpRZkYzCS1Tqxh3CEOw78ZwBwiOhLAAICLAFzs26cfwBVEdC+coO6BIP9+PalVhot7zN7+naVWexMzdoswk8FDF7wFgPu3DpT9fAjABScoMlziDj5petlqiVJFGZS9EqS5bmkcvLPxa9rWV0goyDKNVJ/rDUfuxKLX7sBhGBdceyL9MeNeyqEwKezyB4bj1MCXMbcHeOB/6fcxMZh1qhSXEiarKiSRDb8QYpSIrgCwAU46591CiJ1EdFnx+e8CeAxOKufLcNI5Px/1vNWiHhkuB0cLpb/3D+etzmeSuREUB/A/JwA8/ZIkvhL37Cjq8cJWUbqv8xYTtWXV17TlblQYMgvj4J29z5Rl+QAVBlH2uToFYWsBco41i/Zhdfv38VcfOQIL559Vem3s32FTF4R3wFbmycc4ew1q0zjynvN5hslKqkdaZw0Lw7iAy0etM1xqcb4jVzwqnX+5gRfVcxXdpOIueonreLJVA6D/AakyKNqy8sphKaR2J3nY3H8nZm5dg8OwDwWk0EaFyp1M3rPh/Yr0nVLVRsRR2AU4fn1/R7awBBXrAcFZMU1YyFWrrJ6motYZLrU4ny54axXYDZod2So7xjXbcoN0biMMoFLh84FlwA+XjF/fg5fJl/jGRh8o6xWrYsd6LHzhenTRPqQIaKOC3Anizk51GN6v0N8plTLqnDPsG+K4QV9/sxZZR7awqM7hJSjIG7bZj+13PWGw4fdRywyXvm0DSCmqUuM8n04e2Eo6WCdFHEZOuVqtFaXVuwJ49Vfj12et5a5IMw16n5JrIdnxTAyi4f0K+g73bRvAolVP4cgVj2LRqqfGU0vjbm4/twdon1S53WuMoxpQ1Tm86CYSYdI66yQdHids+H1E1VA3xfXDyjI04j6fTpPfSq9fNzsKk4NcrdaKkf2zPqOcyTpppkG9Yq2uRTLvj+l+ud/hJamN2Nh+JV6ZcDE2TbgSt334t/o+wCp/udvc3ruqikMaIS4DGvR5B00kbN9bDfPtqwWrc/qIqojoR5VdoeqbmiaqShWmTofGWKNGF3x6YJn8NUGzLdXxoqDrlGSEzyi3ZZ0U03NvcWamMqOtep+21xLD/eqe34Wu3Y/g2OfGq4a7sA9dL1yPR3cMIpc/sWz/XH4Mzz+6Dt0gSN9blBWYLkUxrowa3T2WTSSiZqYlKSAcEjb8EuLSvNdlV6j8rQUhkqG3r0KVahk2BzmKdr8KkxxvF0oHu31cNwxg/z5VKXqqIHKU++UxaAspBcD3vvI5fLHwY/wAJ1a89IsjPwZSijC/agVmYkB1KYqqdMwDu4OzcYLOAcgDyXFkpiVV58cCdvVUEV0aZd2qZatFtdw2YZjb47hmVH55l0wW+NR3g/cDxmeiQe/T77MG5D7ks1fHe792rAce+lJgDGNm6m2r7cqm5KZuGp0PXVfta+PykZ1j6feAa1+tvPY43DRJ+q6HhGf8VUSXXXHrhfOq0wS9XiStOcW5t1RWAM85wwlU+q9Pla7ox/V1A/L3qZpNnne7OjUwrvv1+LVOZWwA72dnIDuWrvjevZ+dgY6cpKZSFdewcdOoVim6lZaty8d05RiHmyZp3/UQsOGvIjo5BZtYQlI0+wOphtsmCqbXEyT/4OIu5VXHtfVZe47Tt20Aax/bhQX3Xofr2n+GP8U+kI1BMUlDzWTRcfaNuHnsuIrvU0f6RrtCuDgMaFABlvdYcVWMx+WmSdp33RI2/FUkSE7BJJZQN638VqJsBqcyRBpft0sYY7hjPYYfX4klw2/iz8UkfCDzPtoxWnxdXLoxVGYsuyH77ljOYoMMaFj/v5/eyZXbotyXGsoiJBmu3K0yUWfrddPKryfVbEYRdOxHvgqx5ftlXn8BgI78GPDX/fpj21aBmlSe6l7vZfWR8ll/dqrj644Dfy/jg38s76rlVskCcuN6/MWVrjbAPBDvJ2x1bY2andSLmrVeZNREzRCqm1Z+vaimWqLBsYd3PoYO38sIwPCb/69iewW2s8mgNpEuqhVDRVP5FACPHEQq4wSR48B/73LvOFr62amVjexvPVbu8tryfc978sQ/whI2fbLB3TRxwFk9Cafpsn+CsMm6sK36NDj2xNyb0peqtpdhWwVqarhk/md/Vo2rd+/tPWDaoN4E2b0bG3GqZr1SGTad0dx7HzYNsoHSJ5MGz/gTThya/Q2FqZ9cNnt/YJmTG+5vvVeaGWsqU4sMFg7FrFSlguZg4VDMKjuWwk1gM5s0Ke5SrRh0hjgu144XE50m057H/tcvXWf/2hb0y8cJz/gTjpWkQjNgqt+j0uMBynPKy2bGwee8q/0SDIv2sqeHRTvuar8kfo0WST74QZHGO+IQFITTqrFixeCucgwGsUBsVkxBn4up20r2ev9KKTsVSGly/JPQJrHB4Rl/BGrZsKVpDb0fUz95kIHzunCCDNLUPysa0z34SuaD+PnYX+DU1POYSW9jUByK23ARFp+zDHh8sT5d0zZoWHxuz8+vK51rzWgP+guLARSlsed6pLFNZtUm7o8d68t7EQDBsZSgzyWMv937ev9KyX+NqircJg7SVhM2/CHhNMsqYVocY+ImMTVGrz4Dd7UwJf97fDq9HyvyX0R/YTEIwF+edDi605vUufJewbGgoLTEWF3Y8T2j9pmBs2pTOWHV4BFUc+Beg+xzsdUkCtLlD3KZJallYgPC6Zwhack0yyRhNPstVp2GEGwbFSmkUMCgmIa72i9B76T71cfRncebcqho/rL5uBvwuc0fqojj3Lz0OGfACWpn6J7HZMZrFHwl+xm0qY8/Mwk477boxrkJG6jEBTdiqSItl2aZNMr8woBUSvm0laEDgG1UQIqAWal9uCZ/h95YzjnDzOeuyCpa+LtvyeM46U3lMQUVrrGLIllQRoj4hdRP3y7ZsSDZ5sMk9tAECpn1hF09IdHJMTA1wusO0Pl7H7wUECqDo5Ai9lDRHN1L+ySnEbsKr89dY6ykcZxbDQKmttktNi6ZqHo5sll50DFNXThNoJBZT3jGH5JaNWxhDNE101AafTgqnn6lRWMISE9QG2e/UbbtOKadvVp0wvJy2kqnsMuUKDPoMLNy0zqOJlDIrCds+EPScmmWjYxKYZLSjoKnx0VRIJufhHCqVlW4BstNKx15r3IfnbFSDhSzndx3wKldsGlZOLcHmPABs3111xDltbpjmg4Wc3scCQhX2pnSzuOgQbDBe+XGBbt6ItBSaZaNisrgAo4ssE82OSUNUqo6UxkEjw/sBvouB4gqZZODMltUKZRzzgidQYS5PfrByos7KJmmTfr3m/pnlUHpoFm5qQtnx3rHxeZKO4sx5/HhJ0V3I7UAPONnmhf3h66TLPa7EWSyCzJ3kDd4HOQqKuTlWvntk+TFWUFNXH77i2B3iK7YTDXjzk6tPBdgVrQmO9+rv0L5gEnBs3JTF06YhipN0Cs3LnjGzzQvYUXQZDnkpaYuux23grcj13m3m6Vc6s5r08TFpL+xzsidttJZhXiVNV38s3mV4Jo/QGt0r4UzaOkwreMIEz/gTKASkWb8RDSViJ4got8W/5+i2O+/iegFInqeiJKZmM80H8Y/aBHs753bMz4bdd0LXleBG1hWxRNkeGfeNrNRE995kJEjSbtJt7ew9z4oj7O7fIVimilk8pnoAvUuYeIHYV7TpER19awA8KQQYg6AJ4uPVZwihJgXVFjAMLFh84M2yVs3Mc4mrh+XkffGA7822jsm7hCdkXvyRnWbRv/70d1Db49fU+IysmGyejgTqERUw38+gB8W//4hgO6Ix2OY+JD90F0NeRlB/l4TV0FQYZmX3DuOy+WhL6n3oVRlBoqJ/LPOyAXNur3Pn7ZS/R4MevxKzx8HthLYYV/TpESSbCCiISFEp+fxfiFEhbuHiF4FsB/O1OBOIcQ6zTGXAVgGAIcffvgJr732WujrY5gySWZKO24aba9XclwMMnQyAaetrPRLmzZxN8XtcGUjoyDzlQfJNlAa+NR3x88ja38YxOTZ6ub2TFUxkWwINPxE9EsAMyRPfR3ADw0N/0whxCARfRDAEwC+LIR4JugNJFmrh6kDYdUYbVM0ZUbcVd9UtRTc/tPK7WFkioOIQ4vGRFfHO8jYNFcBoB08maoTS+tFIcQnNCf4PREdJoR4g4gOA/CW4hiDxf/fIqIHAZwIINDwM0yJKDnYSu1+n/E3zZH3DgpzzgC2/mA84OuSz42vMOIkjgyUkpTFZerr87q9VHUQKlowWNpoRPXx9wP46+Lffw3gIf8ORDSJiD7g/g3gDAAtLZ/Xt20Ai1Y9hSNXPIpFq55C37aBel9S8omSg62crQr7HHlvxsmcM4Atd6uNpxgLDvSmMk7cwUsmq45DxGFU3ZVT0KDkDnq6Ogg/LRosbTSiGv5VAE4not8COL34GEQ0k4geK+7zpwA2EtF2AP8J4FEhxL9EPG/D4ur4DwzlIDCu48/GP4AoOdhuWb9suz9t0Kb145a7EaiaWVEM9oXyx913AOd/p3zb8RfLj2eqt6+TJDDpSObi1iuYkp3assHSRiNSAZcQ4m0Ap0m2DwL4ZPHvVwAcH+U8zcTaDbvKdNcBIJcfw9oNu1j+QUcUNUbdjDzseZ68EVqj7xppWWcpfxGTX2VU5n8PkneQvVblpjIy5mTupjK5NiZRsGRDjWEd/5BEycFWFVXJtpueR7fSoLR85mvSs1dlmP3yDv7j3nqs02g+yB1mU9SmE7dzVydLv+c0d2ej31Cw4a8xKr1+1vE3oM1zj2zcCjaDhmmut3KlQeWpkF5M4hS2Li0T1433taYxAje7SXbfPvVdfVUtk3jY8NcY1vEPgUxsbdRihWRbuGMiGSCt0CVH0E11XBOjbisrYOK68b7WpLI4lXEyeR5Y5gy22alo9YKnZoNF2mqM68dfu2EXBodymNmZxfIzj2L/vg7dTDlsdyhTVLUDpmJiXrJT5BkyWU/pi0qKWeXSMgnSzjlj/G9VSqpbaJWdAoy8O36duXec8y9dVx+DH7Z2g9HChr8OsI6/JfVSVQwKloYdTHTYDCiPfNXsmLpgsp9bj60cnGwH2bhg/fyqwYafST716q8ax0rDi6oBin+7yYBSSic14MBux6CbzJaTIF3sldnwU69BqMlgHz+TfKJk9ERptRe3EYxTFjgondTPgd1O1s/qI/X3oN7SxbbBaiYUbPiZ5BNWVdEkfVJH3EYwTlngsMZPprnvpd7SxbbBaiYUbPiZxsAk08ZP1FZ7cRpB133havgATrZMW9a+YTqgN34quQcX3T2ot3Rx0IDGkhCxwD5+pnkxcdXoskbCZO7I2LHe0dx39evFGJBKAwf/ON7+0DZwKcv+cdNJz70lWFFTZ2BNYwzVyLZRxXOA8doC9u9Hhg0/07wEBYVNskbiyNx5/NrKpiWFMQASRU/TwGXQoCQdGDxkpxQHhxCGu5rZNqoBzSaewQQSqRFLtWE9fiYSKv38IJ35ODTvvdg2Muk9EM95d6x3Bh1/emYq4/TcLRuMisbVZFZd7ftWltUjkc7mIjItJnr87ONnmpcgf3USUhcrIDtfv465PY6OztLvld+DCR+QtE0sGleTAHi175sbz5k8GxUzfZsYDaOEXT1Mc6Nz1cRdH6Dye2enWmjaC727J4xv3X8Pejv1+we5nGpVV5HIgbk54Bk/07rEnbWjSh09e7XjXjHFSJAtRHqqi4mB1hnXWqV81rumoIlhw8+0LnGmLgZV+XbfEZxm6WIjyBbG9WEi1KYzrrVK+ax3TUETw64eprWJS28nyC0xt8cx0EEuH60gW0yuj7KMIEUANci4VkOnSHYOgEXaqgAbfoaJAxO/t9ZAU7Bhi9O37jf+bmP4pOXK12KAaUHY1cMwcTDnDDgzZw/+mbPSZz3brCK5ajEJjDeGT5LRZ6oGG36GicqO9cD2n6I89ZCcpuleIxrVcNcqJsE0PezqYRgZNmmTUmExIdfBd/cP67OuVUyCaWrY8DOMH1tJAqURlfjjk+KzrlePAyYRsKuHYfzYukF0jdfjqsKNG06VbGnY8DOMH1s3yGkrURHYBVCqwk0i9ZZfZupKJMNPRJ8hop1EVCAipSgQEZ1FRLuI6GUiWhHlnAxTdWwrRuf2QKkemWSfeZgeB0xTEHXG/yKApQCeUe1ARGkA3wFwNoAPA/gsEX044nkZpnqEcYNMnq3Yzj5zJnlEMvxCiN8IIXYF7HYigJeFEK8IIUYA3Avg/CjnZZiqEsYNUgufeZT+wQzjoRZZPV0AvOkDewB8VLUzES0DsAwADj/88OpeGcOosM2+qba8QDWbnzAtR6DhJ6JfApgheerrQoiHDM6hiHrJEUKsA7AOcBqxGByfYZJBNVM1H79WLwLHMBYEGn4hxCcinmMPAK8DdBaAwYjHZJjoVKtvbNzsWK8Wd0ty8JhJLLVw9WwGMIeIjgQwAOAiABfX4LwMo6aRXCe6lFAOHjMhiJrO+Ski2gPgZACPEtGG4vaZRPQYAAghRgFcAWADgN8AWC+E2BntshkmIo2kVRPUFIVhLIk04xdCPAjgQcn2QQCf9Dx+DMBjUc7FMLHSSFo1KnmF7NTy1UmjuK6YusOVu0xr0kht/VSpomevHn8cV1tGpiVgw8+0Jo2kVWNSV9BIrium7rA6J9OaNFpbv6BUUVvXFbuFWho2/EzrkhSJ5DiwkVlupIwmpiqwq4dhmgEb1xW7hVoeNvwM0wzY6As1UkYTUxXY1cMwzYKp64q7b7U8bPgZphXwBnOzU4B0OzA2Mv58UjOamKrArh6GaXb8Of65dwAhnAIw7r7VkvCMn2GaHVkwt5AH2icB175an2ti6grP+Bmm2eFgLuODDT/DNDuNJE/B1AQ2/AzT7DSSPAVTE9jwM0yzE6aHMNPUcHCXYVqBZpKnYCLDM36GYZgWgw0/wzBMi8GGn2EYpsVgw88wDNNisOFnGIZpMUgIUe9rUEJEewG8Vu/rqCPTAOyr90UkCL4f4/C9GIfvxTjTAEwSQkzX7ZRow9/qENEWIcSCel9HUuD7MQ7fi3H4Xoxjei/Y1cMwDNNisOFnGIZpMdjwJ5t19b6AhMH3Yxy+F+PwvRjH6F6wj59hGKbF4Bk/wzBMi8GGn2EYpsVgw59wiOgzRLSTiApE1JIpa0R0FhHtIqKXiWhFva+nXhDR3UT0FhG9WO9rqTdENJuIniai3xR/H1+p9zXVCyKaSET/SUTbi/fihqDXsOFPPi8CWArgmXpfSD0gojSA7wA4G8CHAXyWiD5c36uqGz8AcFa9LyIhjAL4mhDifwA4CcCXWvh7cRDAqUKI4wHMA3AWEZ2kewEb/oQjhPiNEGJXva+jjpwI4GUhxCtCiBEA9wI4v87XVBeEEM8AeKfe15EEhBBvCCGeK/79RwC/AdBV36uqD8Lh3eLDTPGfNmuHDT+TdLoA7PY83oMW/YEzcojoCADzAfy6zpdSN4goTUTPA3gLwBNCCO294A5cCYCIfglghuSprwshHqr19SQMkmzjHGQGAEBEhwC4H8BVQog/1Pt66oUQYgzAPCLqBPAgER0rhFDGgtjwJwAhxCfqfQ0JZg+A2Z7HswAM1ulamARBRBk4Rv8nQogH6n09SUAIMURE/wonFqQ0/OzqYZLOZgBziOhIImoHcBGA/jpfE1NniIgAfB/Ab4QQt9T7euoJEU0vzvRBRFkAnwDwku41bPgTDhF9ioj2ADgZwKNEtKHe11RLhBCjAK4AsAFOAG+9EGJnfa+qPhDRPQD+A8BRRLSHiL5Q72uqI4sA/BWAU4no+eK/T9b7ourEYQCeJqIdcCZKTwghHtG9gCUbGIZhWgye8TMMw7QYbPgZhmFaDDb8DMMwLQYbfoZhmBaDDT/DMEyLwYafYRimxWDDzzAM02L8f9cfvh6jvhk3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X[y == 0, 0], X[y == 0, 1])\n",
    "plt.scatter(X[y == 1, 0], X[y == 1, 1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "automated-vehicle",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "1. max_depth = None：决策树的最高深度；（如果不做设置，按其它参数条件结束划分）\n",
    "2. max_features = 'auto'：节点划分时，进行所搜的特征的种类数；（默认搜索全部特征）\n",
    "3. max_leaf_nodes = None：划分结束时，模型最多的叶子数；（如果不做设置，按其它参数条件结束划分）\n",
    "4. min_impurity_decrease = 0.0："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "chubby-works",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(n_estimators=500, n_jobs=-1, oob_score=True,\n",
       "                       random_state=666)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf_clf = RandomForestClassifier(n_estimators=500, random_state=666, oob_score=True, n_jobs=-1)\n",
    "rf_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "respiratory-question",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.896"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_clf.oob_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "initial-affiliate",
   "metadata": {},
   "source": [
    "#### scikit-learn 中：Extra-Trees（极其随机的森林）\n",
    "\n",
    "+ 特点：决策树子模型在节点划分时，使用随机的特征和随机的阈值；\n",
    "    也就是说，节点划分时，选择的特征及对应的特征值不是搜索比较所得，而是随机抽取一个特征，再从该特征中随机抽取一个特征值，作为该节点划分的依据；\n",
    "+ 理论支撑：只要子模型的准确率大于 50%，并且集成的子模型的数量足够多，最终整个集成系统的准确率就能达到要求；\n",
    "+ 优点：提供额外的随机性，抑制过拟合；并且具有更快的训练速度；\n",
    "+ 缺点：增大了 bias（偏差）；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "patent-commander",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExtraTreesClassifier(bootstrap=True, n_estimators=500, oob_score=True,\n",
       "                     random_state=666)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "et_clf = ExtraTreesClassifier(n_estimators=500, bootstrap=True, oob_score=True, random_state=666)\n",
    "et_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "surgical-british",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.896"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_clf.oob_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "verbal-secondary",
   "metadata": {},
   "source": [
    "## Regresstion Problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "buried-trade",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import ExtraTreesClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "temporal-chassis",
   "metadata": {},
   "source": [
    "##  Boosting\n",
    "### Ada Boosting\n",
    "\n",
    "![title](attachment/20.png)\n",
    "\n",
    "第一次进行学习得到第一个子模型，根据第一子模型的预测结果重新定义数据集——将预测错误的点（深色点）划分较高权重，将预测成功的点（浅色点）划分较低权重；第二次进行学习时，使用上一次学习后被重新定义的数据集进行训练，再根据模型的预测结果重新定义数据集——将预测错误的点（深色点）划分较高权重，将预测成功的点（浅色点）划分较低权重；以此类推，最终得到 n 个子模型；\n",
    "\n",
    "\n",
    "+ 特点：\n",
    "    1. 每一次生成的子模型都在想办法弥补上一次生成的子模型没有成功预测到的样本点，或者说是弥补上一子模型所犯的错误；也可以说，每一个子模型都在想办法推动（Boosting）整个基础系统，使得整个集成系统准确率更高；\n",
    "    2. 每一个子模型都是基于同一数据集的样本点，只是样本点的权重不同，也就是样本对于每一个子模型的重要程度不同，因此每份子模型也是有差异的；最终以所有子模型综合投票的结果作为 Ada Boosting 模型的最终学习结果；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "contained-quebec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "understood-split",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X, y = datasets.make_moons(n_samples=500, noise=0.3, random_state=42)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "negative-invite",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cardiovascular-macedonia",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AdaBoostClassifier(base_estimator=DecisionTreeClassifier(max_depth=2),\n",
       "                   n_estimators=500)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ada_clf = AdaBoostClassifier(DecisionTreeClassifier(max_depth=2), n_estimators=500)\n",
    "ada_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bibliographic-chicago",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.872"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ada_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "right-shelf",
   "metadata": {},
   "source": [
    "### Gradient Boosting\n",
    "\n",
    "+ 思想\n",
    "\n",
    "+ 解释（一）：\n",
    "    1. 使用整体的数据集训练第一个子模型 m1，产生错误 e1（m1 模型预测错误的样本数据）；\n",
    "    2. 使用 e1 数据集训练第二个子模型 m2，产生错误 e2；\n",
    "    3. 使用 e2 数据集训练第三个子模型 m3，产生错误 e3；\n",
    "    \n",
    "     。。。\n",
    "     \n",
    "    4. 最终的预测结果是：m1 + m2 + m3  +  。。。（回归问题）\n",
    "    \n",
    "    ![title](attachment/21.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "stuffed-strike",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingClassifier(max_depth=2, n_estimators=30)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "gd_clf = GradientBoostingClassifier(max_depth=2, n_estimators=30)\n",
    "gd_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "middle-generator",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.912"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gd_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collaborative-undergraduate",
   "metadata": {},
   "source": [
    "## Stacking\n",
    "![title](attachment/22.png)\n",
    "\n",
    "#### 训练 Stacking 的集成分类器\n",
    "\n",
    "+ 思路：如下图所示；\n",
    "\n",
    "    1. 将训练数据集分割为 3 份（有几层就将 X_train 分成几份）：X_train_1、X_train_2、X_train_3，使用  X_train_1 训练出 3 个模型（训练方式可以有多种）；（得到第一层的 3 个模型）\n",
    "    2. 将 X_train _2 数据集传入 3 个模型，得到 3 组预测结果，将 3 组预测结果与 X_train_2 数据集中的  y  值一起组合成一个新的数据集 X_train_new_1；（得到第一个新的数据集：X_train_new_1）\n",
    "    3. 使用 X_train_new_1 数据集再训练出 3 个模型，为第二层的模型；（得到第二层的 3 个模型）\n",
    "    4. 将 X_train _3 数据集传入第二层的 3 个模型，得到 3 组预测结果，再将 3 组预测结果与 X_train_3 数据集中的  y  值一起组合成一个新的数据集 X_train_new_2；（得到第二个新的数据集：X_train_new_2）\n",
    "    5. 使用 X_train_new_2 训练出一个模型，作为最高层的模型；（得到第三层的 1 个模型）\n",
    "    \n",
    "    ![title](attachment/23.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}