notebooks/supervised/classification/ensemble/classification_bagging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "actual-presentation",
   "metadata": {},
   "source": [
    "## Classification with bagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "interracial-adrian",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn import tree, ensemble, datasets, metrics, model_selection, preprocessing, pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "competent-participant",
   "metadata": {},
   "source": [
    "Load the data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "wooden-linux",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _wine_dataset:\n",
      "\n",
      "Wine recognition dataset\n",
      "------------------------\n",
      "\n",
      "**Data Set Characteristics:**\n",
      "\n",
      "    :Number of Instances: 178 (50 in each of three classes)\n",
      "    :Number of Attributes: 13 numeric, predictive attributes and the class\n",
      "    :Attribute Information:\n",
      " \t\t- Alcohol\n",
      " \t\t- Malic acid\n",
      " \t\t- Ash\n",
      "\t\t- Alcalinity of ash  \n",
      " \t\t- Magnesium\n",
      "\t\t- Total phenols\n",
      " \t\t- Flavanoids\n",
      " \t\t- Nonflavanoid phenols\n",
      " \t\t- Proanthocyanins\n",
      "\t\t- Color intensity\n",
      " \t\t- Hue\n",
      " \t\t- OD280/OD315 of diluted wines\n",
      " \t\t- Proline\n",
      "\n",
      "    - class:\n",
      "            - class_0\n",
      "            - class_1\n",
      "            - class_2\n",
      "\t\t\n",
      "    :Summary Statistics:\n",
      "    \n",
      "    ============================= ==== ===== ======= =====\n",
      "                                   Min   Max   Mean     SD\n",
      "    ============================= ==== ===== ======= =====\n",
      "    Alcohol:                      11.0  14.8    13.0   0.8\n",
      "    Malic Acid:                   0.74  5.80    2.34  1.12\n",
      "    Ash:                          1.36  3.23    2.36  0.27\n",
      "    Alcalinity of Ash:            10.6  30.0    19.5   3.3\n",
      "    Magnesium:                    70.0 162.0    99.7  14.3\n",
      "    Total Phenols:                0.98  3.88    2.29  0.63\n",
      "    Flavanoids:                   0.34  5.08    2.03  1.00\n",
      "    Nonflavanoid Phenols:         0.13  0.66    0.36  0.12\n",
      "    Proanthocyanins:              0.41  3.58    1.59  0.57\n",
      "    Colour Intensity:              1.3  13.0     5.1   2.3\n",
      "    Hue:                          0.48  1.71    0.96  0.23\n",
      "    OD280/OD315 of diluted wines: 1.27  4.00    2.61  0.71\n",
      "    Proline:                       278  1680     746   315\n",
      "    ============================= ==== ===== ======= =====\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "    :Class Distribution: class_0 (59), class_1 (71), class_2 (48)\n",
      "    :Creator: R.A. Fisher\n",
      "    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n",
      "    :Date: July, 1988\n",
      "\n",
      "This is a copy of UCI ML Wine recognition datasets.\n",
      "https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data\n",
      "\n",
      "The data is the results of a chemical analysis of wines grown in the same\n",
      "region in Italy by three different cultivators. There are thirteen different\n",
      "measurements taken for different constituents found in the three types of\n",
      "wine.\n",
      "\n",
      "Original Owners: \n",
      "\n",
      "Forina, M. et al, PARVUS - \n",
      "An Extendible Package for Data Exploration, Classification and Correlation. \n",
      "Institute of Pharmaceutical and Food Analysis and Technologies,\n",
      "Via Brigata Salerno, 16147 Genoa, Italy.\n",
      "\n",
      "Citation:\n",
      "\n",
      "Lichman, M. (2013). UCI Machine Learning Repository\n",
      "[https://archive.ics.uci.edu/ml]. Irvine, CA: University of California,\n",
      "School of Information and Computer Science. \n",
      "\n",
      ".. topic:: References\n",
      "\n",
      "  (1) S. Aeberhard, D. Coomans and O. de Vel, \n",
      "  Comparison of Classifiers in High Dimensional Settings, \n",
      "  Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of  \n",
      "  Mathematics and Statistics, James Cook University of North Queensland. \n",
      "  (Also submitted to Technometrics). \n",
      "\n",
      "  The data was used with many others for comparing various \n",
      "  classifiers. The classes are separable, though only RDA \n",
      "  has achieved 100% correct classification. \n",
      "  (RDA : 100%, QDA 99.4%, LDA 98.9%, 1NN 96.1% (z-transformed data)) \n",
      "  (All results using the leave-one-out technique) \n",
      "\n",
      "  (2) S. Aeberhard, D. Coomans and O. de Vel, \n",
      "  \"THE CLASSIFICATION PERFORMANCE OF RDA\" \n",
      "  Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of \n",
      "  Mathematics and Statistics, James Cook University of North Queensland. \n",
      "  (Also submitted to Journal of Chemometrics).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "wine = datasets.load_wine()\n",
    "print(wine.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "lightweight-shift",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = pd.DataFrame(wine.data, columns=wine.feature_names)\n",
    "y = wine.target"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "requested-vietnamese",
   "metadata": {},
   "source": [
    "Stratify the data by the target label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "friendly-sleeping",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train samples: 89\n",
      "test samples 89\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAVeElEQVR4nO3dfZBV9Z3n8fdX6LEhMkCgJShBGMZypNS0sWWSMTXxIUQ0M0El6pr4QNbddtaHMtkJpZlKnJjKbNwyMepuYoopEV0lRiVKJtEd1CVrfIrTMIyiENGMjq1GOrAoqDhCvvvHvWDL423sc4/Neb+qbvU5v3POPd9brR9+/Tvn/k5kJpKk6tir7AIkSc1l8EtSxRj8klQxBr8kVYzBL0kVM7jsAhoxevTonDBhQtllSNKAsnjx4t9lZtvW7QMi+CdMmEBXV1fZZUjSgBIRz2+v3aEeSaoYg1+SKsbgl6SKGRBj/JLUV2+//Tbd3d1s2LCh7FIK19rayrhx42hpaWlof4Nf0h6pu7ubYcOGMWHCBCKi7HIKk5msXr2a7u5uJk6c2NAxDvVI2iNt2LCBUaNG7dGhDxARjBo1qk9/2Rj8kvZYe3rob9bXz2nwS1LFGPyStAtr167lBz/4Qdll9Bsv7up95YhZN5VdQp8svvLssktQE2wO/vPPP7/sUvqFPX5J2oVLL72UZ599lvb2dk499VTuuuuuLdu+8IUvsGDBAubOncv06dM5+uijOfDAA7n88su37HPzzTczZcoU2tvbOe+889i0aVMJn+IdhQV/RLRGxGMR8S8R8WREXF5vnxsR/xoRS+uv9qJqkKT+cMUVVzBp0iSWLl3KhRdeyNy5cwF49dVXefjhh/nMZz4DwGOPPcb8+fN5/PHHuf322+nq6mL58uX8+Mc/5qGHHmLp0qUMGjSIW265pcRPU+xQz1vAsZm5PiJagAcj4p76tlmZeUeB55akQnzyk5/k/PPPp6enh/nz5zNjxgwGD65F6dSpUxk1ahQAp5xyCg8++CCDBw9m8eLFHHnkkQC8+eab7LvvvqXVDwUGf9ae4r6+vtpSf/lkd0kD3tlnn83NN9/Mrbfeyg033LClfevbKiOCzOScc87h29/+drPL3KFCL+5GxCBgMfDHwPcz81cR8V+Av4uIy4D7gUsz863tHNsJdAKMHz++yDKl3fZv3zy07BL6bPxlT5RdwoAzbNgw1q1bt2V95syZTJkyhQ996ENMnjx5S/u9997LmjVrGDJkCHfddRdz5sxh6NChTJ8+nS9/+cvsu+++rFmzhnXr1nHAAQeU8VGAgi/uZuamzGwHxgFTIuIQ4KvAnwBHAh8ELtnBsbMzsyMzO9ratnmOgCQ1zahRozjqqKM45JBDmDVrFmPGjOHggw/mi1/84rv2mzJlCjNmzOCwww5jxowZdHR0MHnyZL71rW/x6U9/msMOO4ypU6fy8ssvl/RJappyO2dmro2IRcC0zPxOvfmtiLgB+EozapCk92LevHlblt944w1WrlzJGWec8a59xo0b9647fjY7/fTTOf3004susWFF3tXTFhEj6stDgKnAiogYW28L4CRgWVE1SFJ/u++++zj44IO56KKLGD58eNnl7JYie/xjgRvr4/x7Abdl5s8i4v9ERBsQwFLgrwqsQZL61ac+9Smef37bJxrOnDmTmTNnNr+g3VDkXT2PA4dvp/3Yos4pSdo1v7krSRVj8EtSxRj8klQxzs4pqRL6e+bXRmZmXbt2LfPmzevzrJ4nnngi8+bNY8SIEbtZ3c7Z45ekguxoHv+NGzfu9Li77767sNAHe/ySVJje0zm3tLTQ2trKyJEjWbFiBU8//TQnnXQSL7zwAhs2bODiiy+ms7MTgAkTJtDV1cX69es54YQT+MQnPsHDDz/M/vvvz4IFCxgyZMh7qssevyQVpPd0zldeeSVLlizhmmuu4emnnwZgzpw5LF68mK6uLq699lpWr169zXusXLmSCy64gCeffJIRI0Ywf/7891yXPX5JapIpU6YwceLELevXXnstd955JwAvvPACK1eu3DKt82YTJ06kvb0dgCOOOILnnnvuPddh8EtSk3zgAx/YsvyLX/yC++67j0ceeYShQ4dy9NFHs2HDhm2O2XvvvbcsDxo0iDfffPM91+FQjyQVZOvpnHt79dVXGTlyJEOHDmXFihU8+uijTavLHr+kSmjk9sv+1ns65yFDhjBmzJgt26ZNm8YPf/hDDj74YA466CA+9rGPNa0ug1+SCtR7Oufe9t57b+65557tbts8jj969GiWLXtnAuOvfKV/ZrF3qEeSKsbgl6SKMfglqWIMfkmqGINfkirG4JekivF2TkmV8G/fPLRf32/8ZU/scp/dnZYZ4Oqrr6azs5OhQ4fuTnk7ZY9fkgqyo2mZG3H11Vfzxhtv9HNFNYX1+COiFXgA2Lt+njsy828jYiJwKzAKWAyclZn/XlQdklSW3tMyT506lX333ZfbbruNt956i5NPPpnLL7+c119/ndNOO43u7m42bdrE17/+dV555RVeeukljjnmGEaPHs2iRYv6ta4ih3reAo7NzPUR0QI8GBH3AP8V+F5m3hoRPwTOBa4rsA5JKsUVV1zBsmXLWLp0KQsXLuSOO+7gscceIzP57Gc/ywMPPEBPTw/77bcfP//5z4HaHD7Dhw/nqquuYtGiRYwePbrf6ypsqCdr1tdXW+qvBI4F7qi33wicVFQNkvR+sXDhQhYuXMjhhx/ORz/6UVasWMHKlSs59NBDuffee7nkkkv45S9/yfDhwwuvpdCLuxExiNpwzh8D3weeBdZm5ubnjnUD++/g2E6gE2D8+PFFllm6/r7oVLRGLmpJerfM5Ktf/SrnnXfeNtuWLFnC3Xffzde+9jWOO+44LrvsskJrKfTibmZuysx2YBwwBfiTPhw7OzM7MrOjra2tqBIlqTC9p2U+/vjjmTNnDuvX1wZCXnzxRVatWsVLL73E0KFDOfPMM5k1axZLlizZ5tj+1pTbOTNzbUQsAj4OjIiIwfVe/zjgxWbUIKnayvhLtfe0zCeccAKf//zn+fjHPw7APvvsw80338wzzzzDrFmz2GuvvWhpaeG662qXPDs7O5k2bRr77bffwLm4GxFtwNv10B8CTAX+O7AI+By1O3vOARYUVYMklW3raZkvvvjid61PmjSJ448/fpvjLrroIi666KJCaiqyxz8WuLE+zr8XcFtm/iwingJujYhvAf8MXF9gDZKkrRQW/Jn5OHD4dtp/Q228X5JUgj1uyoYjZt1Udgl9duewsiuQ9kyZSUSUXUbhMrNP+ztlg6Q9UmtrK6tXr+5zKA40mcnq1atpbW1t+Jg9rscvSQDjxo2ju7ubnp6eskspXGtrK+PGjWt4f4Nf0h6ppaWFiRMnll3G+5JDPZJUMQa/JFWMwS9JFWPwS1LFGPySVDEGvyRVjMEvSRVj8EtSxRj8klQxBr8kVYzBL0kVY/BLUsUY/JJUMQa/JFWMwS9JFWPwS1LFFBb8EfHhiFgUEU9FxJMRcXG9/RsR8WJELK2/TiyqBknStop8AtdG4K8zc0lEDAMWR8S99W3fy8zvFHhuSdIOFBb8mfky8HJ9eV1ELAf2L+p8kqTGNGWMPyImAIcDv6o3XRgRj0fEnIgYuYNjOiOiKyK6qvCwZElqlsKDPyL2AeYDX8rM14DrgElAO7W/CL67veMyc3ZmdmRmR1tbW9FlSlJlFBr8EdFCLfRvycyfAGTmK5m5KTN/D/w9MKXIGiRJ71bkXT0BXA8sz8yrerWP7bXbycCyomqQJG2ryLt6jgLOAp6IiKX1tr8BzoiIdiCB54DzCqxBkrSVIu/qeRCI7Wy6u6hzSpJ2rcgev6QKOWLWTWWX0Gd3Druy7BL6bPxlT7zn93DKBkmqGINfkirG4JekijH4JaliDH5JqhiDX5IqxuCXpIox+CWpYgx+SaoYg1+SKsbgl6SKMfglqWIMfkmqGINfkirG4JekijH4JaliDH5JqpiGgj8i7m+kTZL0/rfTRy9GRCswFBgdESN55xm6fwjsX3BtkqQC7OqZu+cBXwL2AxbzTvC/BvzPnR0YER8GbgLGAAnMzsxrIuKDwI+BCcBzwGmZ+f92r3xJUl/tdKgnM6/JzInAVzLzjzJzYv31kczcafADG4G/zszJwMeACyJiMnApcH9mHgjcX1+XJDXJrnr8AGTm/4iIP6PWSx/cq/2mnRzzMvByfXldRCynNjw0HTi6vtuNwC+AS/peuiRpdzQU/BHxv4BJwFJgU705qQ3lNHL8BOBw4FfAmPo/CgC/pTYUtL1jOoFOgPHjxzdyGklSAxoKfqADmJyZ2dcTRMQ+wHzgS5n5WkRs2ZaZGRHbfc/MnA3MBujo6OjzeSVJ29foffzLgA/19c0jooVa6N+SmT+pN78SEWPr28cCq/r6vpKk3ddoj3808FREPAa8tbkxMz+7owOi1rW/HliemVf12vRT4BzgivrPBX0tWpK0+xoN/m/sxnsfBZwFPBERS+ttf0Mt8G+LiHOB54HTduO9JUm7qdG7ev5vX984Mx/knfv+t3ZcX99PktQ/Gr2rZx21u3gA/gBoAV7PzD8sqjBJUjEa7fEP27xcH7ufTu1LWZKkAabPs3NmzV3A8f1fjiSpaI0O9ZzSa3Uvavf1byikIklSoRq9q+cvey1vpDa52vR+r0aSVLhGx/i/WHQhkqTmaPRBLOMi4s6IWFV/zY+IcUUXJ0nqf41e3L2B2jdu96u//qHeJkkaYBoN/rbMvCEzN9Zfc4G2AuuSJBWk0eBfHRFnRsSg+utMYHWRhUmSitFo8P9HanPq/Jbaw1U+B8wsqCZJUoEavZ3zm8A5m5+NW39u7neo/YMgSRpAGu3xH9b7geiZuYbaE7UkSQNMo8G/V0SM3LxS7/E3+teCJOl9pNHw/i7wSETcXl8/Ffi7YkqSJBWp0W/u3hQRXcCx9aZTMvOp4sqSJBWl4eGaetAb9pI0wPV5WmZJ0sBm8EtSxRQW/BExpz6h27Jebd+IiBcjYmn9dWJR55ckbV+RPf65wLTttH8vM9vrr7sLPL8kaTsKC/7MfABYU9T7S5J2Txlj/BdGxOP1oaCRu95dktSfmh381wGTgHZqk719d0c7RkRnRHRFRFdPT0+TypOkPV9Tgz8zX8nMTZn5e+DvgSk72Xd2ZnZkZkdbm1P/S1J/aWrwR8TYXqsnA8t2tK8kqRiFTbQWET8CjgZGR0Q38LfA0RHRDiTwHHBeUeeXJG1fYcGfmWdsp/n6os4nSWqM39yVpIox+CWpYgx+SaoYg1+SKsbgl6SKMfglqWIMfkmqGINfkirG4JekijH4JaliDH5JqhiDX5IqxuCXpIox+CWpYgx+SaoYg1+SKsbgl6SKMfglqWIMfkmqGINfkiqmsOCPiDkRsSoilvVq+2BE3BsRK+s/RxZ1fknS9hXZ458LTNuq7VLg/sw8ELi/vi5JaqLCgj8zHwDWbNU8HbixvnwjcFJR55ckbV+zx/jHZObL9eXfAmN2tGNEdEZEV0R09fT0NKc6SaqA0i7uZmYCuZPtszOzIzM72tramliZJO3Zmh38r0TEWID6z1VNPr8kVV6zg/+nwDn15XOABU0+vyRVXpG3c/4IeAQ4KCK6I+Jc4ApgakSsBD5VX5ckNdHgot44M8/YwabjijqnJGnX/OauJFWMwS9JFWPwS1LFGPySVDEGvyRVjMEvSRVj8EtSxRj8klQxBr8kVYzBL0kVY/BLUsUY/JJUMQa/JFWMwS9JFWPwS1LFGPySVDEGvyRVjMEvSRVj8EtSxRj8klQxhT1sfWci4jlgHbAJ2JiZHWXUIUlVVErw1x2Tmb8r8fySVEkO9UhSxZQV/AksjIjFEdG5vR0iojMiuiKiq6enp8nlSdKeq6zg/0RmfhQ4AbggIv586x0yc3ZmdmRmR1tbW/MrlKQ9VCnBn5kv1n+uAu4EppRRhyRVUdODPyI+EBHDNi8DnwaWNbsOSaqqMu7qGQPcGRGbzz8vM/93CXVIUiU1Pfgz8zfAR5p9XklSjbdzSlLFGPySVDEGvyRVjMEvSRVj8EtSxRj8klQxBr8kVYzBL0kVY/BLUsUY/JJUMQa/JFWMwS9JFWPwS1LFGPySVDEGvyRVjMEvSRVj8EtSxRj8klQxBr8kVYzBL0kVU0rwR8S0iPh1RDwTEZeWUYMkVVXTgz8iBgHfB04AJgNnRMTkZtchSVVVRo9/CvBMZv4mM/8duBWYXkIdklRJkZnNPWHE54Bpmfmf6utnAX+amRdutV8n0FlfPQj4dVMLba7RwO/KLkK7xd/dwLan//4OyMy2rRsHl1FJIzJzNjC77DqaISK6MrOj7DrUd/7uBraq/v7KGOp5Efhwr/Vx9TZJUhOUEfz/BBwYERMj4g+A/wD8tIQ6JKmSmj7Uk5kbI+JC4B+BQcCczHyy2XW8z1RiSGsP5e9uYKvk76/pF3clSeXym7uSVDEGvyRVjMFfIqeuGLgiYk5ErIqIZWXXor6LiA9HxKKIeCoinoyIi8uuqZkc4y9JfeqKp4GpQDe1u53OyMynSi1MDYmIPwfWAzdl5iFl16O+iYixwNjMXBIRw4DFwElV+f/PHn95nLpiAMvMB4A1Zdeh3ZOZL2fmkvryOmA5sH+5VTWPwV+e/YEXeq13U6H/8KT3i4iYABwO/KrkUprG4JdUWRGxDzAf+FJmvlZ2Pc1i8JfHqSukEkVEC7XQvyUzf1J2Pc1k8JfHqSukkkREANcDyzPzqrLraTaDvySZuRHYPHXFcuA2p64YOCLiR8AjwEER0R0R55Zdk/rkKOAs4NiIWFp/nVh2Uc3i7ZySVDH2+CWpYgx+SaoYg1+SKsbgl6SKMfglqWIMflVeRIyIiPObcJ6TImJy0eeRdsXgl2AE0HDwR83u/L9zEmDwq3Tex6/Ki4jNM6P+GlgEHAaMBFqAr2XmgvpEXv9IbSKvI4ATgbOBM4EeahPuLc7M70TEJOD7QBvwBvCfgQ8CPwNerb9mZOazzfqMUm9Nf9i69D50KXBIZrZHxGBgaGa+FhGjgUcjYvNUGgcC52TmoxFxJDAD+Ai1fyCWUJvTHWoP8P6rzFwZEX8K/CAzj62/z88y845mfjhpawa/9G4B/Lf6g1Z+T22q7DH1bc9n5qP15aOABZm5AdgQEf8AW2Z7/DPg9tp0MADs3azipUYY/NK7fYHaEM0Rmfl2RDwHtNa3vd7A8XsBazOzvZjypPfOi7sSrAOG1ZeHA6vqoX8McMAOjnkI+MuIaK338v8CoD6n+79GxKmw5ULwR7ZzHqk0Br8qLzNXAw/VH5zeDnRExBPULt6u2MEx/0RtGu3HgXuAJ6hdtIXaXw3nRsS/AE/yziM1bwVmRcQ/1y8AS6Xwrh5pN0XEPpm5PiKGAg8AnZuf4yq9nznGL+2+2fUvZLUCNxr6Gijs8UtSxTjGL0kVY/BLUsUY/JJUMQa/JFWMwS9JFfP/AWJo8+R0CDk2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, train_size=0.5, stratify=y)\n",
    "\n",
    "df_train = pd.DataFrame(y_train, columns=['target'])\n",
    "df_train['type'] = 'train'\n",
    "\n",
    "df_test = pd.DataFrame(y_test, columns=['target'])\n",
    "df_test['type'] = 'test'\n",
    "\n",
    "df_set = df_train.append(df_test)\n",
    "\n",
    "_ = sns.countplot(x='target', hue='type', data=df_set)     \n",
    "\n",
    "print('train samples:', len(X_train))\n",
    "print('test samples', len(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "removed-worth",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BaggingClassifier(base_estimator=DecisionTreeClassifier(max_depth=3))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base_estimator = tree.DecisionTreeClassifier(max_depth=3)\n",
    "model = ensemble.BaggingClassifier(base_estimator=base_estimator, n_estimators=10)\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "constitutional-stage",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>29</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0   1   2\n",
       "0  29   1   0\n",
       "1   2  29   4\n",
       "2   0   3  21"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted = model.predict(X_test)\n",
    "\n",
    "confusion_matrix = pd.DataFrame(metrics.confusion_matrix(y_test, predicted))\n",
    "confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "permanent-confusion",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVoAAAD4CAYAAACt8i4nAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAUw0lEQVR4nO3de3RU5bnH8d8TYpSbR7wwogRvUSxqD+Kt1BuKFq8FteKt3qrEaqnictXS4tFWq0VFPYtqtahUbS1qiy6wWj0ukEOhooBFQEDxggqGIF6WIRyFJM/5IyOmNGQyybzZO2++H9a7MrMn8+6HWVm/PHn33jPm7gIAhFOUdAEAEDuCFgACI2gBIDCCFgACI2gBILDi0DvofOBITmsIrPKl8UmXEL2SYnqStrBNsay1c+STOf/3z7tbvb/m4KcHAAIL3tECQJuy9PWPBC2AuBR1SrqCf0PQAoiLtcmya14IWgBxYekAAAKjowWAwOhoASAwOloACIyzDgAgMJYOACAwlg4AIDA6WgAIjKAFgMA6cTAMAMJijRYAAmPpAAACo6MFgMDoaAEgMDpaAAiMS3ABIDCWDgAgMJYOACAwOloACIygBYDAOBgGAIGlcI02fT02ALSGFTV/NDWNWamZvWhmS8zsdTO7Krv9F2a2yswWZMdJuUqiowUQl8J1tDWSrnH3V82su6T5ZvZC9rG73H1ccyciaAFExQoUtO5eIakie7vKzJZK2rUlc7F0ACAqZpbPKDezeQ1G+Rbm3F3SgZJezm4aaWYLzWyimfXIVRNBCyAqVmTNHu4+wd0PbjAm/Nt8Zt0kTZY0yt0/l3SvpL0k9Vd9x3tHrpo69NJB78x2euCmC9Rzh+5ylyZOnq17Js3QAfvsqt+MOVtdO2+t9z78WBePeVhV1V8kXW4Ubrx+jGbNnKEe22+vx598OulyojX77zN169ibVVdbp9POOFOXjGi0UYtSoZYOsnNtpfqQfdTdn5Qkd69s8Pj9kv6aa54O3dHW1NZp9J1PasAZN+voC8bpsrOO0r577qx7rz9X142fokOG36KpL76mqy8cnHSp0Thl6DCNv/ffmgYUUG1trW65+Ub99r4H9NTUZ/Tcs3/V22+9lXRZbSafpYMc85ikByUtdfc7G2zv1eDbTpO0OFdNHTpoV6/9XAuWrZQkrVv/pZa9u1q77LSdyvr01Kz59T+Y0+cs07DB/ROsMi4DDjpE2267XdJlRG3xooUqLd1NvUtLtVVJiU446WTNeHFa0mW1mUIFraTDJZ0v6djNTuW6zcwWmdlCScdIujrXRDmXDsxsX0lD9fXRtlWSprr70lzPbU/69Npe/fv21tzFK7T0nQqdOuibenrGQp1+/AD1zuRc6wZSY01lpXbutfOm+z0zGS1auDDBitpYgVYO3H3WFmZ7Nt+5muxozeynkh7L7uyV7DBJk8xsdL47S6uunUs0adyl+sm4yaqq/kKX/eJRlQ8/UrMfvVbdumytDRtrky4RQDMVsKMtmFwd7SWS9nP3jQ03mtmdkl6XNLaxJ2VPkSiXpOLeg1S8434FKDWM4uIiTRo3Qo//bZ6mTH9NkvTmikqdesU9kqSyPj114pHprR/YXM9MRqsrVm+6v6ayUplMJsGK2lZRUfpWRHNVVCdpl0a298o+1qiGp0ykOWQl6b4bztMb767W+D9O37Rtpx7dJNX/Zhw9Yoju/8uspMoD8rbf/gfo/fdXaOXKD7RxwwY99+wzOvqYY5Muq820x452lKRpZrZc0gfZbX0klUkaGbCuNvHt/nvqvFMO06I3V2nOY/UrITfcPVVlpT112VlHSZKmTF+gR6bMSbLMqIz56TWaP+8VffbZZzr5+EEqv3ykhp7+vaTLikpxcbF+NuZ6XV5+qerqajXstDNUVrZ30mW1nfS9p4zM3Zv+BrMiSYfqXw+GzXX3Zi1cdj5wZNM7QKtVvjQ+6RKiV1Kcvj9HY7RNcetjcseLHmt25qx96Ow2ieWcZx24e50kWjoA7UJbLgk0V4e+MgxAfKyIoAWAoOhoASAwghYAAiNoASAwghYAQktfzhK0AOKSxktwCVoAUWHpAABCS1/OErQA4kJHCwCBEbQAEBhBCwCB8V4HABAYHS0ABEbQAkBgKcxZghZAXOhoASCwIg6GAUBYKWxoCVoAcaGjBYDA6GgBIDAOhgFAYCnMWaXvHXIBoBWKioqaPZpiZqVm9qKZLTGz183squz27c3sBTNbnv3aI2dNBfq/AUAqmDV/5FAj6Rp37yfpW5J+ZGb9JI2WNM3d95Y0LXu/SQQtgKiYWbNHU9y9wt1fzd6ukrRU0q6Shkp6OPttD0salqsmghZAVPLpaM2s3MzmNRjljc9pu0s6UNLLkjLuXpF9aLWkTK6aOBgGICr5nHXg7hMkTcgxXzdJkyWNcvfPG87v7m5mnms/dLQAolLANVqZ2VaqD9lH3f3J7OZKM+uVfbyXpDW55iFoAUSlqMiaPZpi9a3rg5KWuvudDR6aKunC7O0LJU3JVVPwpYP3Z/536F10eJmBVyZdQvQWP3970iV0CHv17NzqOQp4wcLhks6XtMjMFmS3/VzSWElPmNklkt6TNDzXRKzRAohKoXLW3Wdpyx9ePjifuQhaAFHhElwACCyFOUvQAogLb5MIAIGxdAAAgRG0ABBYCnOWoAUQFzpaAAgshTlL0AKIC2cdAEBgRSlsaQlaAFFJYc4StADiwsEwAAgshUu0BC2AuHAwDAACsy2+s2FyCFoAUUlhQ0vQAogLB8MAILAU5ixBCyAuXLAAAIFx1gEABJbChpagBRAXlg4AILD0xSxBCyAynN4FAIGl8FgYQQsgLpx1AACBsXQAAIGlsKFVUdIFAEAhmVmzRzPmmmhma8xscYNtvzCzVWa2IDtOyjUPQQsgKpbHaIaHJJ3QyPa73L1/djybaxKWDgBEpVMB1w7cfaaZ7d7aeQjarMrVFfrVDT/Tp598LJnpu6edqeHnnJ90We1e78x2euCmC9Rzh+5ylyZOnq17Js3QAfvsqt+MOVtdO2+t9z78WBePeVhV1V8kXW5UamtrddWIc7XDjj31y9t+k3Q5bSafg2FmVi6pvMGmCe4+oRlPHWlmF0iaJ+kad/+0qW8maLM6FRdr5NXXqu++/bS+ulo/OP9MHXLYQO2xZ1nSpbVrNbV1Gn3nk1qwbKW6ddla//jTTzXt5WW69/pzNfqupzRr/lu6YOi3dPWFg3Xjb59JutyoTPnzn1S62x5aX12ddCltKp+TDrKh2pxgbeheSTdJ8uzXOyT9oKknsEabteOOO6nvvv0kSV26dtXuu++ptWvWJFxV+7d67edasGylJGnd+i+17N3V2mWn7VTWp6dmzX9LkjR9zjING9w/wSrjs3ZNpea+9HcNOeX0pEtpc0VmzR4t4e6V7l7r7nWS7pd0aM6aWrQnSWZ2cUufm3YVH67Sm28sVb/9v5l0KVHp02t79e/bW3MXr9DSdyp06qD61/f04weod6ZHwtXF5Xfjb9cPrhiVypP3QzNr/mjZ/Narwd3TJC3e0vd+pTUd7S+bKKTczOaZ2bxHfn9/K3bR9tavr9aYa0fpqmtGq2u3bkmXE42unUs0adyl+sm4yaqq/kKX/eJRlQ8/UrMfvVbdumytDRtrky4xGi/PnqntevTQ3n37JV1KIgp8etckSS9J6mtmK83sEkm3mdkiM1so6RhJV+eap8k12uxEjT4kKbOl5zVc9/ioqsZzFZEWNTUbdd21o/SdE07W0ccen3Q50SguLtKkcSP0+N/macr01yRJb66o1KlX3CNJKuvTUyceuV+SJUZlyaIFmjP7fzV3zixt3LBB66urdfuNP9dPrr8l6dLaRKcCXhnm7uc0svnBfOfJdTAsI2mIpM2PqJmkf+S7szRzd/36xuu12x576uzvX5R0OVG574bz9Ma7qzX+j9M3bdupRzd99Ok6mZlGjxii+/8yK8EK43LxD6/UxT+8UpK08J9zNXnSIx0mZKV0XhmWK2j/Kqmbuy/Y/AEzmxGioKQsfO1VPf/sVO1Vto8uOrf+AMJlV4zSwCOOSriy9u3b/ffUeaccpkVvrtKcx0ZLkm64e6rKSnvqsrPqX9sp0xfokSlzkiwTEUlj0Jp72L/s29PSQXvV56hRSZcQvcXP3550CR3CXj07tzomr3n6jWZnzh2n9m2TWOY8WgBRSWNHS9ACiEoK3yWRoAUQl+IUJi1BCyAqKcxZghZAXPi4cQAILIU5S9ACiAtnHQBAYIV84+9CIWgBRCWFOUvQAoiLNffTwNoQQQsgKnS0ABAYQQsAgeXz4YxthaAFEJVOKfwkRIIWQFS4MgwAAmONFgACS2FDS9ACiEsR59ECQFh0tAAQWHEKF2kJWgBRoaMFgMA4vQsAAkthzhK0AOKSwgvDCFoAcWHpAAACS2PQprHLBoAWszxGzrnMJprZGjNb3GDb9mb2gpktz37tkWseghZAVMyaP5rhIUknbLZttKRp7r63pGnZ+00iaAFExcyaPXJx95mSPtls81BJD2dvPyxpWK55CFoAUSnKY5hZuZnNazDKm7GLjLtXZG+vlpTJ9QQOhgGISj4Hw9x9gqQJLd2Xu7uZea7vCx603TuT5aEtn35H0iVE77hbZyRdQoewbOyQVs/RBh9lU2lmvdy9wsx6SVqT6wksHQCISj5LBy00VdKF2dsXSpqS6wm0mwCiUsiO1swmSRokaUczWynpBkljJT1hZpdIek/S8FzzELQAolLIhQN3P2cLDw3OZx6CFkBUOqXwyjCCFkBUUpizBC2AuBifGQYAYdHRAkBgfAouAARGRwsAgaXx/WgJWgBRSeGnjRO0AOLCWQcAEFgKVw4IWgBxoaMFgMBYowWAwDjrAAACS1/MErQAIkNHCwCBpS9mCVoAsUlh0hK0AKLC0gEABJa+mCVoAcQmhUlL0AKICleGAUBgKVyiJWgBxCWFOUvQAoiLpbClJWgBRCWFOUvQAohLCnOWoAUQmRQmLUELICqc3pVys/8+U7eOvVl1tXU67YwzdcmI8qRLis6GL7/UqMsv1sYNG1RbW6ujjj1OF434UdJltXs7/8c2unX4AdqhW4lc0hOvfKA/zH5fQw7IaORxZdprp64afs8cLV71edKlBlfINVozWyGpSlKtpBp3P7gl8xC0WbW1tbrl5hv1u/t/r0wmo3PP+p4GHXOs9iorS7q0qGxVUqI77n5Anbt0UU3NRl1VfqEOHXiE+u3/n0mX1q7V1tXp1meWacmHVepa0kmTfzxQ/1j+sZavXqcr//BP/fL0/ZIusc0EOBh2jLuvbc0ERYWqpL1bvGihSkt3U+/SUm1VUqITTjpZM16clnRZ0TEzde7SRZJUU1OjmpqaVP6p1958VLVBSz6skiRVb6jV2x9VK7PtNnrno2q9u3Z9wtW1LcvjX1vJGbRmtq+ZDTazbpttPyFcWW1vTWWldu6186b7PTMZVVZWJlhRvGpra1V+/pk648RBOujQgfrG/t9MuqSo7NpjG31jl+567YPPki4lEWb5DCs3s3kNxubrhS7pf8xsfiOPNVuTQWtmV0qaIunHkhab2dAGD9/S0p2iY+vUqZMm/OHPenzqC1q2ZLHefXt50iVFo0tJJ40/r79+/fQyVX9Zm3Q5ibA8hrtPcPeDG4wJm013hLsPkHSipB+Z2VEtqSlXRztC0kHuPkzSIEn/ZWZXNfj/NKrhb4kH79+87nTqmclodcXqTffXVFYqk8kkWFH8unXfVv0POkRz58xOupQoFBeZxn+/v55eUKEXXl+TdDnJySdpc3D3VdmvayQ9JenQlpSUK2iL3H1ddkcrVB+2J5rZnU2V2fC3RHs5cr/f/gfo/fdXaOXKD7RxwwY99+wzOvqYY5MuKzqfffqJ1lXVH/n+8osvNP+Vl1S62x4JVxWHX31vP729ploPzXov6VISVWTW7NEUM+tqZt2/ui3pO5IWt6SmXGcdVJpZf3dfIEnuvs7MTpE0UdIBLdlhWhUXF+tnY67X5eWXqq6uVsNOO0NlZXsnXVZ0Pl67VrfddJ1qa2vlXqejBw/RwCOOTrqsdm/Abttp2IBd9UZFlZ66cqAk6a7nl6ukuEjXffcb2r5rie67aICWVVTp0onzE642rAIe4spIeir73gnFkv7k7s+1qCZ33/KDZr1Vf+7Y6kYeO9zdc/7N90WNtrwDFMTaqi+TLiF6x906I+kSOoRlY4e0OiffrFzf7MzZJ9OlTU49aLKjdfeVTTzGwhqA1Enj6YJcsAAgKrx7FwAElsKcJWgBxIU3/gaAwFKYswQtgLikMGcJWgCRSWHSErQAosLpXQAQGGu0ABBYEUELAKGlL2kJWgBRYekAAAJLYc4StADiQkcLAIFxCS4ABJa+mCVoAUQmhQ0tQQsgLlwZBgChpS9nCVoAcUlhzhK0AOKS62PEk0DQAohKCnNWRUkXAACxo6MFEJU0drQELYCocHoXAARGRwsAgRG0ABAYSwcAEFgaO1pO7wIQFctj5JzL7AQze8PM3jKz0S2tiaAFEJcCJa2ZdZJ0j6QTJfWTdI6Z9WtJSSwdAIhKAS/BPVTSW+7+jiSZ2WOShkpaku9EwYN2m+IUrkznYGbl7j4h6Tqaq3ePrZMuIW/t7TVeNnZI0iXkrb29xoWST+aYWbmk8gabJjR4zXaV9EGDx1ZKOqwlNbF00Ljy3N+CVuI1Do/XOAd3n+DuBzcYQX4xEbQA0LhVkkob3O+d3ZY3ghYAGjdX0t5mtoeZlUg6W9LUlkzEwbDGdbh1rQTwGofHa9wK7l5jZiMlPS+pk6SJ7v56S+Yydy9ocQCAf8XSAQAERtACQGAEbQOFutwOW2ZmE81sjZktTrqWWJlZqZm9aGZLzOx1M7sq6Zo6OtZos7KX270p6XjVn5g8V9I57p73VSDYMjM7StI6SY+4+/5J1xMjM+slqZe7v2pm3SXNlzSMn+Xk0NF+bdPldu6+QdJXl9uhgNx9pqRPkq4jZu5e4e6vZm9XSVqq+quckBCC9muNXW7HDyfaNTPbXdKBkl5OuJQOjaAFImVm3SRNljTK3T9Pup6OjKD9WsEutwOSZmZbqT5kH3X3J5Oup6MjaL9WsMvtgCSZmUl6UNJSd78z6XpA0G7i7jWSvrrcbqmkJ1p6uR22zMwmSXpJUl8zW2lmlyRdU4QOl3S+pGPNbEF2nJR0UR0Zp3cBQGB0tAAQGEELAIERtAAQGEELAIERtAAQGEELAIERtAAQ2P8DOgoSBDaEiEUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = sns.heatmap(confusion_matrix, annot=True, cmap=\"Blues\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bright-mixer",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.888\n",
      "precision: 0.887\n",
      "recall: 0.888\n",
      "f1 score: 0.887\n"
     ]
    }
   ],
   "source": [
    "print(\"accuracy: {:.3f}\".format(metrics.accuracy_score(y_test, predicted)))\n",
    "print(\"precision: {:.3f}\".format(metrics.precision_score(y_test, predicted, average='weighted')))\n",
    "print(\"recall: {:.3f}\".format(metrics.recall_score(y_test, predicted, average='weighted')))\n",
    "print(\"f1 score: {:.3f}\".format(metrics.f1_score(y_test, predicted, average='weighted')))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}