lectures/cap08_2_metricas_avaliacao_desempenho.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cap08_2_metricas_avaliacao_desempenho.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyP8V4DdEgs+AYkZT0XpUACk",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/viniciusrpb/cic0269_natural_language_processing/blob/main/lectures/cap08_2_metricas_avaliacao_desempenho.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Capítulo 8 - Classificação de Textos\n",
        "\n",
        "## 8.2. Métricas de Avaliação de Desempenho de Classificação\n",
        "\n",
        "Nesta seção vamos estudar como avaliar e comparar os desempenhos entre classificadores"
      ],
      "metadata": {
        "id": "whiaqg-5QDf6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np"
      ],
      "metadata": {
        "id": "zsXuuS_FAQ73"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Começando pela Classificação Binária\n",
        "\n",
        "Considere um dataset qualquer em que os labels sejam apenas denotados por \"positivo\" (representado pelo inteiro 0) ou \"negativo\" (representando pelo inteiro 1). A label de interesse (target) é a positiva. Suponha as seguintes predições:"
      ],
      "metadata": {
        "id": "hXES_TaiQzHt"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "VEkpERItiRmU"
      },
      "outputs": [],
      "source": [
        "y_true = [0,0,1,1,0,0,1,0,1,0,1,0,1,1,0]\n",
        "y_pred = [0,0,1,1,0,1,1,0,0,0,1,0,0,0,0]"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Construindo a matriz de confusão"
      ],
      "metadata": {
        "id": "BfcNlEt0ix23"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mat_conf = [[0,0],[0,0]]\n",
        "\n",
        "for i in range(0,len(y_true)):\n",
        "    mat_conf[y_true[i]][y_pred[i]]+=1"
      ],
      "metadata": {
        "id": "iPGSenfni0en"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "np.array(mat_conf)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tRsZcbdVjlFs",
        "outputId": "a18ca502-2494-400e-8511-80be2baa76f7"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[7, 1],\n",
              "       [3, 4]])"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tp = mat_conf[0][0]\n",
        "fp = mat_conf[1][0]\n",
        "fn = mat_conf[0][1]\n",
        "tn = mat_conf[1][1]"
      ],
      "metadata": {
        "id": "nxIrjDxCr2eU"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tn"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Yv0VLuSdtpGn",
        "outputId": "bd75c18c-2019-4d2a-bd33-1efc17c75867"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Acurácia (Taxa de classificação correta): métrica não confiável para dados com desbalanceamento entre classes"
      ],
      "metadata": {
        "id": "FnIpFfWArxXk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "acc = (tp+tn)/(tp+fn+fp+tn)\n",
        "print(f'Acuracia: {acc:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-Q00xER1rwv-",
        "outputId": "3b397283-c7b0-43a9-bee3-bb7d557bb210"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Acuracia: 0.733\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Precisão"
      ],
      "metadata": {
        "id": "4LL-2LJxsqGy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prec = tp/(tp+fp)\n",
        "print(f'Precisao: {prec:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R-_-UOhQsr55",
        "outputId": "0e3ba3d7-ae57-4c97-acb0-df9f802c3052"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Precisao: 0.700\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Revocação"
      ],
      "metadata": {
        "id": "kFBuTpqisvWh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rec = tp/(tp+fn)\n",
        "print(f'Revocacao: {rec:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b5HTsPGFsvf5",
        "outputId": "397651a6-1097-4a2c-bcc9-fb6e7a925f92"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Revocacao: 0.875\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "F1-Score"
      ],
      "metadata": {
        "id": "h_gef2jYugnb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "f1 = 2*(prec*rec)/(prec+rec)\n",
        "print(f'F1 Score: {f1:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xY3-yBN3uguK",
        "outputId": "c8d37e62-41e5-4fb2-9b28-919a07359b52"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "F1 Score: 0.778\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Generalizando para Classificação Multi-classes\n",
        "\n",
        "Vamos tratar aqui a abordadem \"One Against All\" para calcular as métricas acima, em que tomamos classe por classe"
      ],
      "metadata": {
        "id": "O1tLWYl7Q8bF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_true = [0,2,1,0,0,1,1,2,0,1,1,0,1,2,1,2,1,0,2,1,0]\n",
        "y_pred = [0,2,1,2,0,1,2,2,0,1,2,1,2,0,0,0,1,0,2,1,1]"
      ],
      "metadata": {
        "id": "xomNjbyHxIsU"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Extraindo-se o suporte para cada classe, isto é, a quantidade de instâncias verdadeiras de cada classe:"
      ],
      "metadata": {
        "id": "U2tVrSfe654s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sup = []\n",
        "c = 3\n",
        "for i in range(0,c):\n",
        "    v = len([x for x in y_true if x == i])\n",
        "    sup.append(v)\n",
        "    print(f'Suporte Label {i}: {v}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bdjJZoSv69Eo",
        "outputId": "70151598-2f86-4394-95d5-02a34159d47a"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Suporte Label 0: 7\n",
            "Suporte Label 1: 9\n",
            "Suporte Label 2: 5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "mat_conf = []\n",
        "c = 3\n",
        "n = len(y_true)\n",
        "\n",
        "for i in range(0,c):\n",
        "    mat_conf.append([0]*c)\n",
        "\n",
        "for i in range(0,len(y_true)):\n",
        "    mat_conf[y_true[i]][y_pred[i]]+=1"
      ],
      "metadata": {
        "id": "zT8oDN9q1F5H"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "np.array(mat_conf)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-jSpgroL2PT7",
        "outputId": "ac9350e9-e884-4c23-9822-c08aa9879745"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[4, 2, 1],\n",
              "       [1, 5, 3],\n",
              "       [2, 0, 3]])"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Acurácia geral"
      ],
      "metadata": {
        "id": "RbDeRs7t12NE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tp_0 = mat_conf[0][0]\n",
        "tp_1 = mat_conf[1][1]\n",
        "tp_2 = mat_conf[2][2]\n",
        "acc = (tp_0+tp_1+tp_2)/(n)\n",
        "print(f'Acuracia: {acc:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Oz0g2Umc1338",
        "outputId": "222c6ee1-fce5-48f6-a8ca-7930c2407f3f"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Acuracia: 0.571\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "prec = [0]*c\n",
        "rec = [0]*c\n",
        "f1 = [0]*c"
      ],
      "metadata": {
        "id": "YBgKVu655Y0A"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Métricas para a classe \"0\""
      ],
      "metadata": {
        "id": "7NYtyfFr1kFk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tp = mat_conf[0][0]\n",
        "fp = mat_conf[1][0]+mat_conf[2][0]\n",
        "fn = mat_conf[0][1]+mat_conf[0][2]"
      ],
      "metadata": {
        "id": "e-9KzLEw1nCt"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prec[0] = tp/(tp+fp)\n",
        "print(f'Precisao: {prec[0]:.3f}')\n",
        "rec[0] = tp/(tp+fn)\n",
        "print(f'Revocacao: {rec[0]:.3f}')\n",
        "f1[0] = 2*(prec[0]*rec[0])/(prec[0]+rec[0])\n",
        "print(f'F1 Score: {f1[0]:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p6EMWUp65HsY",
        "outputId": "29010dc3-f3e4-420d-f00a-8cb7aba824de"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Precisao: 0.571\n",
            "Revocacao: 0.571\n",
            "F1 Score: 0.571\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Métricas para a classe \"1\""
      ],
      "metadata": {
        "id": "xopmekgQ5EF5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tp = mat_conf[1][1]\n",
        "fp = mat_conf[0][1]+mat_conf[2][1]\n",
        "fn = mat_conf[1][0]+mat_conf[1][2]"
      ],
      "metadata": {
        "id": "Tci_v3125GaR"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prec[1] = tp/(tp+fp)\n",
        "print(f'Precisao: {prec[1]:.3f}')\n",
        "rec[1] = tp/(tp+fn)\n",
        "print(f'Revocacao: {rec[1]:.3f}')\n",
        "f1[1] = 2*(prec[1]*rec[1])/(prec[1]+rec[1])\n",
        "print(f'F1 Score: {f1[1]:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qUhwOFiQ6KRG",
        "outputId": "1b0ca8f5-547c-4fd8-b8ac-9b85573669c8"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Precisao: 0.714\n",
            "Revocacao: 0.556\n",
            "F1 Score: 0.625\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Métricas para a classe \"2\""
      ],
      "metadata": {
        "id": "WZcPTavm6Q81"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tp = mat_conf[2][2]\n",
        "fp = mat_conf[0][2]+mat_conf[1][2]\n",
        "fn = mat_conf[2][0]+mat_conf[2][1]"
      ],
      "metadata": {
        "id": "siMcGX0j6Tks"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prec[2] = tp/(tp+fp)\n",
        "print(f'Precisao: {prec[2]:.3f}')\n",
        "rec[2] = tp/(tp+fn)\n",
        "print(f'Revocacao: {rec[2]:.3f}')\n",
        "f1[2] = 2*(prec[2]*rec[2])/(prec[2]+rec[2])\n",
        "print(f'F1 Score: {f1[2]:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nRoM7RCX6UrM",
        "outputId": "ce4780d1-baf0-4c72-e4b5-c9401e1d6ab0"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Precisao: 0.429\n",
            "Revocacao: 0.600\n",
            "F1 Score: 0.500\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Métricas Weighted-Average e Macro**\n",
        "\n",
        "Atenção: para dados que apresentam desbalanceamento entre classes, a abordagem *Macro* é a mais apropriada por desconsiderar o suporte (quantidade de instâncias por classe) no cálculo da precisão, da revocação e da F1-Score gerais.\n",
        "\n",
        "Precisão Weighted-Average"
      ],
      "metadata": {
        "id": "w8QoN0fV6d9M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "avg_weighted_prec = 0\n",
        "for i in range(0,len(prec)):\n",
        "    avg_weighted_prec+=prec[i]*(sup[i]/n)\n",
        "\n",
        "print(f'Weighted-Avg Precisao: {avg_weighted_prec:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xYbjai-66hgc",
        "outputId": "56f2c0bc-c09d-4741-ad02-568c69581adc"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Weighted-Avg Precisao: 0.599\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Precisão Macro"
      ],
      "metadata": {
        "id": "v2IOBUJ99mtx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prec_macro = 0\n",
        "for i in range(0,len(prec)):\n",
        "    prec_macro+=prec[i]\n",
        "prec_macro/=c\n",
        "\n",
        "print(f'Precisao Macro: {prec_macro:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6beUHFIH9mcZ",
        "outputId": "f812a273-48c6-47f7-eb7c-99023c232e26"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Precisao Macro: 0.571\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Revocação Weighted-Average"
      ],
      "metadata": {
        "id": "3JOrd1YH8gA-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "avg_weighted_rec = 0\n",
        "for i in range(0,len(rec)):\n",
        "    avg_weighted_rec+=rec[i]*(sup[i]/n)\n",
        "\n",
        "print(f'Weighted-Avg Revocacao: {avg_weighted_rec:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L-fDauA38gHm",
        "outputId": "ca192956-e545-4744-a44b-3f0c25885dca"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Weighted-Avg Revocacao: 0.571\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Revocação Macro"
      ],
      "metadata": {
        "id": "-O7gOmGHBUE6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rec_macro = 0\n",
        "for i in range(0,len(rec)):\n",
        "    rec_macro+=rec[i]\n",
        "rec_macro/=c\n",
        "\n",
        "print(f'Revocacao Macro: {rec_macro:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xL41pkY_BULi",
        "outputId": "c094e030-9c51-4ed9-8407-5976517458f7"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Revocacao Macro: 0.576\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "F1-Score Weighted-Average\n"
      ],
      "metadata": {
        "id": "1LNONYSW9L8L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "avg_weighted_f1 = 0\n",
        "for i in range(0,len(f1)):\n",
        "    avg_weighted_f1+=f1[i]*(sup[i]/n)\n",
        "#avg_weighted_f1/=c\n",
        "\n",
        "print(f'Weighted-Avg F1-Score: {avg_weighted_f1:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jqadWtLZ9MDq",
        "outputId": "c59871e1-977d-4da3-d6a3-8aa805b4bffa"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Weighted-Avg F1-Score: 0.577\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Macro F1-Score"
      ],
      "metadata": {
        "id": "k3tzi02mBpio"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "f1_macro = 0\n",
        "for i in range(0,len(f1)):\n",
        "    f1_macro+=f1[i]\n",
        "f1_macro/=c\n",
        "\n",
        "print(f'F1-Score Macro: {f1_macro:.3f}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4Baqb0zKBpzg",
        "outputId": "f12c7d81-cd05-4237-e162-6082b21e1de4"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "F1-Score Macro: 0.565\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Matriz de Confusão e Cálculo de Métricas com Bibliotecas Python\n",
        "\n",
        "Pode-se utilizar implementações de bibliotecas para calcular as métricas e gerar visualizações para melhor entendimento.\n",
        "\n",
        "**Matriz de confusão**\n",
        "\n",
        "A documentação da matriz de confusão no sklearn pode ser encontrada [aqui](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html). Basta utilizar os seguintes comandos:"
      ],
      "metadata": {
        "id": "1DG_Bcf9B_-2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "print(confusion_matrix(y_true, y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a4SMk3S6CAHn",
        "outputId": "a746ec5a-50ea-4512-a5bd-1aeef719c165"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[4 2 1]\n",
            " [1 5 3]\n",
            " [2 0 3]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Como apresentar visualmente essa matriz de confusão? Podemos utilizar alguma das APIs gráficas do Python, como matplotlib e plotly."
      ],
      "metadata": {
        "id": "tH6fiS-3CYUl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import ConfusionMatrixDisplay\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "ConfusionMatrixDisplay.from_predictions(y_true, y_pred)\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "S1zC1wdyCdXd",
        "outputId": "aae557d7-d9dd-445b-abe6-a32106a46b0f"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAASwAAAEKCAYAAACoiGheAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAXTklEQVR4nO3de7BdZZ3m8e9zkkPuCYSEkIQgAYKQpg1QEZB0U0DbiGJpS9HVgM3UdGmjtjY63dSMiNW2TInTQ42IQl8yQimDQHP10gIRwZBQXiCBACExgJEmhEQ4hJjkJOFc9m/+2OvgIeTss3aydtZ6d56PtYq99lmX3zkVn3rXu971LkUEZmYp6Ci7ADOzvBxYZpYMB5aZJcOBZWbJcGCZWTIcWGaWjJFlF2Bm+y9JLwBbgX6gLyLmN9regWVmZTszIrrybOhLQjNLhqo00n3sQaNi0oyxZZdRWZs3jS+7hMob2V2df89VtHPn6/T2dGtvjvG+M8fFa5v6c227/Kk3ngF2DvpqYUQsHFiR9BvgdSCAfxv8s92p1CXhpBlj+atbzyy7jMr63q1/XHYJlTftsTfKLqHSlj163V4f47VN/Ty66PBc246Y/tzOYfql/igi1ks6BHhA0q8iYslQG/uS0MyaEkAt5/+GPVbE+uy/rwD3ACc32t6BZWZNCYLe6M+1NCJpnKQJA5+Bs4GVjfap1CWhmaUhT+sph2nAPZKgnkW3RMT9jXZwYJlZU4Kgv4CbdRGxFpjXzD4OLDNrWo1y7sY6sMysKQH0O7DMLBVuYZlZEgLoLWnAuQPLzJoShC8JzSwRAf0lPQHlwDKzptRHupfDgWVmTRL97NXz03vMgWVmTal3ujuwzCwB9XFYDiwzS0TNLSwzS4FbWGaWjED0lzQzlQPLzJrmS0IzS0IgemJEKed2YJlZU+oDR31JaGaJcKe7mSUhQvSHW1hmloiaW1hmloJ6p3s50eHAMrOmuNPdzJLS73FYZpYCj3Q3s6TUfJfQzFJQf/jZgWVmCQhErx/NqZboh1UXddB5CBzzzbJmsK6mQ8dv46qzH+TgsTsI4M6Vc7l5xbvKLqtSLvvrpZx6wjo2bxnNxy8/r+xyChVBaQNHW3pWSedIWiPpeUmfb+W5ivbbW8To2SW9GqTi+mri6qWn8eGbL+Cifz+PC961kiMnbyq7rEpZtGQOl199dtlltIio5VyK1rLAkjQCuB54PzAXuFDS3Fadr0g9v4XNS8XU8xxYu9O1fRyrX50KwPbeA1i76SCmje8uuapqeXrNoWzZNqrsMloiqLew8ixFa2UL62Tg+YhYGxE9wG3Ah1t4vsK8eHUHsz5Xo6SnD5IyY8IWjjuki6c2Tiu7FNuH+unItRStlYE1E1g3aP2l7LtK27wERh4UjEuiLViuMZ29XHPuIv7p4QV09xxQdjm2jwSiFvmWopXe6S7pEuASgInTx5RcDWxdITY/LJ58RNR6oNYNv/6COOoqXx4ONrKjn6+fu4gfrTmGn/z6yLLLsX2o/pqv9nuWcD0wa9D6Ydl3bxERC4GFANP/4KDSU2HWpcGsS+tlbHkMNt7U4bB6m+DK9y5m7aYDuemJeWUXY/tce75I9TFgjqTZ1IPqAuCiFp7P9pETZ2zkQ8c9y7Ndk7nzotsBuPZnp7D0hXeUXFl1XPHpnzLvuI1MGr+T275xG9+56yTue/iYsssqRNCGI90jok/SZ4BFwAjgxoh4plXna4WJ74aJ7/YYrF098fJ0jr/2U2WXUWlfuf7MsktoqXZsYRER9wL3tvIcZrZvRajQFlY2BGoZsD4iPtho29I73c0sLfVO90IfzfkssBqYONyG5VyImlnCVNjAUUmHAecC38pzZrewzKwp9U733H1YUyQtG7S+MBsZMODrwH8HJuQ5mAPLzJrWxCj2roiYv7sfSPog8EpELJd0Rp6DObDMrCkDI90LsAD4kKQPAKOBiZJujoi/HGoH92GZWdNqdORaGomIyyPisIg4gvo4zYcahRW4hWVmTYqA3lqbDRw1s/ZUvyQsNrAiYjGweLjtHFhm1rS2HOluZu2nyWENhXJgmVmTir8kzMuBZWZNa8V87Xk4sMysKfW7hH7Nl5kloMCBo01zYJlZ03xJaGZJ8F1CM0uK7xKaWRIiRJ8Dy8xS4UtCM0uC+7DMLCkOLDNLgsdhmVlSPA7LzJIQAX2ewM/MUuFLQjNLgvuwzCwp4cAys1S4093MkhDhPiwzS4bo911CM0uF+7CAbevG8fO/O7nsMipr5c3/XHYJlTdn8X8tu4RK61kbe30MP0toZumIej9WGRxYZtY03yU0sySEO93NLCW+JDSzZPguoZklIcKBZWYJ8bAGM0uG+7DMLAmBqPkuoZmloqQGlgPLzJpUUKe7pNHAEmAU9Sy6MyK+1GgfB5aZNa+YJtYbwFkRsU1SJ/CIpPsi4hdD7eDAMrOmFdHCiogAtmWrndnSMAqHDCxJ32y0c0Rcugc1mlniAqjVcgfWFEnLBq0vjIiFAyuSRgDLgaOB6yPil40O1qiFtazBz8xsfxVA/hZWV0TMH/JQEf3ACZIOBO6RdHxErBxq+yEDKyK+M3hd0tiI2J63SjNrX0WPw4qIzZJ+CpwDDBlYww6mkPQeSauAX2Xr8yR5Jjmz/VnkXBqQNDVrWSFpDPCnZDkzlDyd7l8H3gf8ACAinpR0eo79zKwtqahnCacD38n6sTqA2yPiPxrtkOsuYUSsk95SYP8el2hm6SvgkjAingJObGafPIG1TtJpQGRjJT4LrN6D+sysHQRE/ruEhcrzQNAngU8DM4GXgROydTPbbynnUqxhW1gR0QV8tPAzm1m6SnqYMM9dwiMl/VDSq5JekfR9SUfui+LMrKIKuEu4J/JcEt4C3E69R38GcAdwa/GlmFkSBgaO5lkKliewxkbE/4uIvmy5GRhdeCVmloyIfEvRGj1LODn7eJ+kzwO3Uc/WvwDuLb4UM0tGSXcJG3W6L6ceUAOVfWLQzwK4vFVFmVm1qWpTJEfE7H1ZiJklokUd6nnkGuku6XhgLoP6riLiplYVZWZV1poO9TyGDSxJXwLOoB5Y9wLvBx4BHFhm+6uqjsMCzgf+BNgYEX8FzAMmtbQqM6u2Ws6lYHkuCXdERE1Sn6SJwCvArOJLqY7L/nopp56wjs1bRvPxy88ru5xK+i8nz2XM+H46OmDEyOC6+58tu6TKUE+N6Vc+j/pqqB+6T5nE6+dPL7us4jQ3gV+h8gTWsmzOmv9L/c7hNuDnw+0k6Ubgg8ArEXH8XlW5jy1aMofvP3Ac/+MTS8oupdL+9x3PM+lgT9yxq+gUG754FDF6BPQFM778HNvnTeSNOePKLq0wlbtLOCAi/ib7+K+S7gcmZtNCDOfbwHUk2Nf19JpDmTZla9llWKqkelgB6g/UH614DrhcVQssSSc1+llEPN7owBGxRNIRe16aVZqCL1x4FAjOvfg1PvCXr5VdUbXUgplXrKFzYw9bzp7CG0e3T+uqTI1aWP+nwc8COKuIAiRdAlwCMGr0gUUc0vaBr33veaZM72Vz10g+f8FRzDp6J394anfZZVVHh1j/1WPp6O5j2jUv0LluB72zxpRdVWEqd0kYEWfuiwKyV/4sBJgw8bCy3oBtTZoyvReAA6f0seCc3/GrJ8Y6sHajNm4kO+aOZ+yTW/lduwRWUNqjOXmGNZi9xc7tHWzf1vHm5+UPT+CIY3eWXFV1dGzpo6O7D6jfMRzz9FZ6Z4wquaqClTS9jN/8vBtXfPqnzDtuI5PG7+S2b9zGd+46ifsePqbssirj9VdH8uWP1Z/c6u+DMz+ymXef6ZsUA0Zu7mXqv7wItUAB2049kO0ntdfQxcpdEu4tSbdSHyE/RdJLwJci4oZWna9IX7l+n1wNJ2v6O3r415+sKbuMyuo5fAzrv/rOsstoraoGluqvy/kocGREXCnpcODQiHi00X4RcWFBNZpZ1VT40Zx/Bt4DDATQVuD6llVkZpWmyL8ULc8l4SkRcZKkJwAi4nVJBxRfipklo4IT+A3ozd7MGlB/vTQteazRzFJRVqd7nkvCbwD3AIdI+gr1qWWuamlVZlZtVR3WEBHflbSc+hQzAv4sIvzmZ7P9VYv6p/LIc5fwcGA78MPB30XEi60szMwqrKqBBfyI37+MYjQwG1gD/EEL6zKzClNJvdh5Lgn/cPB6NovD3wyxuZlZyzQ90j0iHpd0SiuKMbNEVPWSUNLfDVrtAE4CXm5ZRWZWbVXudAcmDPrcR71P667WlGNmSahiYGUDRidExGX7qB4zS0HVAkvSyIjok7RgXxZkZtUmqnmX8FHq/VUrJP0AuAN4c0rJiLi7xbWZWRUV1IclaRb1l9RMqx+VhRFxbaN98vRhjQZeoz6H+8B4rAAcWGb7q2IuCfuAv89GHkwAlkt6ICJWDbVDo8A6JLtDuJLfB1Wx5ZpZmgpIgIjYAGzIPm+VtBqYCexRYI0AxrP7N6o5sMz2Y01cEk6RtGzQ+sLsxTNvPV79lYAnAr9sdLBGgbUhIq7MXZaZ7T/yB1ZXRMxvtIGk8dSHSn0uIrY02rZRYLXbu2rNrAhR3F1CSZ3Uw+q7eW7kNQqsPymmJDNrO8XcJRRwA7A6Ir6WZ58hJ/CLiE17X5KZtaOC5nRfAFwMnCVpRbZ8oNEOfi+hmTWvmLuEj9Bk15MDy8ya06Lpj/NwYJlZU0S1Z2swM3sLB5aZpcOBZWbJcGCZWRIqPuOomdlbObDMLBVVnMBvn+sbJ3777lFll1FZ75txQtklVN8tZRewf/AloZmlwQNHzSwpDiwzS4FHuptZUlQrJ7EcWGbWHPdhmVlKfEloZulwYJlZKtzCMrN0OLDMLAkFvjWnWQ4sM2uKx2GZWVrC47DMLBFuYZlZGjxw1MxS4k53M0uGA8vM0hC4093M0uFOdzNLhwPLzFLggaNmlo4IT+BnZglxC8vMUuFLQjNLQwC+JDSzZJTUwuoo57RmljJFvmXY40g3SnpF0so853VgmVnTVItcSw7fBs7Je14Hlpk1J5pYhjtUxBJgU95Tuw/LzJpSHziauxNriqRlg9YXRsTCPT23A8vMmpd/toauiJhf1GkdWGbWtCZaWIVyYO3i0PHbuOrsBzl47A4CuHPlXG5e8a6yy6qc+Wds4ZP/82VGdAT33TqZ26+bVnZJlaGeGtOvfB711VA/dJ8yidfPn152WcVpxxlHJc0CbgKmUf/1FkbEta06X1H6auLqpaex+tWpjO3s4fYL7+RnLx7G2k2Tyy6tMjo6gk9ftZ7LLziSrg2dfPPe5/jFokm8+NzoskurhOgUG754FDF6BPQFM778HNvnTeSNOePKLq0gxT1LKOlW4AzqfV0vAV+KiBuG2r6VLaw+4O8j4nFJE4Dlkh6IiFUtPOde69o+jq7t9X9Y23sPYO2mg5g2vtuBNcg7T9zOyy8cwMYXRwGw+PsH8p73/c6BNUCqhxWg/kD9Ue+pbicFXRJGxIXNbN+ywIqIDcCG7PNWSauBmUClA2uwGRO2cNwhXTy10Zc7gx18aC+vvnzAm+tdGzo59qTtJVZUQbVg5hVr6NzYw5azp/DG0e3SuqL9X6Qq6QjgROCX++J8RRjT2cs15y7inx5eQHfPAcPvYDZYh1j/1WPp6O5j2jUv0LluB72zxpRdVXFK6nRv+cBRSeOBu4DPRcSW3fz8EknLJC3r297d6nJyGdnRz9fPXcSP1hzDT359ZNnlVM5rGzuZOqPnzfUp03vp2tBZYkXVVRs3kh1zxzP2ya1ll1KsggaONqulgSWpk3pYfTci7t7dNhGxMCLmR8T8kWOr0GwOrnzvYtZuOpCbnphXdjGVtGbFWGbO7mHarDcY2VnjjA9v5hc/nlR2WZXRsaWPju4+oH7HcMzTW+mdMarkqoqlWi3XUrRW3iUUcAOwOiK+1qrzFO3EGRv50HHP8mzXZO686HYArv3ZKSx94R0lV1YdtX5x/RUzueqWtXSMgB/fNpn/fNYd7gNGbu5l6r+8CLVAAdtOPZDtJ7VRoAfNDBwtVCv7sBYAFwNPS1qRffeFiLi3hefca0+8PJ3jr/1U2WVU3mMPTeSxhyaWXUYl9Rw+hvVffWfZZbSMiPYbOBoRj9B+N3PNDPxeQjNLiAPLzJLQpn1YZtamWnEHMA8Hlpk1KXxJaGaJCBxYZpYQ92GZWSrabhyWmbUxB5aZJSEC+n2X0MxS4RaWmSXDgWVmSQigoDndm+XAMrMmBYT7sMwsBYE73c0sIe7DMrNkOLDMLA1++NnMUhGAp5cxs2S4hWVmafCjOWaWioDwOCwzS4ZHuptZMtyHZWZJiPBdQjNLiFtYZpaGIPr7SzmzA8vMmuPpZcwsKSUNa+go5axmlqwAoha5luFIOkfSGknPS/r8cNs7sMysOZFN4JdnaUDSCOB64P3AXOBCSXMb7eNLQjNrWkGd7icDz0fEWgBJtwEfBlYNtYOipNuTuyPpVeA/y65jkClAV9lFVJj/PsOr2t/oHRExdW8OIOl+6r9XHqOBnYPWF0bEwuw45wPnRMTHs/WLgVMi4jNDHaxSLay9/UMWTdKyiJhfdh1V5b/P8NrxbxQR55R1bvdhmVlZ1gOzBq0fln03JAeWmZXlMWCOpNmSDgAuAH7QaIdKXRJW0MKyC6g4/32G57/RECKiT9JngEXACODGiHim0T6V6nQ3M2vEl4RmlgwHlpklw4G1G80+LrC/kXSjpFckrSy7liqSNEvSTyWtkvSMpM+WXVO7cB/WLrLHBZ4F/hR4ifqdjAsjYsjRt/sbSacD24CbIuL4suupGknTgekR8bikCcBy4M/8b2jvuYX1dm8+LhARPcDA4wKWiYglwKay66iqiNgQEY9nn7cCq4GZ5VbVHhxYbzcTWDdo/SX8j832kKQjgBOBX5ZbSXtwYJm1iKTxwF3A5yJiS9n1tAMH1ts1/biA2a4kdVIPq+9GxN1l19MuHFhv1/TjAmaDSRJwA7A6Ir5Wdj3txIG1i4joAwYeF1gN3D7c4wL7G0m3Aj8H3inpJUkfK7umilkAXAycJWlFtnyg7KLagYc1mFky3MIys2Q4sMwsGQ4sM0uGA8vMkuHAMrNkOLASIqk/u0W+UtIdksbuxbG+nb21BEnfavQ+OElnSDptD87xgqS3vV1lqO932WZbk+f6R0mXNVujpcWBlZYdEXFCNkNCD/DJwT+UtEdTXkfEx4eZSeAMoOnAMiuaAytdS4Gjs9bPUkk/AFZJGiHpakmPSXpK0iegPvpa0nXZPF8/AQ4ZOJCkxZLmZ5/PkfS4pCclPZg9vPtJ4L9lrbs/ljRV0l3ZOR6TtCDb92BJP87mgPoWoOF+CUnfk7Q82+eSXX52Tfb9g5KmZt8dJen+bJ+lko4t4o9piYgIL4kswLbsvyOB7wOfot766QZmZz+7BPhi9nkUsAyYDZwHPEB9sv8ZwGbg/Gy7xcB8YCr1mSoGjjU5++8/ApcNquMW4I+yz4dTfwQF4BvAP2SfzwUCmLKb3+OFge8HnWMMsBI4OFsP4KPZ538Arss+PwjMyT6fAjy0uxq9tOfit+akZYykFdnnpdSfVzsNeDQifpN9fzbwroH+KWASMAc4Hbg1IvqBlyU9tJvjnwosGThWRAw159V7gbn1R+YAmJjNTHA69WAkIn4k6fUcv9Olkj6SfZ6V1foaUAP+Pfv+ZuDu7BynAXcMOveoHOewNuHASsuOiDhh8BfZ/3G7B38F/G1ELNpluyKfZesATo2Iwa8gZ1CI5CLpDOrh956I2C5pMfVXm+9OZOfdvOvfwPYf7sNqP4uAT2XTmyDpGEnjgCXAX2R9XNOBM3ez7y+A0yXNzvadnH2/FZgwaLsfA387sCJpIECWABdl370fOGiYWicBr2dhdSz1Ft6ADmCglXgR8EjU55T6jaQ/z84hSfOGOYe1EQdW+/kWsAp4PHtJxL9Rb0nfAzyX/ewm6rMtvEVEvEq9D+xuSU/y+0uyHwIfGeh0By4F5med+qv4/d3KL1MPvGeoXxq+OEyt9wMjJa0G/hf1wBzQDZyc/Q5nAVdm338U+FhW3zN4+ur9imdrMLNkuIVlZslwYJlZMhxYZpYMB5aZJcOBZWbJcGCZWTIcWGaWjP8PZyCBGOWKg3YAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Classification Report"
      ],
      "metadata": {
        "id": "Y6dC8W72CJuV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import classification_report\n",
        "print(classification_report(y_true, y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sIPo0L_MCUil",
        "outputId": "10beca49-a2cc-4b1c-dac3-670b72b1d867"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.57      0.57      0.57         7\n",
            "           1       0.71      0.56      0.63         9\n",
            "           2       0.43      0.60      0.50         5\n",
            "\n",
            "    accuracy                           0.57        21\n",
            "   macro avg       0.57      0.58      0.57        21\n",
            "weighted avg       0.60      0.57      0.58        21\n",
            "\n"
          ]
        }
      ]
    }
  ]
}