NN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fb0a7333",
   "metadata": {},
   "source": [
    "###  NN-Model based classification using Pytorch with a tabular vector borne disease dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3de90eb2-027e-4386-afc1-b7c39c1485b3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-16T09:22:58.331676Z",
     "iopub.status.busy": "2023-07-16T09:22:58.331373Z",
     "iopub.status.idle": "2023-07-16T09:22:58.335256Z",
     "shell.execute_reply": "2023-07-16T09:22:58.334258Z",
     "shell.execute_reply.started": "2023-07-16T09:22:58.331650Z"
    },
    "tags": []
   },
   "source": [
    "#### TEAM MEMBERS:\n",
    "* #### Nnamdi Ogbonnaya Odii     (11160333)\n",
    "* #### Maryam Jahangir           (11159172)\n",
    "* #### Niels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7ced39b",
   "metadata": {},
   "source": [
    "#### 1.0 The problem\n",
    "\n",
    "The \"Classification with a Tabular Vector Borne Disease Dataset\" competition is part of Kaggle's Playground Series for the year 2023. The competition aims to develop robust neural networok model that can accurately classify instances of vector-borne diseases based on a tabular dataset containing various features. The goal is to create models that can generalize well to unseen data and provide accurate predictions on the test set.\n",
    "\n",
    "Our task is to train the neural network model,evaluate its performance using the map@k and then predict likely top3 vector borne diseases, given some features."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fe88485",
   "metadata": {},
   "source": [
    "#### 1.0 Introduction\n",
    "\n",
    "In this report, we present a detailed analysis of a neural network model for disease classification. The aim of the study is to optimize the model's hyperparameters using Optuna, evaluate the performance of the model by using the mean average precision at K (MAPK) metric, and utilize the trained and optimized model to generate predictions for the test dataset. The dataset used for training and testing consists of various diseases and their corresponding symptoms.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "372fc2b8",
   "metadata": {},
   "source": [
    "#### 2.0 The Dataset\n",
    "\n",
    "The dataset for this competition (both train and test) was generated from a deep learning model trained on the vector-borne-disease-prediction[1]. Our task is to predict the target prognosis of Vector-borne diseases.\n",
    "\n",
    "Vectors[2] are living organisms that can transmit infectious pathogens between humans, or from animals to humans. Many of these vectors are bloodsucking insects, which ingest disease-producing microorganisms during a blood meal from an infected host (human or animal) and later transmit it into a new host, after the pathogen has replicated. Often, once a vector becomes infectious, they are capable of transmitting the pathogen for the rest of their life during each subsequent bite/blood meal.\n",
    "\n",
    "Vector-borne diseases[3] are human illnesses caused by parasites, viruses and bacteria that are transmitted by vectors. Every year there are more than 700,000 deaths from diseases such as malaria, dengue, schistosomiasis, human African trypanosomiasis, leishmaniasis, Chagas disease, yellow fever, Japanese encephalitis and onchocerciasis.\n",
    "\n",
    "Two datasets were provided for this task: the train dataset, labeled as train.csv and the test.csv, for prediction purpose.\n",
    "\n",
    "#### 2.0.1 Train.csv\n",
    "* The train dataset consists of 707x64 features(symptoms) and 707x1 labels(Prognosis). \n",
    "* The features are binary digits (0s and 1s) where 0 represents absence of a particular symptom and 1 represents its presence. Each feature column is depicted by a particular symptom. \n",
    "* The labels are string-values of different kinds of vector borne diseases.This target variable serves as the ground truth for training the classification model.\n",
    "\n",
    "#### 2.0.2 Test.csv\n",
    "* The Test.csv dataset consists of 303x64 features(symptoms).\n",
    "* The test dataset is also a tabular dataset that follows a similar structure to the training dataset.\n",
    "* However, unlike the training dataset, the test dataset does not contain the target variable.\n",
    "* The purpose of the test dataset is to evaluate the performance and generalization ability of the Neural network model developed. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daf258c0",
   "metadata": {},
   "source": [
    "#### 3.0   The Methods\n",
    "#### 3.0.1 Data Preprocessing:\n",
    "\n",
    "We did the following preprocessing activities on the data:\n",
    "* Loaded dataset:The dataset is loaded from a CSV file and split into features (X) and labels (y).\n",
    "* Encoding:We used the OrdinalEncoder class to encode the categorical label column into numerical representation.\n",
    "\n",
    "#### 3.0.2 Feature Engingeering:\n",
    "The 64 given features represent only one single symptom each. However, illnesses often come along with combinations of symptoms. Also, one single symptom may apply to a whole range of illnesses. The applied feature engineering intents to combine the most likely symptoms of an illness to get a \"Fingerprint\" of every disease. This fingerprint will be added as an additional column, resulting in 11 features more. Taking the four most likely symptoms, this results in 14 possible combinations of AND and OR, referring to 8 ways to arrange the four symptoms and 6 more by setting logical paranthesis. Out of these 14 combinations, the one that bests represents the certain disease is selected. For the validation, every combination is applied to the whole dataset. An ideal fingerprint combination of symptoms results in a logical True for all records that have the prognosis of the certain illness and results in a logical False for all records that have a different prognosis. However, an ideal fingerprint is not achievable. Hence, the number of false negatives (False although prognosis is positive) and false positives (True although prognosis was different illness) is counted. The algorithm intents to reduce the number of false negatives as much as possible. However, just validating on false negatives leads to the OR-OR-OR combination being always chosen and respectively a high number of false positives. As a consequence, the threshold of n times false negatives limits the number of acceptable false positives to a multiple of the counted false negatives. This validation can be seen as a method of feature reduction. The number of features increased by 11 from 64 to 75 respectively.\n",
    "\n",
    "#### 3.0.3 Hyperparameter Tuning:\n",
    "\n",
    "To optimize the model's performance, we employed Optuna, a hyperparameter tuning tool. Optuna was used to search for the best combination of hyperparameters, including the choice of optimizer, hidden size, learning rate, and number of epochs. By iterating over multiple trials, Optuna helped identify the hyperparameters that maximize the model's validation MAPK score.\n",
    "\n",
    "* Optimizer: The optimizer determines the update rule for adjusting the weights and biases of the neural network during the training process. The three optimizers used in the code are:\n",
    "*  a. Adam: Adam (Adaptive Moment Estimation) is an adaptive learning rate optimization algorithm that combines the benefits of both AdaGrad and RMSProp. It maintains adaptive learning rates for different parameters and performs well in a wide range of tasks.\n",
    "*  b. SGD: SGD (Stochastic Gradient Descent) is a classic optimization algorithm that updates the model parameters by computing gradients on randomly selected small batches of data. It is a simple and widely used optimizer but can be sensitive to the learning rate.\n",
    "*  c. RMSprop: RMSprop (Root Mean Square Propagation) is an adaptive learning rate method that divides the learning rate by the exponentially decaying average of squared gradients. It tends to converge faster and overcome some limitations of SGD.\n",
    "* Hidden Size: The hidden size refers to the number of units or neurons in the hidden layer(s) of the neural network. It determines the model's capacity to learn complex representations and capture intricate relationships in the data. In the provided code, the hidden size can take values of 32, 64, or 128, offering different levels of model complexity.\n",
    "* Learning Rate: The learning rate controls the step size of weight updates during training. It determines how quickly or slowly the model learns from the data. A high learning rate may cause the model to overshoot optimal weights, leading to instability, while a low learning rate may slow down the learning process. The provided code explores learning rates ranging from 1e-5 to 1e-1 on a logarithmic scale, covering a wide range of learning rates.\n",
    "* Number of Epochs: An epoch is a complete pass through the entire training dataset during the training process. The number of epochs defines how many times the model goes through the entire dataset. It influences the amount of training time and the model's ability to converge to an optimal solution. In the provided code, the number of epochs ranges from 10 to 200 with a step of 10, allowing for exploration of different training durations.\n",
    "\n",
    "The choice of optimizers, hidden size, learning rate, and number of epochs can significantly impact the model's ability to learn and generalize patterns from the data, achieve convergence, and avoid overfitting. \n",
    "\n",
    "#### 3.0.4 Neural Network Architecture:\n",
    "\n",
    "The code implements a neural network model architecture specifically designed for disease classification.The architecture consists of three fully connected layers.\n",
    "* Batch normalization is applied after the first and second hidden layers to improve training efficiency. It addresses the issue of internal covariate shift, ensuring stable activations during training and enhancing gradient propagation\n",
    "* The ReLU activation function is used after each hidden layer. It introduces non-linearity, enabling the model to learn complex patterns and avoid the vanishing gradient problem.\n",
    "\n",
    "#### 3.0.5 Model Training and Evaluation:\n",
    "\n",
    "The performance of our trained and optimized neural network model was evaluated using the MAPK metric. The MAPK score measures the average precision at various levels of recall, providing an overall assessment of the model's ability to rank the predicted diseases correctly. In the code:\n",
    "* The training data is divided into five folds using Stratified K-Fold cross-validation. For each fold, the model is trained using    the specified hyperparameters and evaluated on the validation set.\n",
    "* Early stopping is implemented to prevent overfitting by monitoring the validation loss.\n",
    "* The MAPK score is calculated for each fold and averaged to obtain the final validation MAPK.\n",
    "\n",
    "#### 3.0.6 Best Model Selection and Training:\n",
    "\n",
    "* The best hyperparameters and validation accuracy are determined based on the optimization results.\n",
    "* The best model is created using the optimal hyperparameters and trained on the full dataset.\n",
    "* The model is trained for the specified number of epochs using the Adam optimizer and cross-entropy loss.\n",
    "\n",
    "#### 3.0.7 Test Set Evaluation:\n",
    "To evaluate the model's performance on unseen data, a test dataset was utilized. The test dataset consists of symptom information for a set of patients with unknown diseases. The trained model was applied to make predictions on this dataset.\n",
    "\n",
    "The test dataset was preprocessed in the same manner as the training data, including feature scaling using the MinMaxScaler. The scaled test features were converted to PyTorch tensors for compatibility with the model.\n",
    "\n",
    "The best-trained model, determined by the optimization process, was used to generate predictions for the test dataset. The top 3 predicted diseases were extracted for each patient, based on the highest probability scores obtained from the model.\n",
    "\n",
    "The predicted disease labels were then decoded using the label encoder to obtain the actual disease names. These predictions provide insights into potential diseases for each patient in the test dataset. Example Predictions: Predicted Diseases: \n",
    "['Zika' 'Yellow_Fever' 'Japanese_encephalitis']."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f36038e",
   "metadata": {},
   "source": [
    "#### 4.0 Results\n",
    "Our trained and optimized neural network model achieved a validation MAPK score of [0.34035394399493885]. This indicates that our model performs well in ranking and predicting the vector-borne diseases correctly, considering the given features.\n",
    "\n",
    "During the training and hyperparameter tuning process, we explored various combinations of hyperparameters using Optuna. By leveraging Optuna's capabilities, we identified the hyperparameters that provided the best performance on the validation set.\n",
    "\n",
    "* Optimizer: RMSprop\n",
    "* Hidden Size: 64\n",
    "* Learning Rate: 0.0004992938012543013\n",
    "* Number of Epochs: 30\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28c0476a",
   "metadata": {},
   "source": [
    "#### 5.0 The discussion\n",
    "\n",
    "The obtained hyperparameters suggest that RMSprop is the preferred optimizer for this classification task. The hidden size of 64 indicates that a moderate number of hidden units is sufficient for effective disease classification. The chosen learning rate and number of epochs are suitable for achieving a good balance between model performance and training time.\n",
    "\n",
    "The validation MAPK score of 0.34035394399493885 indicates that the model performs reasonably well in predicting diseases based on symptoms. However, further improvements can be explored by considering more complex architectures, ensembling methods, or increasing the size of the dataset.\n",
    "\n",
    "A comparison of the results with and without Feature Engineering shows only slightly differences. Looking at the feature selection, there are a lot of false diagnosis from the chosen fingerprints, meaning that the prints are blurred to some extend. Higher fingerprint accuracy could be obtained by including more symptoms, e.g. by increasing to the 5 most likely ones or including the least likely ones.\n",
    "\n",
    "A submission to Kaggle delivers the results shown in below picture:\n",
    "\n",
    "![Submission](./Submission_Kaggle_NN.JPG)\n",
    "\n",
    "_Image 1: Results of the Kaggle Submission_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6241ee3",
   "metadata": {},
   "source": [
    "#### 6.0 The conclusion \n",
    "\n",
    "In this study, we successfully optimized the hyperparameters of a neural network model for disease classification using Optuna. The best model achieved a validation MAPK score of 0.34035394399493885. The findings suggest that the chosen hyperparameters and model architecture are effective for the given task.\n",
    "\n",
    "Additionally, the model demonstrated promising performance when making predictions on unseen test data. The top 3 predicted diseases for each patient provide valuable insights into potential diagnoses. Further research and evaluation can be conducted to assess the model's generalizability and reliability in real-world healthcare scenarios."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb46b58f",
   "metadata": {},
   "source": [
    "#### 7.0 The References:\n",
    "\n",
    "The authors would like to thank Yuganshu Wadhwa for supporting the implementation of the Feature Engineering model in Python\n",
    "\n",
    "[1] Dataset: Vector-borne-disease-prediction. https://www.kaggle.com/datasets/richardbernat/vector-borne-disease-\n",
    "\n",
    "[2] dataset: vectors. https://www.who.int/news-room/fact-sheets/detail/vector-borne-diseases#:~:text=and%20community%20mobilisation.-,Vectors,-Vectors%20are%20living\n",
    "\n",
    "[3] Dataset: Vector borne disease. https://www.who.int/news-room/fact-sheets/detail/vector-borne-diseases#:~:text=bite/blood%20meal.-,Vector%2Dborne%20diseases,-Vector%2Dborne%20diseases\n",
    "\n",
    "[4] Optuna: A Hyperparameter Optimization Framework. https://optuna.org/\n",
    "\n",
    "[5] Sklearn StratifiedKFold Documentation. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html\n",
    "\n",
    "[6] PyTorch Documentation. https://pytorch.org/docs/stable/index.html\n",
    "\n",
    "[7] Pandas Documentation. https://pandas.pydata.org/docs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a1d4383e-847f-40ca-bd31-d87b12e36462",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-16T00:24:31.581070Z",
     "iopub.status.busy": "2023-07-16T00:24:31.580819Z",
     "iopub.status.idle": "2023-07-16T00:24:31.588868Z",
     "shell.execute_reply": "2023-07-16T00:24:31.587687Z",
     "shell.execute_reply.started": "2023-07-16T00:24:31.581043Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "features = 'yes'\n",
    "#features = 'no'\n",
    "\n",
    "if features == 'yes':\n",
    "    initial_size = 75\n",
    "    train_path = 'train_features.csv'\n",
    "    test_path = 'test_features.csv'\n",
    "else:\n",
    "    initial_size = 64\n",
    "    train_path = 'train.csv'\n",
    "    test_path = 'test.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec60232e",
   "metadata": {},
   "source": [
    "#### 1.0 Import the necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3ba88647-e0a5-47fc-94e7-9faedfac19a4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-16T00:24:31.591578Z",
     "iopub.status.busy": "2023-07-16T00:24:31.590898Z",
     "iopub.status.idle": "2023-07-16T00:24:34.097232Z",
     "shell.execute_reply": "2023-07-16T00:24:34.096429Z",
     "shell.execute_reply.started": "2023-07-16T00:24:31.591550Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.preprocessing import LabelEncoder, MinMaxScaler\n",
    "import optuna\n",
    "from spotPython.utils.metrics import mapk_score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b953c3a",
   "metadata": {},
   "source": [
    "#### 2.0 Load the Training dataset & Make neural network architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3e7ea4a2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-16T00:24:34.103531Z",
     "iopub.status.busy": "2023-07-16T00:24:34.101281Z",
     "iopub.status.idle": "2023-07-16T00:24:34.110681Z",
     "shell.execute_reply": "2023-07-16T00:24:34.110053Z",
     "shell.execute_reply.started": "2023-07-16T00:24:34.103506Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set a random seed for reproducibility\n",
    "torch.manual_seed(42)\n",
    "torch.cuda.manual_seed(42)\n",
    "torch.cuda.manual_seed_all(42)  # Set random seed for all GPUs if using multiple GPUs\n",
    "torch.backends.cudnn.deterministic = True\n",
    "torch.backends.cudnn.benchmark = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1c9b41c5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-16T00:24:34.116426Z",
     "iopub.status.busy": "2023-07-16T00:24:34.114145Z",
     "iopub.status.idle": "2023-07-16T00:24:34.253315Z",
     "shell.execute_reply": "2023-07-16T00:24:34.252573Z",
     "shell.execute_reply.started": "2023-07-16T00:24:34.116372Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    sudden_fever  headache  mouth_bleed  nose_bleed  muscle_pain  joint_pain  \\\n",
      "id                                                                             \n",
      "0              1         1            0           1            1           1   \n",
      "1              0         0            0           0            0           0   \n",
      "2              0         1            1           1            0           1   \n",
      "3              0         0            1           1            1           1   \n",
      "4              0         0            0           0            0           0   \n",
      "\n",
      "    vomiting  rash  diarrhea  hypotension  ...  Zika_print  \\\n",
      "id                                         ...               \n",
      "0          1     0         1            1  ...           0   \n",
      "1          1     0         1            0  ...           0   \n",
      "2          1     1         1            1  ...           0   \n",
      "3          0     1         0            1  ...           0   \n",
      "4          0     0         1            0  ...           0   \n",
      "\n",
      "    Rift_Valley_fever_print  West_Nile_fever_print  Malaria_print  \\\n",
      "id                                                                  \n",
      "0                         0                      1              0   \n",
      "1                         0                      0              0   \n",
      "2                         1                      1              0   \n",
      "3                         1                      1              0   \n",
      "4                         0                      0              0   \n",
      "\n",
      "    Chikungunya_print  Plague_print  Dengue_print  Yellow_Fever_print  \\\n",
      "id                                                                      \n",
      "0                   1             0             0                   0   \n",
      "1                   0             0             0                   0   \n",
      "2                   0             0             1                   1   \n",
      "3                   1             0             1                   0   \n",
      "4                   0             0             0                   0   \n",
      "\n",
      "    Japanese_encephalitis_print          prognosis  \n",
      "id                                                  \n",
      "0                             1       Lyme_disease  \n",
      "1                             0          Tungiasis  \n",
      "2                             0       Lyme_disease  \n",
      "3                             1               Zika  \n",
      "4                             0  Rift_Valley_fever  \n",
      "\n",
      "[5 rows x 76 columns]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Define the neural network architecture\n",
    "#This block defines a neural network model architecture with three fully connected layers (fc1, fc2, and fc3). \n",
    "#The batch normalization layers (bn1 and bn2) are added after each fully connected layer to improve training stability\n",
    "\n",
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self, input_size=64, num_classes=11, hidden_size=64):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.bn1 = nn.BatchNorm1d(hidden_size)\n",
    "        self.fc2 = nn.Linear(hidden_size, hidden_size)\n",
    "        self.bn2 = nn.BatchNorm1d(hidden_size)\n",
    "        self.fc3 = nn.Linear(hidden_size, num_classes)\n",
    "        \n",
    "#The forward method defines the flow of data through the layers of the neural network. The input x is passed through \n",
    "#the fully connected layers (fc1, fc2, and fc3) interspersed with batch normalization layers (bn1 and bn2). \n",
    "#ReLU activation is applied after each fully connected layer, except for the last layer. The method returns the final \n",
    "#output of the network.   \n",
    "    \n",
    "    def forward(self, x):\n",
    "        out = self.fc1(x)\n",
    "        out = self.bn1(out)\n",
    "        out = nn.functional.relu(out)\n",
    "        out = self.fc2(out)\n",
    "        out = self.bn2(out)\n",
    "        out = nn.functional.relu(out)\n",
    "        out = self.fc3(out)\n",
    "        return out\n",
    "\n",
    "dataset = pd.read_csv(train_path,index_col='id')\n",
    "print(dataset.head())\n",
    "\n",
    "# Step 1: Split the dataset into features (X) and labels (y)\n",
    "X = dataset.iloc[:, :-1].values\n",
    "y = dataset.iloc[:, -1].values\n",
    "\n",
    "label_encoder = LabelEncoder()\n",
    "y_encoded = label_encoder.fit_transform(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "097589b9",
   "metadata": {},
   "source": [
    "#### 3.0 Hyperparameter Tuning with Optuna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a820e7b0-5fd8-41fe-98d7-15733d78f8e6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-16T00:24:34.259346Z",
     "iopub.status.busy": "2023-07-16T00:24:34.257318Z",
     "iopub.status.idle": "2023-07-16T00:24:39.057752Z",
     "shell.execute_reply": "2023-07-16T00:24:39.056869Z",
     "shell.execute_reply.started": "2023-07-16T00:24:34.259322Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:34,271] A new study created in memory with name: no-name-7dd7d348-50ed-4b0f-9348-f43ba466587b\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 1, Epoch [45/45], Val Loss: 2.0675\n",
      "Fold: 2, Epoch [45/45], Val Loss: 2.1136\n",
      "Fold: 3, Epoch [45/45], Val Loss: 1.9370\n",
      "Fold: 4, Epoch [45/45], Val Loss: 2.0728\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:34,718] Trial 0 finished with value: 0.33643991609229845 and parameters: {'optimizer': 'RMSprop', 'hidden_size': 32, 'learning_rate': 0.0006676307305247338, 'epochs': 45}. Best is trial 0 with value: 0.33643991609229845.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 5, Epoch [45/45], Val Loss: 2.0533\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.33643991609229845\n",
      "\n",
      "Fold: 1, Epoch [11/80], Val Loss: 2.4119\n",
      "Fold: 2, Epoch [26/80], Val Loss: 2.4048\n",
      "Fold: 3, Epoch [11/80], Val Loss: 2.4143\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:34,877] Trial 1 finished with value: 0.23241767389205203 and parameters: {'optimizer': 'SGD', 'hidden_size': 32, 'learning_rate': 3.553870073566271e-05, 'epochs': 80}. Best is trial 0 with value: 0.33643991609229845.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 4, Epoch [27/80], Val Loss: 2.4055\n",
      "Fold: 5, Epoch [15/80], Val Loss: 2.4253\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.23241767389205203\n",
      "\n",
      "Fold: 1, Epoch [23/95], Val Loss: 2.8908\n",
      "Fold: 2, Epoch [21/95], Val Loss: 2.7683\n",
      "Fold: 3, Epoch [23/95], Val Loss: 2.7325\n",
      "Fold: 4, Epoch [22/95], Val Loss: 2.9355\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:35,339] Trial 2 finished with value: 0.26339970476919833 and parameters: {'optimizer': 'Adam', 'hidden_size': 128, 'learning_rate': 0.008313469391907054, 'epochs': 95}. Best is trial 0 with value: 0.33643991609229845.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 5, Epoch [24/95], Val Loss: 2.8766\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.26339970476919833\n",
      "\n",
      "Fold: 1, Epoch [65/65], Val Loss: 2.1603\n",
      "Fold: 2, Epoch [65/65], Val Loss: 2.1960\n",
      "Fold: 3, Epoch [65/65], Val Loss: 2.1174\n",
      "Fold: 4, Epoch [65/65], Val Loss: 2.0944\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:36,482] Trial 3 finished with value: 0.2704516698298538 and parameters: {'optimizer': 'RMSprop', 'hidden_size': 128, 'learning_rate': 3.5522681768276785e-05, 'epochs': 65}. Best is trial 0 with value: 0.33643991609229845.\n",
      "[I 2023-07-16 02:24:36,625] Trial 4 finished with value: 0.2497938933839443 and parameters: {'optimizer': 'SGD', 'hidden_size': 32, 'learning_rate': 1.8204018430281336e-05, 'epochs': 60}. Best is trial 0 with value: 0.33643991609229845.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 5, Epoch [65/65], Val Loss: 2.2029\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.2704516698298538\n",
      "\n",
      "Fold: 1, Epoch [11/60], Val Loss: 2.4174\n",
      "Fold: 2, Epoch [11/60], Val Loss: 2.4136\n",
      "Fold: 3, Epoch [11/60], Val Loss: 2.4108\n",
      "Fold: 4, Epoch [34/60], Val Loss: 2.4009\n",
      "Fold: 5, Epoch [13/60], Val Loss: 2.4074\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.2497938933839443\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:36,844] Trial 5 finished with value: 0.23428700651505563 and parameters: {'optimizer': 'Adam', 'hidden_size': 64, 'learning_rate': 1.3598600169157745e-05, 'epochs': 35}. Best is trial 0 with value: 0.33643991609229845.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 1, Epoch [20/35], Val Loss: 2.4004\n",
      "Fold: 2, Epoch [13/35], Val Loss: 2.4114\n",
      "Fold: 3, Epoch [16/35], Val Loss: 2.4066\n",
      "Fold: 4, Epoch [14/35], Val Loss: 2.4029\n",
      "Fold: 5, Epoch [14/35], Val Loss: 2.3937\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.23428700651505563\n",
      "\n",
      "Fold: 1, Epoch [43/95], Val Loss: 2.3735\n",
      "Fold: 2, Epoch [11/95], Val Loss: 2.4153\n",
      "Fold: 3, Epoch [13/95], Val Loss: 2.4036\n",
      "Fold: 4, Epoch [21/95], Val Loss: 2.3886\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:37,201] Trial 6 finished with value: 0.22294166892608605 and parameters: {'optimizer': 'SGD', 'hidden_size': 128, 'learning_rate': 0.00012713335676495843, 'epochs': 95}. Best is trial 0 with value: 0.33643991609229845.\n",
      "[I 2023-07-16 02:24:37,311] Trial 7 finished with value: 0.23577314953551093 and parameters: {'optimizer': 'RMSprop', 'hidden_size': 32, 'learning_rate': 0.0031670713130407813, 'epochs': 10}. Best is trial 0 with value: 0.33643991609229845.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 5, Epoch [18/95], Val Loss: 2.4133\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.22294166892608605\n",
      "\n",
      "Fold: 1, Epoch [10/10], Val Loss: 2.1556\n",
      "Fold: 2, Epoch [10/10], Val Loss: 2.1487\n",
      "Fold: 3, Epoch [10/10], Val Loss: 2.0916\n",
      "Fold: 4, Epoch [10/10], Val Loss: 2.2040\n",
      "Fold: 5, Epoch [10/10], Val Loss: 2.1258\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.23577314953551093\n",
      "\n",
      "Fold: 1, Epoch [11/35], Val Loss: 2.4054\n",
      "Fold: 2, Epoch [13/35], Val Loss: 2.4058\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:37,513] Trial 8 finished with value: 0.22658335090659534 and parameters: {'optimizer': 'Adam', 'hidden_size': 64, 'learning_rate': 3.732371513541133e-05, 'epochs': 35}. Best is trial 0 with value: 0.33643991609229845.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 3, Epoch [11/35], Val Loss: 2.4034\n",
      "Fold: 4, Epoch [24/35], Val Loss: 2.3964\n",
      "Fold: 5, Epoch [11/35], Val Loss: 2.4109\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.22658335090659534\n",
      "\n",
      "Fold: 1, Epoch [75/75], Val Loss: 2.2374\n",
      "Fold: 2, Epoch [75/75], Val Loss: 2.1957\n",
      "Fold: 3, Epoch [75/75], Val Loss: 2.2313\n",
      "Fold: 4, Epoch [75/75], Val Loss: 2.2367\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-07-16 02:24:38,961] Trial 9 finished with value: 0.2319816535144674 and parameters: {'optimizer': 'Adam', 'hidden_size': 128, 'learning_rate': 4.998107030731937e-05, 'epochs': 75}. Best is trial 0 with value: 0.33643991609229845.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 5, Epoch [75/75], Val Loss: 2.2117\n",
      "---------------------------------------------\n",
      "5-fold Validation MAPK Val: 0.2319816535144674\n",
      "\n",
      "Best Hyperparameters: {'optimizer': 'RMSprop', 'hidden_size': 32, 'learning_rate': 0.0006676307305247338, 'epochs': 45}\n",
      "Best Validation MAPK: 0.33643991609229845\n"
     ]
    }
   ],
   "source": [
    "\n",
    "num_folds = 5\n",
    "# Initialize lists to store the training and validation accuracies\n",
    "train_accuracies = []\n",
    "val_accuracies = []\n",
    "\n",
    "# Initialize lists to store the training and validation losses\n",
    "train_losses = []\n",
    "val_losses = []\n",
    "\n",
    "mapk_folds = []\n",
    "skf = StratifiedKFold(n_splits=num_folds)\n",
    "\n",
    "def objective(trial):\n",
    "    optimizer_name = trial.suggest_categorical('optimizer', ['Adam', 'SGD', 'RMSprop'])\n",
    "    hidden_size = trial.suggest_categorical('hidden_size', [32, 64, 128])\n",
    "    learning_rate = trial.suggest_float('learning_rate', 1e-5, 1e-1, log=True)\n",
    "    num_epochs = trial.suggest_int('epochs', 10, 100, step=5)\n",
    "\n",
    "    fold_num = 0  # Track the fold number\n",
    "\n",
    "    for train_index, val_index in skf.split(X, y_encoded):\n",
    "        # Split the data into training and validation sets for each fold\n",
    "        X_train, X_val = X[train_index], X[val_index]\n",
    "        y_train, y_val = y_encoded[train_index], y_encoded[val_index]\n",
    "\n",
    "        # Convert data to PyTorch tensors(for compatibility with PyTorch's computational graph and tensor operations)\n",
    "        X_train_tensor = torch.tensor(X_train, dtype=torch.float32)\n",
    "        y_train_tensor = torch.tensor(y_train, dtype=torch.long)\n",
    "        X_val_tensor = torch.tensor(X_val, dtype=torch.float32)\n",
    "        y_val_tensor = torch.tensor(y_val, dtype=torch.long)\n",
    "\n",
    "        # Define the neural network architecture\n",
    "        model = NeuralNetwork(input_size=X.shape[1], num_classes=len(label_encoder.classes_), hidden_size=hidden_size)\n",
    "\n",
    "        # Define the loss function and optimizer\n",
    "        criterion = nn.CrossEntropyLoss()\n",
    "        optimizer_class = getattr(optim, optimizer_name)\n",
    "        optimizer = optimizer_class(model.parameters(), lr=learning_rate)\n",
    "        \n",
    "        # Initialize variables for early stopping\n",
    "        best_val_loss = float('inf')#to keep track of the lowest validation loss observed during training\n",
    "        epochs_without_improvement = 0\n",
    "        patience = 10  #If the validation loss does not improve for 'patience' consecutive epochs, training will be stopped to prevent overfitting or wasting computational resources\n",
    "\n",
    "        # Training loop\n",
    "        for epoch in range(num_epochs):\n",
    "            # Set the model to training mode\n",
    "            model.train()\n",
    "\n",
    "            # Forward pass\n",
    "            outputs = model(X_train_tensor)\n",
    "            loss = criterion(outputs, y_train_tensor)\n",
    "\n",
    "            # Backward and optimize\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            \n",
    "            # Evaluate the model on validation set\n",
    "            model.eval()\n",
    "            val_predictions = model(X_val_tensor)\n",
    "            val_loss = criterion(val_predictions, y_val_tensor)\n",
    "\n",
    "            # Check if validation loss has improved\n",
    "            if val_loss < best_val_loss:\n",
    "                best_val_loss = val_loss\n",
    "                epochs_without_improvement = 0\n",
    "            else:\n",
    "                epochs_without_improvement += 1\n",
    "\n",
    "            # Check early stopping condition\n",
    "            if epochs_without_improvement >= patience:\n",
    "                break\n",
    "\n",
    "        # Evaluate the final model on validation set\n",
    "        model.eval()\n",
    "        val_predictions = model(X_val_tensor)\n",
    "        val_predicted_probs = torch.softmax(val_predictions, dim=1)\n",
    "        mapk_folds.append(mapk_score(y_val, val_predicted_probs.cpu().detach().numpy()))\n",
    "\n",
    "        fold_num += 1  # Increment the fold number\n",
    "        print(f\"Fold: {fold_num}, Epoch [{epoch+1}/{num_epochs}], Val Loss: {val_loss.item():.4f}\")\n",
    "\n",
    "    mapk_final_val = np.average(mapk_folds)\n",
    "    print(f'---------------------------------------------\\n{num_folds}-fold Validation MAPK Val: {mapk_final_val}\\n')\n",
    "\n",
    "    return mapk_final_val\n",
    "\n",
    "# Create the study and optimize the objective function\n",
    "#A study is a container that manages the optimization process.The direction parameter is set to 'maximize', indicating\n",
    "#that the objective function should be maximized during the optimization process.\n",
    "#the optimize() method starts the optimization by repeatedly calling the objective function with different hyperparameter\n",
    "#configurations.The goal is to find the hyperparameter configuration that maximizes the objective function,after n_trials\n",
    "study = optuna.create_study(direction='maximize')\n",
    "study.optimize(objective, n_trials=10)\n",
    "\n",
    "# Get the best trial\n",
    "best_trial = study.best_trial\n",
    "\n",
    "# Print the best hyperparameters and validation accuracy(objective function)\n",
    "best_params = study.best_params\n",
    "best_val_accuracy = study.best_value\n",
    "\n",
    "# Create and load the best model using the best parameters\n",
    "best_model = NeuralNetwork(input_size=X.shape[1], num_classes=len(label_encoder.classes_), hidden_size=best_params['hidden_size'])\n",
    "# Train the best model on the full dataset\n",
    "X_tensor = torch.tensor(X, dtype=torch.float32)\n",
    "y_tensor = torch.tensor(y_encoded, dtype=torch.long)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(best_model.parameters(), lr=best_params['learning_rate'])#Consistent optimizer across different runs for reproducible & comparable results\n",
    "num_epochs = best_params['epochs']\n",
    "best_model.train()\n",
    "for epoch in range(num_epochs):\n",
    "    optimizer.zero_grad()#clears the gradients of all optimized parameters to prepare for the backward pass.\n",
    "    outputs = best_model(X_tensor)#computes the forward pass of the best_model using the input features X_tensor to obtain the predicted outputs\n",
    "    loss = criterion(outputs, y_tensor)#calculates the loss by comparing the predicted outputs with the ground truth labels y_tensor\n",
    "    loss.backward()#computes the gradients of the model parameters with respect to the loss, performing backpropagation\n",
    "    optimizer.step()#updates the model's parameters based on the computed gradients and the chosen optimization algorithm\n",
    "    \n",
    "# Set the best model to evaluation mode\n",
    "#This mode disables techniques like dropout and batch normalization that are typically used during training. \n",
    "#Instead, the model will use its learned parameters to make predictions consistently.\n",
    "best_model.eval()\n",
    "\n",
    "print(f\"Best Hyperparameters: {best_params}\")\n",
    "print(f\"Best Validation MAPK: {best_val_accuracy}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee94cc2b",
   "metadata": {},
   "source": [
    "### 4.0 Top 3 Predictions on the Test.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0c7e12e",
   "metadata": {},
   "source": [
    "#### 4.0.1 Load the Training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "67e041f6-0126-492a-a2d6-c16bea5dfeca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-16T00:24:39.063340Z",
     "iopub.status.busy": "2023-07-16T00:24:39.061296Z",
     "iopub.status.idle": "2023-07-16T00:24:39.081813Z",
     "shell.execute_reply": "2023-07-16T00:24:39.081225Z",
     "shell.execute_reply.started": "2023-07-16T00:24:39.063316Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     sudden_fever  headache  mouth_bleed  nose_bleed  muscle_pain  joint_pain  \\\n",
      "id                                                                              \n",
      "707             0         0            0           0            0           0   \n",
      "708             1         1            0           1            0           1   \n",
      "709             1         1            0           1            1           1   \n",
      "710             0         1            0           0            0           1   \n",
      "711             0         0            1           0            1           1   \n",
      "\n",
      "     vomiting  rash  diarrhea  hypotension  ...  Tungiasis_print  Zika_print  \\\n",
      "id                                          ...                                \n",
      "707         0     0         0            1  ...                0           1   \n",
      "708         1     1         1            1  ...                0           0   \n",
      "709         1     0         1            0  ...                1           0   \n",
      "710         1     1         0            0  ...                0           0   \n",
      "711         0     0         1            1  ...                0           0   \n",
      "\n",
      "     Rift_Valley_fever_print  West_Nile_fever_print  Malaria_print  \\\n",
      "id                                                                   \n",
      "707                        0                      0              0   \n",
      "708                        0                      1              0   \n",
      "709                        0                      1              0   \n",
      "710                        0                      0              0   \n",
      "711                        0                      1              1   \n",
      "\n",
      "     Chikungunya_print  Plague_print  Dengue_print  Yellow_Fever_print  \\\n",
      "id                                                                       \n",
      "707                  0             0             0                   0   \n",
      "708                  0             0             0                   0   \n",
      "709                  1             0             0                   0   \n",
      "710                  0             0             0                   0   \n",
      "711                  0             0             0                   0   \n",
      "\n",
      "     Japanese_encephalitis_print  \n",
      "id                                \n",
      "707                            0  \n",
      "708                            0  \n",
      "709                            0  \n",
      "710                            0  \n",
      "711                            0  \n",
      "\n",
      "[5 rows x 75 columns]\n"
     ]
    }
   ],
   "source": [
    "# Load the new dataset\n",
    "test_data = pd.read_csv(test_path, index_col='id')\n",
    "print(test_data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67b867c8",
   "metadata": {},
   "source": [
    "#### 4.0.2 Generating Predictions for Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c17df3ae-a6f7-4950-8840-6bcd1b57386d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-16T00:26:01.356209Z",
     "iopub.status.busy": "2023-07-16T00:26:01.355878Z",
     "iopub.status.idle": "2023-07-16T00:26:01.425208Z",
     "shell.execute_reply": "2023-07-16T00:26:01.424596Z",
     "shell.execute_reply.started": "2023-07-16T00:26:01.356182Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 3 Predictions:\n",
      "707 : ['Rift_Valley_fever' 'Tungiasis' 'Dengue']\n",
      "708 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "709 : ['West_Nile_fever' 'Lyme_disease' 'Zika']\n",
      "710 : ['Tungiasis' 'Rift_Valley_fever' 'Lyme_disease']\n",
      "711 : ['Zika' 'West_Nile_fever' 'Malaria']\n",
      "712 : ['Yellow_Fever' 'Japanese_encephalitis' 'Malaria']\n",
      "713 : ['Malaria' 'Plague' 'Japanese_encephalitis']\n",
      "714 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "715 : ['Lyme_disease' 'Japanese_encephalitis' 'Yellow_Fever']\n",
      "716 : ['Malaria' 'Zika' 'Plague']\n",
      "717 : ['West_Nile_fever' 'Japanese_encephalitis' 'Lyme_disease']\n",
      "718 : ['West_Nile_fever' 'Dengue' 'Rift_Valley_fever']\n",
      "719 : ['Rift_Valley_fever' 'Japanese_encephalitis' 'Dengue']\n",
      "720 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "721 : ['West_Nile_fever' 'Dengue' 'Japanese_encephalitis']\n",
      "722 : ['Zika' 'Tungiasis' 'Yellow_Fever']\n",
      "723 : ['West_Nile_fever' 'Malaria' 'Zika']\n",
      "724 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "725 : ['Chikungunya' 'West_Nile_fever' 'Rift_Valley_fever']\n",
      "726 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "727 : ['West_Nile_fever' 'Lyme_disease' 'Japanese_encephalitis']\n",
      "728 : ['West_Nile_fever' 'Chikungunya' 'Japanese_encephalitis']\n",
      "729 : ['Lyme_disease' 'West_Nile_fever' 'Rift_Valley_fever']\n",
      "730 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "731 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "732 : ['Chikungunya' 'Dengue' 'Rift_Valley_fever']\n",
      "733 : ['Chikungunya' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "734 : ['Tungiasis' 'Lyme_disease' 'Japanese_encephalitis']\n",
      "735 : ['Tungiasis' 'Rift_Valley_fever' 'Lyme_disease']\n",
      "736 : ['Lyme_disease' 'Japanese_encephalitis' 'Malaria']\n",
      "737 : ['Tungiasis' 'Lyme_disease' 'Japanese_encephalitis']\n",
      "738 : ['Tungiasis' 'Dengue' 'Rift_Valley_fever']\n",
      "739 : ['West_Nile_fever' 'Lyme_disease' 'Rift_Valley_fever']\n",
      "740 : ['Lyme_disease' 'Plague' 'Yellow_Fever']\n",
      "741 : ['Lyme_disease' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "742 : ['Lyme_disease' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "743 : ['Japanese_encephalitis' 'Lyme_disease' 'Tungiasis']\n",
      "744 : ['Tungiasis' 'Rift_Valley_fever' 'Zika']\n",
      "745 : ['Lyme_disease' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "746 : ['West_Nile_fever' 'Dengue' 'Zika']\n",
      "747 : ['Lyme_disease' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "748 : ['Chikungunya' 'Dengue' 'Tungiasis']\n",
      "749 : ['Plague' 'Yellow_Fever' 'Malaria']\n",
      "750 : ['Yellow_Fever' 'Tungiasis' 'Malaria']\n",
      "751 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "752 : ['West_Nile_fever' 'Dengue' 'Rift_Valley_fever']\n",
      "753 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "754 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "755 : ['Lyme_disease' 'Japanese_encephalitis' 'Yellow_Fever']\n",
      "756 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "757 : ['Lyme_disease' 'Rift_Valley_fever' 'Tungiasis']\n",
      "758 : ['Chikungunya' 'Rift_Valley_fever' 'Dengue']\n",
      "759 : ['Lyme_disease' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "760 : ['Plague' 'Malaria' 'Japanese_encephalitis']\n",
      "761 : ['Rift_Valley_fever' 'Tungiasis' 'Japanese_encephalitis']\n",
      "762 : ['Lyme_disease' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "763 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "764 : ['Chikungunya' 'Dengue' 'Rift_Valley_fever']\n",
      "765 : ['Chikungunya' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "766 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "767 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "768 : ['Yellow_Fever' 'Dengue' 'Japanese_encephalitis']\n",
      "769 : ['Plague' 'Lyme_disease' 'West_Nile_fever']\n",
      "770 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "771 : ['Tungiasis' 'Japanese_encephalitis' 'Rift_Valley_fever']\n",
      "772 : ['Lyme_disease' 'Japanese_encephalitis' 'Tungiasis']\n",
      "773 : ['West_Nile_fever' 'Tungiasis' 'Japanese_encephalitis']\n",
      "774 : ['West_Nile_fever' 'Lyme_disease' 'Rift_Valley_fever']\n",
      "775 : ['Rift_Valley_fever' 'Dengue' 'Tungiasis']\n",
      "776 : ['Tungiasis' 'Dengue' 'Japanese_encephalitis']\n",
      "777 : ['Chikungunya' 'Japanese_encephalitis' 'Dengue']\n",
      "778 : ['Lyme_disease' 'Japanese_encephalitis' 'Plague']\n",
      "779 : ['Lyme_disease' 'Japanese_encephalitis' 'Malaria']\n",
      "780 : ['Tungiasis' 'Lyme_disease' 'West_Nile_fever']\n",
      "781 : ['Plague' 'Lyme_disease' 'Yellow_Fever']\n",
      "782 : ['Chikungunya' 'Dengue' 'Rift_Valley_fever']\n",
      "783 : ['Plague' 'Lyme_disease' 'Malaria']\n",
      "784 : ['Japanese_encephalitis' 'Malaria' 'West_Nile_fever']\n",
      "785 : ['West_Nile_fever' 'Dengue' 'Japanese_encephalitis']\n",
      "786 : ['Lyme_disease' 'Japanese_encephalitis' 'Plague']\n",
      "787 : ['Malaria' 'Zika' 'Plague']\n",
      "788 : ['Plague' 'Lyme_disease' 'Yellow_Fever']\n",
      "789 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "790 : ['West_Nile_fever' 'Japanese_encephalitis' 'Dengue']\n",
      "791 : ['Rift_Valley_fever' 'Tungiasis' 'Dengue']\n",
      "792 : ['Plague' 'Lyme_disease' 'Malaria']\n",
      "793 : ['Lyme_disease' 'Japanese_encephalitis' 'Plague']\n",
      "794 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "795 : ['Dengue' 'Tungiasis' 'West_Nile_fever']\n",
      "796 : ['Plague' 'Lyme_disease' 'Zika']\n",
      "797 : ['Rift_Valley_fever' 'Yellow_Fever' 'Lyme_disease']\n",
      "798 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "799 : ['Zika' 'Yellow_Fever' 'Tungiasis']\n",
      "800 : ['Tungiasis' 'Rift_Valley_fever' 'Dengue']\n",
      "801 : ['West_Nile_fever' 'Chikungunya' 'Japanese_encephalitis']\n",
      "802 : ['Chikungunya' 'Japanese_encephalitis' 'Dengue']\n",
      "803 : ['Lyme_disease' 'Plague' 'Dengue']\n",
      "804 : ['West_Nile_fever' 'Lyme_disease' 'Dengue']\n",
      "805 : ['Lyme_disease' 'Japanese_encephalitis' 'Tungiasis']\n",
      "806 : ['Yellow_Fever' 'Lyme_disease' 'Tungiasis']\n",
      "807 : ['Malaria' 'Zika' 'West_Nile_fever']\n",
      "808 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "809 : ['Malaria' 'Japanese_encephalitis' 'Plague']\n",
      "810 : ['Lyme_disease' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "811 : ['Tungiasis' 'Chikungunya' 'Rift_Valley_fever']\n",
      "812 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "813 : ['Lyme_disease' 'Tungiasis' 'Japanese_encephalitis']\n",
      "814 : ['Lyme_disease' 'Rift_Valley_fever' 'West_Nile_fever']\n",
      "815 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "816 : ['Malaria' 'Plague' 'Japanese_encephalitis']\n",
      "817 : ['Tungiasis' 'Zika' 'Rift_Valley_fever']\n",
      "818 : ['Dengue' 'Lyme_disease' 'Tungiasis']\n",
      "819 : ['Plague' 'Malaria' 'Zika']\n",
      "820 : ['Tungiasis' 'Dengue' 'Japanese_encephalitis']\n",
      "821 : ['Zika' 'West_Nile_fever' 'Rift_Valley_fever']\n",
      "822 : ['Lyme_disease' 'Rift_Valley_fever' 'West_Nile_fever']\n",
      "823 : ['Japanese_encephalitis' 'Tungiasis' 'Rift_Valley_fever']\n",
      "824 : ['Chikungunya' 'Dengue' 'Rift_Valley_fever']\n",
      "825 : ['Rift_Valley_fever' 'West_Nile_fever' 'Dengue']\n",
      "826 : ['West_Nile_fever' 'Rift_Valley_fever' 'Dengue']\n",
      "827 : ['Tungiasis' 'Dengue' 'Lyme_disease']\n",
      "828 : ['Chikungunya' 'Dengue' 'Tungiasis']\n",
      "829 : ['Malaria' 'Plague' 'Lyme_disease']\n",
      "830 : ['Japanese_encephalitis' 'Lyme_disease' 'Rift_Valley_fever']\n",
      "831 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "832 : ['Malaria' 'Plague' 'Yellow_Fever']\n",
      "833 : ['Tungiasis' 'Rift_Valley_fever' 'Dengue']\n",
      "834 : ['Rift_Valley_fever' 'Tungiasis' 'Malaria']\n",
      "835 : ['Chikungunya' 'Tungiasis' 'Rift_Valley_fever']\n",
      "836 : ['Rift_Valley_fever' 'Dengue' 'Tungiasis']\n",
      "837 : ['Lyme_disease' 'West_Nile_fever' 'Plague']\n",
      "838 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "839 : ['Rift_Valley_fever' 'Yellow_Fever' 'Lyme_disease']\n",
      "840 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "841 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "842 : ['Lyme_disease' 'Japanese_encephalitis' 'Plague']\n",
      "843 : ['West_Nile_fever' 'Chikungunya' 'Japanese_encephalitis']\n",
      "844 : ['Rift_Valley_fever' 'Tungiasis' 'Dengue']\n",
      "845 : ['Rift_Valley_fever' 'Japanese_encephalitis' 'Tungiasis']\n",
      "846 : ['Chikungunya' 'Rift_Valley_fever' 'Dengue']\n",
      "847 : ['Malaria' 'Japanese_encephalitis' 'Lyme_disease']\n",
      "848 : ['Japanese_encephalitis' 'West_Nile_fever' 'Lyme_disease']\n",
      "849 : ['Lyme_disease' 'Japanese_encephalitis' 'Yellow_Fever']\n",
      "850 : ['West_Nile_fever' 'Japanese_encephalitis' 'Dengue']\n",
      "851 : ['Rift_Valley_fever' 'Chikungunya' 'West_Nile_fever']\n",
      "852 : ['Chikungunya' 'Rift_Valley_fever' 'Dengue']\n",
      "853 : ['West_Nile_fever' 'Chikungunya' 'Tungiasis']\n",
      "854 : ['Lyme_disease' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "855 : ['Lyme_disease' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "856 : ['Japanese_encephalitis' 'Lyme_disease' 'Yellow_Fever']\n",
      "857 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "858 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "859 : ['Japanese_encephalitis' 'Malaria' 'West_Nile_fever']\n",
      "860 : ['West_Nile_fever' 'Japanese_encephalitis' 'Dengue']\n",
      "861 : ['Dengue' 'West_Nile_fever' 'Chikungunya']\n",
      "862 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "863 : ['Lyme_disease' 'West_Nile_fever' 'Dengue']\n",
      "864 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "865 : ['Tungiasis' 'Lyme_disease' 'Japanese_encephalitis']\n",
      "866 : ['Chikungunya' 'Tungiasis' 'Dengue']\n",
      "867 : ['Lyme_disease' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "868 : ['Chikungunya' 'Dengue' 'Rift_Valley_fever']\n",
      "869 : ['Yellow_Fever' 'Plague' 'Tungiasis']\n",
      "870 : ['Chikungunya' 'Rift_Valley_fever' 'Dengue']\n",
      "871 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "872 : ['Malaria' 'Plague' 'Japanese_encephalitis']\n",
      "873 : ['Malaria' 'Japanese_encephalitis' 'Dengue']\n",
      "874 : ['Chikungunya' 'Dengue' 'Rift_Valley_fever']\n",
      "875 : ['Rift_Valley_fever' 'Japanese_encephalitis' 'Dengue']\n",
      "876 : ['Zika' 'Chikungunya' 'Malaria']\n",
      "877 : ['Tungiasis' 'Lyme_disease' 'Rift_Valley_fever']\n",
      "878 : ['Plague' 'Yellow_Fever' 'Malaria']\n",
      "879 : ['Malaria' 'Japanese_encephalitis' 'Lyme_disease']\n",
      "880 : ['Plague' 'Malaria' 'Lyme_disease']\n",
      "881 : ['Tungiasis' 'Rift_Valley_fever' 'Dengue']\n",
      "882 : ['Japanese_encephalitis' 'Malaria' 'Lyme_disease']\n",
      "883 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "884 : ['Lyme_disease' 'Japanese_encephalitis' 'Plague']\n",
      "885 : ['West_Nile_fever' 'Chikungunya' 'Rift_Valley_fever']\n",
      "886 : ['Tungiasis' 'Dengue' 'Chikungunya']\n",
      "887 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "888 : ['West_Nile_fever' 'Dengue' 'Chikungunya']\n",
      "889 : ['Chikungunya' 'Dengue' 'Rift_Valley_fever']\n",
      "890 : ['Chikungunya' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "891 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "892 : ['West_Nile_fever' 'Japanese_encephalitis' 'Lyme_disease']\n",
      "893 : ['Yellow_Fever' 'Plague' 'Lyme_disease']\n",
      "894 : ['Chikungunya' 'Tungiasis' 'Dengue']\n",
      "895 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "896 : ['Zika' 'Chikungunya' 'West_Nile_fever']\n",
      "897 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "898 : ['Lyme_disease' 'Japanese_encephalitis' 'Yellow_Fever']\n",
      "899 : ['Yellow_Fever' 'Zika' 'West_Nile_fever']\n",
      "900 : ['Plague' 'Yellow_Fever' 'Malaria']\n",
      "901 : ['Malaria' 'Japanese_encephalitis' 'Plague']\n",
      "902 : ['Zika' 'Tungiasis' 'Yellow_Fever']\n",
      "903 : ['Zika' 'Malaria' 'Yellow_Fever']\n",
      "904 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "905 : ['Lyme_disease' 'Tungiasis' 'Japanese_encephalitis']\n",
      "906 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "907 : ['Rift_Valley_fever' 'West_Nile_fever' 'Dengue']\n",
      "908 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "909 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "910 : ['Plague' 'Yellow_Fever' 'Malaria']\n",
      "911 : ['West_Nile_fever' 'Zika' 'Plague']\n",
      "912 : ['West_Nile_fever' 'Rift_Valley_fever' 'Dengue']\n",
      "913 : ['West_Nile_fever' 'Dengue' 'Yellow_Fever']\n",
      "914 : ['Japanese_encephalitis' 'Rift_Valley_fever' 'Plague']\n",
      "915 : ['Lyme_disease' 'Malaria' 'Japanese_encephalitis']\n",
      "916 : ['West_Nile_fever' 'Tungiasis' 'Japanese_encephalitis']\n",
      "917 : ['Malaria' 'Japanese_encephalitis' 'Plague']\n",
      "918 : ['Tungiasis' 'West_Nile_fever' 'Dengue']\n",
      "919 : ['West_Nile_fever' 'Chikungunya' 'Japanese_encephalitis']\n",
      "920 : ['Lyme_disease' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "921 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "922 : ['Chikungunya' 'Rift_Valley_fever' 'Plague']\n",
      "923 : ['Chikungunya' 'Dengue' 'Rift_Valley_fever']\n",
      "924 : ['Lyme_disease' 'Yellow_Fever' 'Plague']\n",
      "925 : ['West_Nile_fever' 'Lyme_disease' 'Dengue']\n",
      "926 : ['Chikungunya' 'West_Nile_fever' 'Rift_Valley_fever']\n",
      "927 : ['West_Nile_fever' 'Rift_Valley_fever' 'Dengue']\n",
      "928 : ['Tungiasis' 'Japanese_encephalitis' 'Lyme_disease']\n",
      "929 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "930 : ['Rift_Valley_fever' 'West_Nile_fever' 'Zika']\n",
      "931 : ['Tungiasis' 'Rift_Valley_fever' 'West_Nile_fever']\n",
      "932 : ['West_Nile_fever' 'Lyme_disease' 'Zika']\n",
      "933 : ['Plague' 'West_Nile_fever' 'Malaria']\n",
      "934 : ['Lyme_disease' 'Plague' 'West_Nile_fever']\n",
      "935 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "936 : ['Lyme_disease' 'Plague' 'West_Nile_fever']\n",
      "937 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "938 : ['Tungiasis' 'Rift_Valley_fever' 'Dengue']\n",
      "939 : ['Plague' 'Lyme_disease' 'Japanese_encephalitis']\n",
      "940 : ['Tungiasis' 'Rift_Valley_fever' 'Chikungunya']\n",
      "941 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "942 : ['Chikungunya' 'Rift_Valley_fever' 'Dengue']\n",
      "943 : ['West_Nile_fever' 'Zika' 'Tungiasis']\n",
      "944 : ['West_Nile_fever' 'Japanese_encephalitis' 'Chikungunya']\n",
      "945 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "946 : ['Chikungunya' 'West_Nile_fever' 'Dengue']\n",
      "947 : ['Japanese_encephalitis' 'Lyme_disease' 'Tungiasis']\n",
      "948 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "949 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "950 : ['Zika' 'Tungiasis' 'Yellow_Fever']\n",
      "951 : ['Lyme_disease' 'Japanese_encephalitis' 'Plague']\n",
      "952 : ['Rift_Valley_fever' 'West_Nile_fever' 'Lyme_disease']\n",
      "953 : ['Lyme_disease' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "954 : ['West_Nile_fever' 'Dengue' 'Chikungunya']\n",
      "955 : ['Japanese_encephalitis' 'Tungiasis' 'Lyme_disease']\n",
      "956 : ['Lyme_disease' 'Japanese_encephalitis' 'Rift_Valley_fever']\n",
      "957 : ['West_Nile_fever' 'Japanese_encephalitis' 'Malaria']\n",
      "958 : ['Chikungunya' 'Rift_Valley_fever' 'West_Nile_fever']\n",
      "959 : ['Rift_Valley_fever' 'Tungiasis' 'West_Nile_fever']\n",
      "960 : ['Japanese_encephalitis' 'West_Nile_fever' 'Chikungunya']\n",
      "961 : ['Tungiasis' 'Rift_Valley_fever' 'Chikungunya']\n",
      "962 : ['Malaria' 'Tungiasis' 'Japanese_encephalitis']\n",
      "963 : ['Japanese_encephalitis' 'West_Nile_fever' 'Malaria']\n",
      "964 : ['Malaria' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "965 : ['Malaria' 'Japanese_encephalitis' 'Plague']\n",
      "966 : ['Rift_Valley_fever' 'West_Nile_fever' 'Dengue']\n",
      "967 : ['Lyme_disease' 'Malaria' 'Japanese_encephalitis']\n",
      "968 : ['Lyme_disease' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "969 : ['Tungiasis' 'Rift_Valley_fever' 'Lyme_disease']\n",
      "970 : ['Rift_Valley_fever' 'Tungiasis' 'Japanese_encephalitis']\n",
      "971 : ['Tungiasis' 'Rift_Valley_fever' 'Lyme_disease']\n",
      "972 : ['Malaria' 'Japanese_encephalitis' 'Plague']\n",
      "973 : ['West_Nile_fever' 'Dengue' 'Chikungunya']\n",
      "974 : ['Lyme_disease' 'Plague' 'Japanese_encephalitis']\n",
      "975 : ['Lyme_disease' 'Plague' 'Malaria']\n",
      "976 : ['West_Nile_fever' 'Lyme_disease' 'Zika']\n",
      "977 : ['Dengue' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "978 : ['Zika' 'Yellow_Fever' 'Plague']\n",
      "979 : ['Malaria' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "980 : ['Malaria' 'Plague' 'West_Nile_fever']\n",
      "981 : ['Lyme_disease' 'Japanese_encephalitis' 'West_Nile_fever']\n",
      "982 : ['Japanese_encephalitis' 'West_Nile_fever' 'Chikungunya']\n",
      "983 : ['Rift_Valley_fever' 'Tungiasis' 'Dengue']\n",
      "984 : ['Lyme_disease' 'Yellow_Fever' 'Plague']\n",
      "985 : ['West_Nile_fever' 'Malaria' 'Japanese_encephalitis']\n",
      "986 : ['Rift_Valley_fever' 'Tungiasis' 'Chikungunya']\n",
      "987 : ['West_Nile_fever' 'Chikungunya' 'Dengue']\n",
      "988 : ['Chikungunya' 'Dengue' 'West_Nile_fever']\n",
      "989 : ['Rift_Valley_fever' 'Tungiasis' 'Lyme_disease']\n",
      "990 : ['West_Nile_fever' 'Dengue' 'Lyme_disease']\n",
      "991 : ['Malaria' 'Lyme_disease' 'Japanese_encephalitis']\n",
      "992 : ['West_Nile_fever' 'Chikungunya' 'Tungiasis']\n",
      "993 : ['Chikungunya' 'Plague' 'Malaria']\n",
      "994 : ['Malaria' 'Plague' 'Japanese_encephalitis']\n",
      "995 : ['West_Nile_fever' 'Dengue' 'Japanese_encephalitis']\n",
      "996 : ['Dengue' 'Chikungunya' 'Rift_Valley_fever']\n",
      "997 : ['Japanese_encephalitis' 'West_Nile_fever' 'Rift_Valley_fever']\n",
      "998 : ['Lyme_disease' 'Malaria' 'Japanese_encephalitis']\n",
      "999 : ['West_Nile_fever' 'Rift_Valley_fever' 'Lyme_disease']\n",
      "1000 : ['Tungiasis' 'Rift_Valley_fever' 'Japanese_encephalitis']\n",
      "1001 : ['Tungiasis' 'Plague' 'Japanese_encephalitis']\n",
      "1002 : ['Yellow_Fever' 'Tungiasis' 'Japanese_encephalitis']\n",
      "1003 : ['Japanese_encephalitis' 'Malaria' 'West_Nile_fever']\n",
      "1004 : ['Tungiasis' 'Japanese_encephalitis' 'Lyme_disease']\n",
      "1005 : ['Zika' 'Rift_Valley_fever' 'West_Nile_fever']\n",
      "1006 : ['Lyme_disease' 'Plague' 'Malaria']\n",
      "1007 : ['Lyme_disease' 'West_Nile_fever' 'Japanese_encephalitis']\n",
      "1008 : ['Lyme_disease' 'West_Nile_fever' 'Rift_Valley_fever']\n",
      "1009 : ['Zika' 'Plague' 'Malaria']\n"
     ]
    }
   ],
   "source": [
    "test = test_data.values\n",
    "\n",
    "# Convert the scaled features to PyTorch tensor\n",
    "test_tensor = torch.tensor(test, dtype=torch.float32)\n",
    "\n",
    "# Set the model to evaluation mode\n",
    "best_model.eval()\n",
    "\n",
    "# Generate predictions for the test data\n",
    "probabilities = best_model(test_tensor)\n",
    "top_k = 3\n",
    "top_k_predictions = torch.topk(probabilities, k=top_k, dim=1).indices.numpy()\n",
    "\n",
    "# Decode the predicted labels using the label encoder\n",
    "predicted_labels = []\n",
    "for sample_predictions in top_k_predictions:\n",
    "    sample_classes = label_encoder.inverse_transform(sample_predictions)\n",
    "    predicted_labels.append(sample_classes)\n",
    "\n",
    "# Show the top 3 predictions \n",
    "print(\"Top 3 Predictions:\")\n",
    "for i, sample_predictions in enumerate(predicted_labels[:]):\n",
    "    index = test_data.index[i]  # Get the index value from the test data\n",
    "    print(index, \":\", sample_predictions)\n",
    "    #test_data[i, 'predictions'] = sample_predictions"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 / pyspotseven",
   "language": "python",
   "name": "py310-pyspotseven"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}