notebooks/1-data-cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML Clustering - Network Intrusion Detection\n",
    "## Part 1 - DATA CLEANING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we load the raw dataset and implement initial cleaning to prepare it for the next processing phase.\n",
    "\n",
    "> **INPUT:** the raw dataset loaded from sklearn library. <br>\n",
    "> **OUTPUT:** a cleaned version of the dataset stored in an intermediate csv file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. INITIALIZATION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary libraries and modules\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set display options\n",
    "pd.set_option('display.max_columns', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. LOADING DATASET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the original KDD Cup '99 dataset was initially created to produce a large training set for supervised learning algorithms, there is a large proportion (80.1%) of\n",
    "abnormal data which is unrealistic in the real world, and inappropriate for unsupervised anomaly detection. \n",
    "\n",
    "For this reason, we use the transformed SA version by sklearn which is obtained by selecting all the normal data, and a small proportion of abnormal data to give an anomaly proportion of 1%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading dataset from sklearn library\n",
    "from sklearn.datasets import fetch_kddcup99\n",
    "kddcup99 = fetch_kddcup99(subset=\"SA\", as_frame=True, percent10=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _kddcup99_dataset:\n",
      "\n",
      "Kddcup 99 dataset\n",
      "-----------------\n",
      "\n",
      "The KDD Cup '99 dataset was created by processing the tcpdump portions\n",
      "of the 1998 DARPA Intrusion Detection System (IDS) Evaluation dataset,\n",
      "created by MIT Lincoln Lab [2]_. The artificial data (described on the `dataset's\n",
      "homepage <https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html>`_) was\n",
      "generated using a closed network and hand-injected attacks to produce a\n",
      "large number of different types of attack with normal activity in the\n",
      "background. As the initial goal was to produce a large training set for\n",
      "supervised learning algorithms, there is a large proportion (80.1%) of\n",
      "abnormal data which is unrealistic in real world, and inappropriate for\n",
      "unsupervised anomaly detection which aims at detecting 'abnormal' data, i.e.:\n",
      "\n",
      "* qualitatively different from normal data\n",
      "* in large minority among the observations.\n",
      "\n",
      "We thus transform the KDD Data set into two different data sets: SA and SF.\n",
      "\n",
      "* SA is obtained by simply selecting all the normal data, and a small\n",
      "  proportion of abnormal data to gives an anomaly proportion of 1%.\n",
      "\n",
      "* SF is obtained as in [3]_\n",
      "  by simply picking up the data whose attribute logged_in is positive, thus\n",
      "  focusing on the intrusion attack, which gives a proportion of 0.3% of\n",
      "  attack.\n",
      "\n",
      "* http and smtp are two subsets of SF corresponding with third feature\n",
      "  equal to 'http' (resp. to 'smtp').\n",
      "\n",
      "General KDD structure :\n",
      "\n",
      "    ================      ==========================================\n",
      "    Samples total         4898431\n",
      "    Dimensionality        41\n",
      "    Features              discrete (int) or continuous (float)\n",
      "    Targets               str, 'normal.' or name of the anomaly type\n",
      "    ================      ==========================================\n",
      "\n",
      "    SA structure :\n",
      "\n",
      "    ================      ==========================================\n",
      "    Samples total         976158\n",
      "    Dimensionality        41\n",
      "    Features              discrete (int) or continuous (float)\n",
      "    Targets               str, 'normal.' or name of the anomaly type\n",
      "    ================      ==========================================\n",
      "\n",
      "    SF structure :\n",
      "\n",
      "    ================      ==========================================\n",
      "    Samples total         699691\n",
      "    Dimensionality        4\n",
      "    Features              discrete (int) or continuous (float)\n",
      "    Targets               str, 'normal.' or name of the anomaly type\n",
      "    ================      ==========================================\n",
      "\n",
      "    http structure :\n",
      "\n",
      "    ================      ==========================================\n",
      "    Samples total         619052\n",
      "    Dimensionality        3\n",
      "    Features              discrete (int) or continuous (float)\n",
      "    Targets               str, 'normal.' or name of the anomaly type\n",
      "    ================      ==========================================\n",
      "\n",
      "    smtp structure :\n",
      "\n",
      "    ================      ==========================================\n",
      "    Samples total         95373\n",
      "    Dimensionality        3\n",
      "    Features              discrete (int) or continuous (float)\n",
      "    Targets               str, 'normal.' or name of the anomaly type\n",
      "    ================      ==========================================\n",
      "\n",
      ":func:`sklearn.datasets.fetch_kddcup99` will load the kddcup99 dataset; it\n",
      "returns a dictionary-like object with the feature matrix in the ``data`` member\n",
      "and the target values in ``target``. The \"as_frame\" optional argument converts\n",
      "``data`` into a pandas DataFrame and ``target`` into a pandas Series. The\n",
      "dataset will be downloaded from the web if necessary.\n",
      "\n",
      ".. topic:: References\n",
      "\n",
      "    .. [2] Analysis and Results of the 1999 DARPA Off-Line Intrusion\n",
      "           Detection Evaluation, Richard Lippmann, Joshua W. Haines,\n",
      "           David J. Fried, Jonathan Korba, Kumar Das.\n",
      "\n",
      "    .. [3] K. Yamanishi, J.-I. Takeuchi, G. Williams, and P. Milne. Online\n",
      "           unsupervised outlier detection using finite mixtures with\n",
      "           discounting learning algorithms. In Proceedings of the sixth\n",
      "           ACM SIGKDD international conference on Knowledge discovery\n",
      "           and data mining, pages 320-324. ACM Press, 2000.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check dataset description\n",
    "print(kddcup99.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>logged_in</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>num_root</th>\n",
       "      <th>num_file_creations</th>\n",
       "      <th>num_shells</th>\n",
       "      <th>num_access_files</th>\n",
       "      <th>num_outbound_cmds</th>\n",
       "      <th>is_host_login</th>\n",
       "      <th>is_guest_login</th>\n",
       "      <th>count</th>\n",
       "      <th>srv_count</th>\n",
       "      <th>serror_rate</th>\n",
       "      <th>srv_serror_rate</th>\n",
       "      <th>rerror_rate</th>\n",
       "      <th>srv_rerror_rate</th>\n",
       "      <th>same_srv_rate</th>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'http'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>215</td>\n",
       "      <td>45076</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'normal.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'http'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>162</td>\n",
       "      <td>4528</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'normal.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'http'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>236</td>\n",
       "      <td>1228</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'normal.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'http'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>233</td>\n",
       "      <td>2032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.33</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'normal.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'http'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>239</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'normal.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976153</th>\n",
       "      <td>0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'private'</td>\n",
       "      <td>b'S0'</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>254</td>\n",
       "      <td>16</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>16</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'neptune.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976154</th>\n",
       "      <td>0</td>\n",
       "      <td>b'icmp'</td>\n",
       "      <td>b'ecr_i'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>1032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511</td>\n",
       "      <td>511</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'smurf.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976155</th>\n",
       "      <td>0</td>\n",
       "      <td>b'icmp'</td>\n",
       "      <td>b'ecr_i'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>1032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511</td>\n",
       "      <td>511</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'smurf.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976156</th>\n",
       "      <td>0</td>\n",
       "      <td>b'icmp'</td>\n",
       "      <td>b'ecr_i'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>1032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511</td>\n",
       "      <td>511</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'smurf.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976157</th>\n",
       "      <td>0</td>\n",
       "      <td>b'icmp'</td>\n",
       "      <td>b'ecr_i'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>1032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511</td>\n",
       "      <td>511</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'smurf.'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>976158 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       duration protocol_type     service   flag src_bytes dst_bytes land  \\\n",
       "0             0        b'tcp'     b'http'  b'SF'       215     45076    0   \n",
       "1             0        b'tcp'     b'http'  b'SF'       162      4528    0   \n",
       "2             0        b'tcp'     b'http'  b'SF'       236      1228    0   \n",
       "3             0        b'tcp'     b'http'  b'SF'       233      2032    0   \n",
       "4             0        b'tcp'     b'http'  b'SF'       239       486    0   \n",
       "...         ...           ...         ...    ...       ...       ...  ...   \n",
       "976153        0        b'tcp'  b'private'  b'S0'         0         0    0   \n",
       "976154        0       b'icmp'    b'ecr_i'  b'SF'      1032         0    0   \n",
       "976155        0       b'icmp'    b'ecr_i'  b'SF'      1032         0    0   \n",
       "976156        0       b'icmp'    b'ecr_i'  b'SF'      1032         0    0   \n",
       "976157        0       b'icmp'    b'ecr_i'  b'SF'      1032         0    0   \n",
       "\n",
       "       wrong_fragment urgent hot num_failed_logins logged_in num_compromised  \\\n",
       "0                   0      0   0                 0         1               0   \n",
       "1                   0      0   0                 0         1               0   \n",
       "2                   0      0   0                 0         1               0   \n",
       "3                   0      0   0                 0         1               0   \n",
       "4                   0      0   0                 0         1               0   \n",
       "...               ...    ...  ..               ...       ...             ...   \n",
       "976153              0      0   0                 0         0               0   \n",
       "976154              0      0   0                 0         0               0   \n",
       "976155              0      0   0                 0         0               0   \n",
       "976156              0      0   0                 0         0               0   \n",
       "976157              0      0   0                 0         0               0   \n",
       "\n",
       "       root_shell su_attempted num_root num_file_creations num_shells  \\\n",
       "0               0            0        0                  0          0   \n",
       "1               0            0        0                  0          0   \n",
       "2               0            0        0                  0          0   \n",
       "3               0            0        0                  0          0   \n",
       "4               0            0        0                  0          0   \n",
       "...           ...          ...      ...                ...        ...   \n",
       "976153          0            0        0                  0          0   \n",
       "976154          0            0        0                  0          0   \n",
       "976155          0            0        0                  0          0   \n",
       "976156          0            0        0                  0          0   \n",
       "976157          0            0        0                  0          0   \n",
       "\n",
       "       num_access_files num_outbound_cmds is_host_login is_guest_login count  \\\n",
       "0                     0                 0             0              0     1   \n",
       "1                     0                 0             0              0     2   \n",
       "2                     0                 0             0              0     1   \n",
       "3                     0                 0             0              0     2   \n",
       "4                     0                 0             0              0     3   \n",
       "...                 ...               ...           ...            ...   ...   \n",
       "976153                0                 0             0              0   254   \n",
       "976154                0                 0             0              0   511   \n",
       "976155                0                 0             0              0   511   \n",
       "976156                0                 0             0              0   511   \n",
       "976157                0                 0             0              0   511   \n",
       "\n",
       "       srv_count serror_rate srv_serror_rate rerror_rate srv_rerror_rate  \\\n",
       "0              1         0.0             0.0         0.0             0.0   \n",
       "1              2         0.0             0.0         0.0             0.0   \n",
       "2              1         0.0             0.0         0.0             0.0   \n",
       "3              2         0.0             0.0         0.0             0.0   \n",
       "4              3         0.0             0.0         0.0             0.0   \n",
       "...          ...         ...             ...         ...             ...   \n",
       "976153        16         1.0             1.0         0.0             0.0   \n",
       "976154       511         0.0             0.0         0.0             0.0   \n",
       "976155       511         0.0             0.0         0.0             0.0   \n",
       "976156       511         0.0             0.0         0.0             0.0   \n",
       "976157       511         0.0             0.0         0.0             0.0   \n",
       "\n",
       "       same_srv_rate diff_srv_rate srv_diff_host_rate dst_host_count  \\\n",
       "0                1.0           0.0                0.0              0   \n",
       "1                1.0           0.0                0.0              1   \n",
       "2                1.0           0.0                0.0              2   \n",
       "3                1.0           0.0                0.0              3   \n",
       "4                1.0           0.0                0.0              4   \n",
       "...              ...           ...                ...            ...   \n",
       "976153          0.06          0.06                0.0            255   \n",
       "976154           1.0           0.0                0.0            255   \n",
       "976155           1.0           0.0                0.0            255   \n",
       "976156           1.0           0.0                0.0            255   \n",
       "976157           1.0           0.0                0.0            255   \n",
       "\n",
       "       dst_host_srv_count dst_host_same_srv_rate dst_host_diff_srv_rate  \\\n",
       "0                       0                    0.0                    0.0   \n",
       "1                       1                    1.0                    0.0   \n",
       "2                       2                    1.0                    0.0   \n",
       "3                       3                    1.0                    0.0   \n",
       "4                       4                    1.0                    0.0   \n",
       "...                   ...                    ...                    ...   \n",
       "976153                 16                   0.06                   0.07   \n",
       "976154                255                    1.0                    0.0   \n",
       "976155                255                    1.0                    0.0   \n",
       "976156                255                    1.0                    0.0   \n",
       "976157                255                    1.0                    0.0   \n",
       "\n",
       "       dst_host_same_src_port_rate dst_host_srv_diff_host_rate  \\\n",
       "0                              0.0                         0.0   \n",
       "1                              1.0                         0.0   \n",
       "2                              0.5                         0.0   \n",
       "3                             0.33                         0.0   \n",
       "4                             0.25                         0.0   \n",
       "...                            ...                         ...   \n",
       "976153                         0.0                         0.0   \n",
       "976154                         1.0                         0.0   \n",
       "976155                         1.0                         0.0   \n",
       "976156                         1.0                         0.0   \n",
       "976157                         1.0                         0.0   \n",
       "\n",
       "       dst_host_serror_rate dst_host_srv_serror_rate dst_host_rerror_rate  \\\n",
       "0                       0.0                      0.0                  0.0   \n",
       "1                       0.0                      0.0                  0.0   \n",
       "2                       0.0                      0.0                  0.0   \n",
       "3                       0.0                      0.0                  0.0   \n",
       "4                       0.0                      0.0                  0.0   \n",
       "...                     ...                      ...                  ...   \n",
       "976153                  1.0                      1.0                  0.0   \n",
       "976154                  0.0                      0.0                  0.0   \n",
       "976155                  0.0                      0.0                  0.0   \n",
       "976156                  0.0                      0.0                  0.0   \n",
       "976157                  0.0                      0.0                  0.0   \n",
       "\n",
       "       dst_host_srv_rerror_rate       labels  \n",
       "0                           0.0   b'normal.'  \n",
       "1                           0.0   b'normal.'  \n",
       "2                           0.0   b'normal.'  \n",
       "3                           0.0   b'normal.'  \n",
       "4                           0.0   b'normal.'  \n",
       "...                         ...          ...  \n",
       "976153                      0.0  b'neptune.'  \n",
       "976154                      0.0    b'smurf.'  \n",
       "976155                      0.0    b'smurf.'  \n",
       "976156                      0.0    b'smurf.'  \n",
       "976157                      0.0    b'smurf.'  \n",
       "\n",
       "[976158 rows x 42 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the loaded dataset as DataFrame\n",
    "data = kddcup99[\"frame\"]\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['duration', 'protocol_type', 'service', 'flag', 'src_bytes',\n",
       "       'dst_bytes', 'land', 'wrong_fragment', 'urgent', 'hot',\n",
       "       'num_failed_logins', 'logged_in', 'num_compromised', 'root_shell',\n",
       "       'su_attempted', 'num_root', 'num_file_creations', 'num_shells',\n",
       "       'num_access_files', 'num_outbound_cmds', 'is_host_login',\n",
       "       'is_guest_login', 'count', 'srv_count', 'serror_rate',\n",
       "       'srv_serror_rate', 'rerror_rate', 'srv_rerror_rate', 'same_srv_rate',\n",
       "       'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count',\n",
       "       'dst_host_srv_count', 'dst_host_same_srv_rate',\n",
       "       'dst_host_diff_srv_rate', 'dst_host_same_src_port_rate',\n",
       "       'dst_host_srv_diff_host_rate', 'dst_host_serror_rate',\n",
       "       'dst_host_srv_serror_rate', 'dst_host_rerror_rate',\n",
       "       'dst_host_srv_rerror_rate', 'labels'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check feature names\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe from this quick overview the following initial points:\n",
    "- The data set contains nearly a million records and 42 columns.\n",
    "- Most columns are numerical with few categorical features.\n",
    "- Categorical columns are represented as byte strings that need to be decoded. \n",
    "- There is a label column denoting the class attribute since the original data set is labeled (this column will be disregarded)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. DATA CLEANING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decoding byte string objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Decoding byte string columns into string objects\n",
    "for c in [\"protocol_type\", \"service\", \"flag\"]:\n",
    "    data[c] = data[c].str.decode(\"utf-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Eliminating not useful columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since we are dealing with an unsupervised problem, the class attribute will be ignored\n",
    "data.drop(\"labels\", axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "duration                       has 9036 unique values.\n",
      "protocol_type                  has 3 unique values.\n",
      "service                        has 53 unique values.\n",
      "flag                           has 10 unique values.\n",
      "src_bytes                      has 7145 unique values.\n",
      "dst_bytes                      has 21463 unique values.\n",
      "land                           has 2 unique values.\n",
      "wrong_fragment                 has 1 unique values.\n",
      "urgent                         has 6 unique values.\n",
      "hot                            has 30 unique values.\n",
      "num_failed_logins              has 5 unique values.\n",
      "logged_in                      has 2 unique values.\n",
      "num_compromised                has 96 unique values.\n",
      "root_shell                     has 2 unique values.\n",
      "su_attempted                   has 3 unique values.\n",
      "num_root                       has 90 unique values.\n",
      "num_file_creations             has 42 unique values.\n",
      "num_shells                     has 3 unique values.\n",
      "num_access_files               has 10 unique values.\n",
      "num_outbound_cmds              has 1 unique values.\n",
      "is_host_login                  has 2 unique values.\n",
      "is_guest_login                 has 2 unique values.\n",
      "count                          has 511 unique values.\n",
      "srv_count                      has 511 unique values.\n",
      "serror_rate                    has 40 unique values.\n",
      "srv_serror_rate                has 61 unique values.\n",
      "rerror_rate                    has 73 unique values.\n",
      "srv_rerror_rate                has 76 unique values.\n",
      "same_srv_rate                  has 87 unique values.\n",
      "diff_srv_rate                  has 87 unique values.\n",
      "srv_diff_host_rate             has 72 unique values.\n",
      "dst_host_count                 has 256 unique values.\n",
      "dst_host_srv_count             has 256 unique values.\n",
      "dst_host_same_srv_rate         has 101 unique values.\n",
      "dst_host_diff_srv_rate         has 101 unique values.\n",
      "dst_host_same_src_port_rate    has 101 unique values.\n",
      "dst_host_srv_diff_host_rate    has 74 unique values.\n",
      "dst_host_serror_rate           has 101 unique values.\n",
      "dst_host_srv_serror_rate       has 100 unique values.\n",
      "dst_host_rerror_rate           has 101 unique values.\n",
      "dst_host_srv_rerror_rate       has 101 unique values.\n"
     ]
    }
   ],
   "source": [
    "# Check the number of unique values in each attribute\n",
    "for c in data.columns:\n",
    "    print(f\"{c:30} has {data[c].nunique()} unique values.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop \"num_outbound_cmds\" since it has only one unique value\n",
    "data.drop(columns=[\"num_outbound_cmds\"], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Checking null values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "duration                       0\n",
       "srv_count                      0\n",
       "serror_rate                    0\n",
       "srv_serror_rate                0\n",
       "rerror_rate                    0\n",
       "srv_rerror_rate                0\n",
       "same_srv_rate                  0\n",
       "diff_srv_rate                  0\n",
       "count                          0\n",
       "srv_diff_host_rate             0\n",
       "dst_host_srv_count             0\n",
       "dst_host_same_srv_rate         0\n",
       "dst_host_diff_srv_rate         0\n",
       "dst_host_same_src_port_rate    0\n",
       "dst_host_srv_diff_host_rate    0\n",
       "dst_host_serror_rate           0\n",
       "dst_host_srv_serror_rate       0\n",
       "dst_host_count                 0\n",
       "is_guest_login                 0\n",
       "is_host_login                  0\n",
       "num_access_files               0\n",
       "protocol_type                  0\n",
       "service                        0\n",
       "flag                           0\n",
       "src_bytes                      0\n",
       "dst_bytes                      0\n",
       "land                           0\n",
       "wrong_fragment                 0\n",
       "urgent                         0\n",
       "hot                            0\n",
       "num_failed_logins              0\n",
       "logged_in                      0\n",
       "num_compromised                0\n",
       "root_shell                     0\n",
       "su_attempted                   0\n",
       "num_root                       0\n",
       "num_file_creations             0\n",
       "num_shells                     0\n",
       "dst_host_rerror_rate           0\n",
       "dst_host_srv_rerror_rate       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the number of null values in each column sorted in a descending order\n",
    "data.isna().sum().sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data set has no null values and there is no need to apply any null value imputations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>logged_in</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>num_root</th>\n",
       "      <th>num_file_creations</th>\n",
       "      <th>num_shells</th>\n",
       "      <th>num_access_files</th>\n",
       "      <th>is_host_login</th>\n",
       "      <th>is_guest_login</th>\n",
       "      <th>count</th>\n",
       "      <th>srv_count</th>\n",
       "      <th>serror_rate</th>\n",
       "      <th>srv_serror_rate</th>\n",
       "      <th>rerror_rate</th>\n",
       "      <th>srv_rerror_rate</th>\n",
       "      <th>same_srv_rate</th>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>215</td>\n",
       "      <td>45076</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>162</td>\n",
       "      <td>4528</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>236</td>\n",
       "      <td>1228</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>233</td>\n",
       "      <td>2032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.33</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>239</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976153</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>private</td>\n",
       "      <td>S0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>254</td>\n",
       "      <td>16</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>16</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976154</th>\n",
       "      <td>0</td>\n",
       "      <td>icmp</td>\n",
       "      <td>ecr_i</td>\n",
       "      <td>SF</td>\n",
       "      <td>1032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511</td>\n",
       "      <td>511</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976155</th>\n",
       "      <td>0</td>\n",
       "      <td>icmp</td>\n",
       "      <td>ecr_i</td>\n",
       "      <td>SF</td>\n",
       "      <td>1032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511</td>\n",
       "      <td>511</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976156</th>\n",
       "      <td>0</td>\n",
       "      <td>icmp</td>\n",
       "      <td>ecr_i</td>\n",
       "      <td>SF</td>\n",
       "      <td>1032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511</td>\n",
       "      <td>511</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976157</th>\n",
       "      <td>0</td>\n",
       "      <td>icmp</td>\n",
       "      <td>ecr_i</td>\n",
       "      <td>SF</td>\n",
       "      <td>1032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511</td>\n",
       "      <td>511</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>976158 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       duration protocol_type  service flag src_bytes dst_bytes land  \\\n",
       "0             0           tcp     http   SF       215     45076    0   \n",
       "1             0           tcp     http   SF       162      4528    0   \n",
       "2             0           tcp     http   SF       236      1228    0   \n",
       "3             0           tcp     http   SF       233      2032    0   \n",
       "4             0           tcp     http   SF       239       486    0   \n",
       "...         ...           ...      ...  ...       ...       ...  ...   \n",
       "976153        0           tcp  private   S0         0         0    0   \n",
       "976154        0          icmp    ecr_i   SF      1032         0    0   \n",
       "976155        0          icmp    ecr_i   SF      1032         0    0   \n",
       "976156        0          icmp    ecr_i   SF      1032         0    0   \n",
       "976157        0          icmp    ecr_i   SF      1032         0    0   \n",
       "\n",
       "       wrong_fragment urgent hot num_failed_logins logged_in num_compromised  \\\n",
       "0                   0      0   0                 0         1               0   \n",
       "1                   0      0   0                 0         1               0   \n",
       "2                   0      0   0                 0         1               0   \n",
       "3                   0      0   0                 0         1               0   \n",
       "4                   0      0   0                 0         1               0   \n",
       "...               ...    ...  ..               ...       ...             ...   \n",
       "976153              0      0   0                 0         0               0   \n",
       "976154              0      0   0                 0         0               0   \n",
       "976155              0      0   0                 0         0               0   \n",
       "976156              0      0   0                 0         0               0   \n",
       "976157              0      0   0                 0         0               0   \n",
       "\n",
       "       root_shell su_attempted num_root num_file_creations num_shells  \\\n",
       "0               0            0        0                  0          0   \n",
       "1               0            0        0                  0          0   \n",
       "2               0            0        0                  0          0   \n",
       "3               0            0        0                  0          0   \n",
       "4               0            0        0                  0          0   \n",
       "...           ...          ...      ...                ...        ...   \n",
       "976153          0            0        0                  0          0   \n",
       "976154          0            0        0                  0          0   \n",
       "976155          0            0        0                  0          0   \n",
       "976156          0            0        0                  0          0   \n",
       "976157          0            0        0                  0          0   \n",
       "\n",
       "       num_access_files is_host_login is_guest_login count srv_count  \\\n",
       "0                     0             0              0     1         1   \n",
       "1                     0             0              0     2         2   \n",
       "2                     0             0              0     1         1   \n",
       "3                     0             0              0     2         2   \n",
       "4                     0             0              0     3         3   \n",
       "...                 ...           ...            ...   ...       ...   \n",
       "976153                0             0              0   254        16   \n",
       "976154                0             0              0   511       511   \n",
       "976155                0             0              0   511       511   \n",
       "976156                0             0              0   511       511   \n",
       "976157                0             0              0   511       511   \n",
       "\n",
       "       serror_rate srv_serror_rate rerror_rate srv_rerror_rate same_srv_rate  \\\n",
       "0              0.0             0.0         0.0             0.0           1.0   \n",
       "1              0.0             0.0         0.0             0.0           1.0   \n",
       "2              0.0             0.0         0.0             0.0           1.0   \n",
       "3              0.0             0.0         0.0             0.0           1.0   \n",
       "4              0.0             0.0         0.0             0.0           1.0   \n",
       "...            ...             ...         ...             ...           ...   \n",
       "976153         1.0             1.0         0.0             0.0          0.06   \n",
       "976154         0.0             0.0         0.0             0.0           1.0   \n",
       "976155         0.0             0.0         0.0             0.0           1.0   \n",
       "976156         0.0             0.0         0.0             0.0           1.0   \n",
       "976157         0.0             0.0         0.0             0.0           1.0   \n",
       "\n",
       "       diff_srv_rate srv_diff_host_rate dst_host_count dst_host_srv_count  \\\n",
       "0                0.0                0.0              0                  0   \n",
       "1                0.0                0.0              1                  1   \n",
       "2                0.0                0.0              2                  2   \n",
       "3                0.0                0.0              3                  3   \n",
       "4                0.0                0.0              4                  4   \n",
       "...              ...                ...            ...                ...   \n",
       "976153          0.06                0.0            255                 16   \n",
       "976154           0.0                0.0            255                255   \n",
       "976155           0.0                0.0            255                255   \n",
       "976156           0.0                0.0            255                255   \n",
       "976157           0.0                0.0            255                255   \n",
       "\n",
       "       dst_host_same_srv_rate dst_host_diff_srv_rate  \\\n",
       "0                         0.0                    0.0   \n",
       "1                         1.0                    0.0   \n",
       "2                         1.0                    0.0   \n",
       "3                         1.0                    0.0   \n",
       "4                         1.0                    0.0   \n",
       "...                       ...                    ...   \n",
       "976153                   0.06                   0.07   \n",
       "976154                    1.0                    0.0   \n",
       "976155                    1.0                    0.0   \n",
       "976156                    1.0                    0.0   \n",
       "976157                    1.0                    0.0   \n",
       "\n",
       "       dst_host_same_src_port_rate dst_host_srv_diff_host_rate  \\\n",
       "0                              0.0                         0.0   \n",
       "1                              1.0                         0.0   \n",
       "2                              0.5                         0.0   \n",
       "3                             0.33                         0.0   \n",
       "4                             0.25                         0.0   \n",
       "...                            ...                         ...   \n",
       "976153                         0.0                         0.0   \n",
       "976154                         1.0                         0.0   \n",
       "976155                         1.0                         0.0   \n",
       "976156                         1.0                         0.0   \n",
       "976157                         1.0                         0.0   \n",
       "\n",
       "       dst_host_serror_rate dst_host_srv_serror_rate dst_host_rerror_rate  \\\n",
       "0                       0.0                      0.0                  0.0   \n",
       "1                       0.0                      0.0                  0.0   \n",
       "2                       0.0                      0.0                  0.0   \n",
       "3                       0.0                      0.0                  0.0   \n",
       "4                       0.0                      0.0                  0.0   \n",
       "...                     ...                      ...                  ...   \n",
       "976153                  1.0                      1.0                  0.0   \n",
       "976154                  0.0                      0.0                  0.0   \n",
       "976155                  0.0                      0.0                  0.0   \n",
       "976156                  0.0                      0.0                  0.0   \n",
       "976157                  0.0                      0.0                  0.0   \n",
       "\n",
       "       dst_host_srv_rerror_rate  \n",
       "0                           0.0  \n",
       "1                           0.0  \n",
       "2                           0.0  \n",
       "3                           0.0  \n",
       "4                           0.0  \n",
       "...                         ...  \n",
       "976153                      0.0  \n",
       "976154                      0.0  \n",
       "976155                      0.0  \n",
       "976156                      0.0  \n",
       "976157                      0.0  \n",
       "\n",
       "[976158 rows x 40 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check cleaned data and its dimensions\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data set has been cleaned and remaining data are ready to move forward to the preprocessing phase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. STORE CLEANED DATASET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set file location and name\n",
    "data_file_location = \"..\\\\data\\\\interim\\\\\"\n",
    "data_file_name = \"kddcup99_cleaned\"\n",
    "data_file_ext = \".csv\"\n",
    "\n",
    "# Store cleaned dataset to a csv file\n",
    "data.to_csv(data_file_location + data_file_name + data_file_ext)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}