Open_conversion_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading data from a csv file, and pre-processing data for further analysis\n",
    "\n",
    "<p style=\"text-align: justify\">In this notebook, we will see how to perform simple and essential tasks when starting data analysis with Python. A procedural approach which is more comprehensive for beginners is used, without using the <a href=\"https://pandas.pydata.org/\">Pandas</a> and <a href=\"https://www.scipy.org/\">SciPy</a> packages, that are regularly used for data analysis and data scraping. This choice is done for better understanding of the different functions that are already implemented in these libraries, and to comprehend from scratch the programming behind. The different lines of code are well commented for the reader to understand. The first step in the current notebook is to import the different built-in modules in Python that are required to compute our code. </p> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv # csv for reading the csv files\n",
    "from math import sqrt # square root function from math module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load CSV file\n",
    "\n",
    "<p style=\"text-align: justify\">The data could be obtained by <a href=\"https://en.wikipedia.org/wiki/Data_scraping\">data scrapping</a> from a website for instance, or via an <a href=\"https://en.wikipedia.org/wiki/Application_programming_interface\">Application Programming Interface (API)</a>. </p> \n",
    "\n",
    "<p style=\"text-align: justify\">Here instead, we consider loading the data from a CSV file as it is one the most common data-exchange format. A <a href=\"https://en.wikipedia.org/wiki/Comma-separated_values\">comma separated values</a> (CSV) file contains different values separated by a delimiter, which acts as a database table or an intermediate form of a database table. In other words, a CSV file file is a set of database rows and columns stored in a text file such that the rows are separated by a new line, while the columns are separated by a semicolon or a comma. A CSV file is primarily used to transport data between two databases of different formats through a computer program.</p>\n",
    "\n",
    "<p style=\"text-align: justify\">The function <i>load_csv()</i> below, scrapes the data from a CSV file by reading the different rows of the file, and by storing the rows one by one in a list.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Load a CSV file\n",
    "def load_csv(filename):\n",
    "    dataset = list()  # creates a list where the row will be stored\n",
    "    with open(filename, 'r') as file: # open the file in reading mode \n",
    "        csv_reader = csv.reader(file) # csv.reader built-in function\n",
    "        for row in csv_reader: # loop on the rows of the file\n",
    "            if not row:\n",
    "                continue\n",
    "            dataset.append(row) # adding row to the dataset\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert column of strings into column of floating numbers\n",
    "\n",
    "<p style=\"text-align: justify\">The data acquired from the function above can be of different types. The data can be made of string of characters or floating numbers for instance. When reading from a CSV file, the data is parsed as a string of character. However, in order to perform computational operations and analysis on the data, this requires that all string of numbers are transformed into floating numbers.</p>   \n",
    "\n",
    "<p style=\"text-align: justify\">The function <i>str_column_to_float()</i> below implements this operation for a column of the dataset. The function can then later be looped on the desired columns to turn strings into floating numbers. \n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Convert string column to float\n",
    "def str_column_to_float(dataset, column):\n",
    "    for row in dataset:\n",
    "        row[column] = float(row[column].strip()) # strip() function strips all characters from the beginning and the end of the string (default whitespace characters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert column of strings into column of integers\n",
    "\n",
    "<p style=\"text-align: justify\">The function <i>str_column_to_float()</i> will usually be used to transform all the features of the dataset into floating numbers to perform different operations on them. However, when considering dataset for classification especially, different classes may be a string of characters as well. For instance, let's consider that in a group of person, these people are classified by age and we have 2 classes, 'Below 30' and 'Above 30'. Operations won't be performed on these classes as they will just be used for comparison with predictions, for instance in a machine learning classification problem.</p>\n",
    "\n",
    "<p style=\"text-align: justify\">However, transforming the data from this column into integers, can ease the classification process. For example, the string 'Below 30' is replaced by the number 0 and 'Above 30' by 1. In this way, predictions from the features are easier to compare with the classification values.</p>\n",
    "\n",
    "<p style=\"text-align: justify\">The function <i>str_column_to_int()</i> below allows this conversion operation for a particular column. First, a list 'class_values' containing the different rows of the column is created. From this list, 'unique' is created which is a set looking at all unique values of the list 'class_values'. A dictionary 'lookup' is then initialised to match integer values to the string of characters. The string of characters can now be referenced with their integer values, which will be usually preferred for data analysis.</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Convert string column to integer\n",
    "def str_column_to_int(dataset, column):\n",
    "    class_values = [row[column] for row in dataset] # comprehension list with data of the considered column stored\n",
    "    unique = set(class_values) # set where all unique values are stored\n",
    "    lookup = dict() #\n",
    "    for i, value in enumerate(unique): # enumerate function (loop + automatic counter of values)\n",
    "        lookup[value] = i # value is the key of dictionary (the string of characters) and i is the number affiliated\n",
    "    for row in dataset:  # modifying column in dataset\n",
    "        row[column] = lookup[row[column]]\n",
    "    return lookup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align: justify\">We can now try all the different functions defined above on a particular dataset. In this notebook, we will use the <a href=\"https://en.wikipedia.org/wiki/Iris_flower_data_set\"><i>Iris</i> flower dataset</a>, which is one of the most famous and simplest dataset available for data analysis and machine learning purposes. The dataset can be downloaded on the <a href=\"https://archive.ics.uci.edu/ml/\">UCI Machine Learning Repository</a> website, which is the biggest open source of available data repositories for machine learning. The dataset can be downloaded at the following address <a href=\"https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\">https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data</a>, and needs to be saved as a CSV file.\n",
    "\n",
    "</p>\n",
    "\n",
    "<p style=\"text-align: justify\">The Iris Flower Dataset involves predicting the flower species given measurements of iris flowers. It is a multiclass classification problem. The number of observations for each class is balanced. There are 150 observations with 4 input variables and 1 output variable. The variable names are as follows:</p>\n",
    "\n",
    "1. Sepal length in cm.\n",
    "2. Sepal width in cm.\n",
    "3. Petal length in cm.\n",
    "4. Petal width in cm.\n",
    "5. Class (Iris Setosa, Iris Versicolour, Iris Virginica).\n",
    "\n",
    "<p style=\"text-align: justify\">The baseline performance of predicting the most prevalent class is a classification accuracy of approximately 26%. A sample of the first 5 rows is listed below.</p>\n",
    "\n",
    "\n",
    "    5.1,3.5,1.4,0.2,Iris-setosa\n",
    "    4.9,3.0,1.4,0.2,Iris-setosa\n",
    "    4.7,3.2,1.3,0.2,Iris-setosa\n",
    "    4.6,3.1,1.5,0.2,Iris-setosa\n",
    "    5.0,3.6,1.4,0.2,Iris-setosa\n",
    "    \n",
    "<p style=\"text-align: justify\">The piece of code below, loads the Iris dataset, and performs the different operations defined above. Some information is printed to see the modifications created on the stored dataset.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data file iris.csv with 150 rows and 5 columns\n",
      "First row of the dataset:  ['5.1', '3.5', '1.4', '0.2', 'Iris-setosa']\n",
      "--------------------------------------\n",
      "First row of modified dataset:  [5.1, 3.5, 1.4, 0.2, 0]\n",
      "{'Iris-setosa': 0, 'Iris-virginica': 1, 'Iris-versicolor': 2}\n"
     ]
    }
   ],
   "source": [
    "# Load iris dataset\n",
    "filename = 'iris.csv'\n",
    "dataset = load_csv(filename) \n",
    "\n",
    "print('Loaded data file {0} with {1} rows and {2} columns'.format(filename, len(dataset), len(dataset[0])))\n",
    "print('First row of the dataset: ', dataset[0]) # print first line of the dataset\n",
    "print('--------------------------------------')\n",
    "\n",
    "# convert string columns to float \n",
    "for i in range(len(dataset[0])-1): # loop on all columns except last one\n",
    "    str_column_to_float(dataset, i)\n",
    "# convert class column (last one) to int \n",
    "lookup = str_column_to_int(dataset, 4)\n",
    "\n",
    "print('First row of modified dataset: ', dataset[0]) # print first line of updated dataset\n",
    "print(lookup) # print lookup dictionary containing the classification of the species and their corresponding number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalization of set of data\n",
    "\n",
    "<p style=\"text-align: justify\">When it comes to analyzing data, feature scaling through normalization or standardization can be an important pre-processing step for many machine learning algorithms. While many algorithms require features to be normalized, intuitively we can think of <a href =\"https://en.wikipedia.org/wiki/Principal_component_analysis\">Principal Component Analysis (PCA)</a> as being a prime example of when normalization is important. PCA is a technique used for identification of a smaller number of uncorrelated variables known as principal components from a larger set of data. Indeed, when a set of data possess many features, it is important to reduce that number of features for training the models in a more efficient way. Also, data normalization makes sense when the features of a dataset are of different genders and units. \n",
    "\n",
    "<p style=\"text-align: justify\">In the next part of this notebook, we consider The UCI Wine Quality Dataset, which can be directly downloaded in CSV format at <a href=\"https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv\">https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv</a>, and it involves predicting the quality of white wines on a scale given chemical measures of each wine. It is a multiclass classification problem, but could also be framed as a regression problem. The number of observations for each class is not balanced. There are 4,898 observations with 11 input variables and 1 output variable. This dataset has continuous features that are heterogeneous in scale due to differing properties that they measure (i.e alcohol content, and malic acid). The variable names are as follows:</p>\n",
    "\n",
    "1. Fixed acidity.\n",
    "2. Volatile acidity.\n",
    "3. Citric acid.\n",
    "4. Residual sugar.\n",
    "5. Chlorides.\n",
    "6. Free sulfur dioxide.\n",
    "7. Total sulfur dioxide.\n",
    "8. Density.\n",
    "9. pH.\n",
    "10. Sulphates.\n",
    "11. Alcohol.\n",
    "12. Quality (score between 0 and 10).\n",
    "\n",
    "<p style=\"text-align: justify\">The baseline performance of predicting the mean value is a <a href=\"https://en.wikipedia.org/wiki/Root-mean-square_deviation\">RMSE</a> of approximately 0.148 quality points. A sample of the first 5 rows is listed below.</p>\n",
    "\n",
    "    7,0.27,0.36,20.7,0.045,45,170,1.001,3,0.45,8.8,6\n",
    "    6.3,0.3,0.34,1.6,0.049,14,132,0.994,3.3,0.49,9.5,6\n",
    "    8.1,0.28,0.4,6.9,0.05,30,97,0.9951,3.26,0.44,10.1,6\n",
    "    7.2,0.23,0.32,8.5,0.058,47,186,0.9956,3.19,0.4,9.9,6\n",
    "    7.2,0.23,0.32,8.5,0.058,47,186,0.9956,3.19,0.4,9.9,6\n",
    " \n",
    "<p style=\"text-align: justify\">As specified above, it can be interesting to normalize the features of the dataset, not only because it is required for some machine learning algorithms, but also as a way to put all data on the same scale. The first way to re-scale the data is to use the <b>Min-Max scaling</b> or more simply called \"Normalization\". That's usually the way we think, when talking about normalization of a dataset. It is the simplest method and consists in rescaling the range of features to scale the range in [0, 1] or [−1, 1]. Selecting the target range depends on the nature of the data. The general formula is given as</p>\n",
    "\n",
    "$$x_{scaled} = \\dfrac{x-min(x)}{max(x)-min(x)} \\quad (1) \\quad ,   $$ \n",
    "\n",
    "<p style=\"text-align: justify\">where $x$ is the original value, $x_{scaled}$ is the normalized value.</p>\n",
    "\n",
    "<p style=\"text-align: justify\">From a coding point of view, it is first necessary to find the minimum and maximum of each feature (column) of the dataset, with the function <i>dataset_minmax()</i>. To do so a list variable 'minmax' is initialized, then a loop is implemented on all columns of the dataset, with the values of a column once at a time stored in the variable 'colvalues'. The minimum and maximum of 'colvalues' can then be found before adding them to the 'minmax' list and continuing the iterative loop. Finally, the 'minmax' list contains the minimum and maximum values of each column of the dataset, as this is necessary for the normalization procedure. The dataset can then be normalized by looping equation (1) over all values in the function <i>Normalize_Dataset()</i>.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### Normalize Data ###########\n",
    "\n",
    "# Find the min and max values for each column\n",
    "def dataset_minmax(dataset):\n",
    "    minmax = list()\n",
    "    for i in range(len(dataset[0])):\n",
    "        colvalues = [row[i] for row in dataset]\n",
    "        min_value = min(colvalues) \n",
    "        max_value = max(colvalues)\n",
    "        minmax.append([min_value, max_value])\n",
    "    return minmax\n",
    "\n",
    "# Normalize the dataset except last row for classification values\n",
    "def Normalize_Dataset(dataset, minmax):\n",
    "    for row in dataset:\n",
    "        for i in range(len(row)-1):\n",
    "            row[i] = (row[i] - minmax[i][0]) / (minmax[i][1] - minmax[i][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align: justify\">The piece of code below, loads the Wine Quality dataset, and performs the different operations defined above. The <i>load_csv()</i> function has been modified a bit to consider ';' as the delimiter and to skip the header of the dataset. Indeed, in real cases and depending on the structure of the dataset, some changes may be required to be able to import data from a CSV file. Of course, <a href=\"https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html\">pandas.read_csv()</a> contains all these type of features, and much more, but it is interesting to see from scratch how data structure can bring some issues. Pre-processing is actually, one of if not the hardest part in machine learning. By printing the first row of the regular and normalized dataset, we can see the normalization with values between [0,1] except for the last column, which is conserved for classification purposes.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First row of regular dataset:  [7.0, 0.27, 0.36, 20.7, 0.045, 45.0, 170.0, 1.001, 3.0, 0.45, 8.8, 6.0]\n",
      "\n",
      "First row of normalized dataset:  [0.30769230769230776, 0.18627450980392157, 0.21686746987951808, 0.308282208588957, 0.10682492581602374, 0.14982578397212543, 0.37354988399071926, 0.26778484673221237, 0.25454545454545446, 0.26744186046511625, 0.12903225806451626, 6.0]\n"
     ]
    }
   ],
   "source": [
    "## Load a CSV file\n",
    "def load_csv(filename):\n",
    "    dataset = list()\n",
    "    with open(filename, 'r') as file:\n",
    "        csv_reader = csv.reader(file, delimiter = \";\") # delimiter specified\n",
    "        #Return next item in iterator to skip header\n",
    "        next(csv_reader)\n",
    "        for row in csv_reader:\n",
    "            if not row:\n",
    "                continue\n",
    "            dataset.append(row)\n",
    "    return dataset\n",
    "\n",
    "# load white wine quality dataset\n",
    "filename = 'winequality-white.csv'\n",
    "dataset = load_csv(filename) \n",
    "# convert string columns to float \n",
    "for i in range(len(dataset[0])): # loop on all columns here\n",
    "    str_column_to_float(dataset, i)\n",
    "\n",
    "print('First row of regular dataset: ', dataset[0])\n",
    "\n",
    "# Calculate min and max for each column\n",
    "minmax = dataset_minmax(dataset)\n",
    "\n",
    "# Normalize columns\n",
    "Normalize_Dataset(dataset, minmax)\n",
    "\n",
    "print()\n",
    "print('First row of normalized dataset: ', dataset[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset standardization\n",
    "\n",
    "<p style=\"text-align: justify\">In machine learning, we can handle various types of data, e.g. audio signals and pixel values for image data, and this data can include multiple dimensions. Feature standardization makes the values of each feature in the data have zero-mean (when subtracting the mean in the numerator) and unit-variance. This method is widely used for normalization in many machine learning algorithms (e.g., <a href=\"https://en.wikipedia.org/wiki/Support_vector_machine\">support vector machines</a>, <a href=\"https://en.wikipedia.org/wiki/Logistic_regression\">logistic regression</a>, and <a href=\"https://en.wikipedia.org/wiki/Artificial_neural_network\">artificial neural networks</a>). The general method of calculation is to determine the distribution <a href=\"https://en.wikipedia.org/wiki/Mean\">mean</a> and <a href=\"https://en.wikipedia.org/wiki/Standard_deviation\">standard deviation</a> for each feature. Next, we subtract the mean from each feature. Then we divide the values (mean is already subtracted) of each feature by its standard deviation.</p>\n",
    "\n",
    "$$x_{std} = \\dfrac{x-\\mu}{\\sigma} \\quad (2) \\quad , $$\n",
    "\n",
    "<p style=\"text-align: justify\">where $x_{std}$ is the standardized instance, $x$ the original instance value, $\\mu$ the mean of that instance, and $\\sigma$ its standard deviation.</p>\n",
    "\n",
    "<p style=\"text-align: justify\">In terms of programming, it requires first defining two functions <i>column_means()</i> and <i>column_stdevs()</i> that respectively calculate the means and standard deviations of the different columns of the dataset, and return the results stored in two lists 'means' and 'stdevs', respectively. The function <i>Standardize_Dataset()</i> then executes equation (2) over all instances of the dataset.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Standardize Data ######\n",
    "\n",
    "# Calculate column means\n",
    "def column_means(dataset):\n",
    "    means = [0 for i in range(len(dataset[0]))]\n",
    "    for i in range(len(dataset[0])):\n",
    "        col_values = [row[i] for row in dataset]\n",
    "        means[i] = sum(col_values) / float(len(dataset)) # sum of the values in column / number of elements\n",
    "    return means\n",
    "\n",
    "# Calculate column standard deviations\n",
    "def column_stdevs(dataset, means):\n",
    "    stdevs = [0 for i in range(len(dataset[0]))] # initialized comprehension list, number of elements = number of columns\n",
    "    for i in range(len(dataset[0])):\n",
    "        variance = [pow(row[i]-means[i], 2) for row in dataset] # variance with (x-mean)^2\n",
    "        stdevs[i] = sum(variance)\n",
    "        stdevs = [sqrt(x/(float(len(dataset)-1))) for x in stdevs] # standard deviation formula\n",
    "    return stdevs\n",
    "\n",
    "# Standardize the dataset\n",
    "def Standardize_Dataset(dataset, means, stdevs):\n",
    "    for row in dataset:\n",
    "        for i in range(len(row)):\n",
    "            row[i] = (row[i] - means[i]) / stdevs[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align: justify\">The piece of code below, loads the Wine Quality dataset, and performs the different operations defined above to standardize the set of data. By printing the first row of the regular and standardized dataset, we can see the differences in the data.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First row of regular dataset:  [7.0, 0.27, 0.36, 20.7, 0.045, 45.0, 170.0, 1.001, 3.0, 0.45, 8.8, 6.0]\n",
      "\n",
      "First row of standardized dataset:  [708.2194955678943, -40.113082681069145, 124.81798171258234, 67353.22272363336, -3.646644578687148, 39760.87498631665, 105672.1888463459, 28.871410918992115, -403.7289363226821, -40.132468023548995, -108.13877471367468, 0.13785606532441566]\n"
     ]
    }
   ],
   "source": [
    "## Load a CSV file\n",
    "def load_csv(filename):\n",
    "    dataset = list()\n",
    "    with open(filename, 'r') as file:\n",
    "        csv_reader = csv.reader(file, delimiter = \";\") # delimiter specified\n",
    "        #Return next item in iterator to skip header\n",
    "        next(csv_reader)\n",
    "        for row in csv_reader:\n",
    "            if not row:\n",
    "                continue\n",
    "            dataset.append(row)\n",
    "    return dataset\n",
    "\n",
    "# Load white wine quality dataset\n",
    "filename = 'winequality-white.csv'\n",
    "dataset = load_csv(filename) \n",
    "# convert string columns to float \n",
    "for i in range(len(dataset[0])): # loop on all columns here\n",
    "    str_column_to_float(dataset, i)\n",
    "\n",
    "print('First row of regular dataset: ', dataset[0])\n",
    "    \n",
    "    \n",
    "# Estimate mean and standard deviation\n",
    "means = column_means(dataset)\n",
    "stdevs = column_stdevs(dataset, means)\n",
    "# standardize dataset\n",
    "Standardize_Dataset(dataset, means, stdevs)\n",
    "\n",
    "print()\n",
    "print('First row of standardized dataset: ', dataset[0])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}