notebooks/20_preprocessing_tabular_data_with_keras.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MyVt3izwGLdg"
      },
      "source": [
        "# Preprocessing tabular data with Keras\n",
        "\n",
        "Keras offers various preprocessing functions for tabular data, which are implemented as layers and can be used as part of a model, just like the ordinary model layers we have used so far.\n",
        "\n",
        "Some of them (like `Normalisation`) need to be adapted to data before we start training, an require a bit of extra work before we can get started. We'll do this extra work in this notebook.\n",
        "\n",
        "The data we use contains diagnostic information related to coronary artery disease, and our goal is to predict the presence of disease. The details of te dataset are available at the UCI ML dataset [repository](https://archive.ics.uci.edu/dataset/45/heart+disease).\n",
        "\n",
        "You can also find additional examples of tabular data preprocessing in the [Keras examples](https://keras.io/examples/structured_data/) section.\n",
        "\n",
        "In this notebook there are no real exercises (apart from the last one), it is more to show how things can be done. But try to pay attention wehn running it so that you understand what is going on.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V07DA-GBX6KC"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import keras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e2s7rhhmIjfO"
      },
      "source": [
        "We can try reading the CSV file using Pandas this time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OegCmaieYjXl"
      },
      "outputs": [],
      "source": [
        "file_url = \"http://storage.googleapis.com/download.tensorflow.org/data/heart.csv\"\n",
        "dataframe = pd.read_csv(file_url)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7QAnFwUZYqej"
      },
      "outputs": [],
      "source": [
        "dataframe.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2_EeEqOaIpbv"
      },
      "source": [
        "Remove the target feature, and create a TensorFlow dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LuHZ_UX085SI"
      },
      "outputs": [],
      "source": [
        "labels = dataframe.pop(\"target\")\n",
        "labels = tf.expand_dims(labels, -1)\n",
        "ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iZGLO7LyIwVt"
      },
      "source": [
        "If we like, we can test the contents of our dataset object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ShWM1Fa29BrZ"
      },
      "outputs": [],
      "source": [
        "for x, y in ds.take(1):\n",
        "    print(x, y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XcfkBzh4eSOk"
      },
      "source": [
        "## Define feature types, and set up functions for data proprocessing\n",
        "\n",
        "We want to treat numerical and categorical features differently, and in this case there are also two variants of categorical features -- those encodes as integers (ordinal), and those encoded as a string.\n",
        "\n",
        "Let's call them `numerical`, `categorical_integer`, and `categorical_string`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EXdXTG9LeY7B"
      },
      "outputs": [],
      "source": [
        "feature_types = {\n",
        "    \"age\":      \"numerical\",\n",
        "    \"sex\":      \"categorical_integer\",\n",
        "    \"cp\":       \"categorical_integer\",\n",
        "    \"trestbps\": \"numerical\",\n",
        "    \"chol\":     \"numerical\",\n",
        "    \"fbs\":      \"categorical_integer\",\n",
        "    \"restecg\":  \"categorical_integer\",\n",
        "    \"thalach\":  \"numerical\",\n",
        "    \"exang\":    \"categorical_integer\",\n",
        "    \"oldpeak\":  \"numerical\",\n",
        "    \"slope\":    \"numerical\",\n",
        "    \"ca\":       \"numerical\",\n",
        "    \"thal\":     \"categorical_string\"\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ANgTFY-KYRa"
      },
      "source": [
        "For the actual treatment of the different features -- loop over all of them and declare the relevant preprocessing layer to apply.\n",
        "\n",
        "We collect them all in a dict."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6EQrexOJcaHd"
      },
      "outputs": [],
      "source": [
        "processing_layers = {}\n",
        "\n",
        "\n",
        "for feature_name, feature_type in feature_types.items():\n",
        "\n",
        "    # In order to adapt the layers to data, we need a dataset that\n",
        "    # contains *only* this feature. Create it here.\n",
        "    feature_ds = ds.map(lambda x, y: x[feature_name])\n",
        "    feature_ds = feature_ds.map(lambda x: tf.expand_dims(x, -1))\n",
        "\n",
        "\n",
        "    if feature_type == \"numerical\":\n",
        "\n",
        "        # Numerical data -> Normalise\n",
        "        normaliser = keras.layers.Normalization()\n",
        "        normaliser.adapt(feature_ds)\n",
        "\n",
        "        processing_layers[feature_name] = normaliser\n",
        "\n",
        "    if feature_type == \"categorical_integer\":\n",
        "\n",
        "        # Ordinal data -> Create one-hot encoded features\n",
        "        integer_lookup = keras.layers.IntegerLookup(output_mode=\"one_hot\")\n",
        "        integer_lookup.adapt(feature_ds)\n",
        "\n",
        "        processing_layers[feature_name] = integer_lookup\n",
        "\n",
        "    if feature_type == \"categorical_string\":\n",
        "\n",
        "        # String data -> Create one-hot encoded features\n",
        "        string_lookup = keras.layers.StringLookup(output_mode=\"one_hot\")\n",
        "        string_lookup.adapt(feature_ds)\n",
        "\n",
        "        processing_layers[feature_name] = string_lookup\n",
        "\n",
        "\n",
        "# Print just to see if it looks right\n",
        "for name, layer in processing_layers.items():\n",
        "    print(name, layer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Nysv8HAMB0r"
      },
      "source": [
        "## Apply the pre-processing\n",
        "\n",
        "Here we have two options:\n",
        "1. Apply to the TensorFlow dataset, and create a new dataset containing preprocessed data\n",
        "2. Add it as part of the Keras model\n",
        "\n",
        "In this case we choose option 1, while in the next notebook we have a look at option 2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XHl3GkzRM8QZ"
      },
      "source": [
        "A short wrapper function to apply our dict of layers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UM7TAyXAfE24"
      },
      "outputs": [],
      "source": [
        "def apply_preprocessing(features, target):\n",
        "\n",
        "    for feature_name in features:\n",
        "\n",
        "        layer = processing_layers[feature_name]\n",
        "        features[feature_name] = layer(features[feature_name])\n",
        "\n",
        "    return dict(features), target"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9_304tANNCxR"
      },
      "source": [
        "Apply it to the dataset to create a new one:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f1VDykMzgDNP"
      },
      "outputs": [],
      "source": [
        "processed_ds = ds.map(apply_preprocessing)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jDVZ8LvAbN1r"
      },
      "source": [
        "Again we can hava a look to be sure.\n",
        "\n",
        "Note that now the different features can have different shapes, because the one-hot encoding adds new columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bDFxp3I97445"
      },
      "outputs": [],
      "source": [
        "for x, y in processed_ds.take(1):\n",
        "    for name in x:\n",
        "        print(name, x[name])\n",
        "    print(\"target:\", y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sALf3KyINXZU"
      },
      "source": [
        "## Define the model\n",
        "\n",
        "All Keras models start with an `Input` layer, but here we need to connect the different outputs to an `Input` layer with the correct shape.\n",
        "\n",
        "Let's connect them by name and set the appropriate data type and shape."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pncbPemgbNYS"
      },
      "outputs": [],
      "source": [
        "input_layers = {}\n",
        "for feature_name, feature_type in feature_types.items():\n",
        "\n",
        "    if feature_type == \"numerical\":\n",
        "        input_layers[feature_name] = keras.layers.Input(name=feature_name, shape=(1,), dtype=\"float32\")\n",
        "\n",
        "    else:\n",
        "        num_categories = len(processing_layers[feature_name].get_vocabulary())\n",
        "        input_layers[feature_name] = keras.layers.Input(name=feature_name, shape=(num_categories,), dtype=\"int32\")\n",
        "\n",
        "\n",
        "for n, l in input_layers.items():\n",
        "    print(n, l)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6aCQOEFfPSmR"
      },
      "source": [
        "### Model definition as a class\n",
        "\n",
        "Let's look at a more advanced way to define a model -- by subclassing the base `Layer` class.\n",
        "\n",
        "Here we need three functions:\n",
        "- `__init__`, containing the layers our model will contain,\n",
        "- `call`, stating what happens when we run the model, and\n",
        "- `build`, which doesn't have to contain anything, but must be defined.\n",
        "\n",
        "This way of defining a model is the standard way for the other big deep learning library, called [PyTorch](https://pytorch.org/). So in case you happen to use that at some later point, you will recognise the structure."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qpg27YDnjXOB"
      },
      "outputs": [],
      "source": [
        "class MyModel(keras.layers.Layer):\n",
        "\n",
        "    def __init__(self, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.dense1 = keras.layers.Dense(32, activation=\"relu\")\n",
        "        self.dense2 = keras.layers.Dense(1, activation=\"sigmoid\")\n",
        "\n",
        "    def call(self, inputs):\n",
        "\n",
        "        # We could also have put the input and processing layers here,\n",
        "        # if not applying them directly to the TF dataset.\n",
        "\n",
        "        all_features = keras.layers.concatenate(list(inputs.values()))  # merge the different inputs\n",
        "        x = self.dense1(all_features)   # apply the first Dense layer\n",
        "        output = self.dense2(x)         # apply the second (classification) layer\n",
        "\n",
        "        return output\n",
        "\n",
        "    def build(self, input_shape):\n",
        "        self.built = True\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "orLqUb4jQ9bU"
      },
      "source": [
        "Create a `keras.Model()` instance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VaALIzokkJg6"
      },
      "outputs": [],
      "source": [
        "def create_model():\n",
        "    output = MyModel()(input_layers)\n",
        "    model = keras.Model(input_layers, output)\n",
        "\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qj-ZsbZ_kYVL"
      },
      "outputs": [],
      "source": [
        "model = create_model()\n",
        "model.compile(\n",
        "    optimizer=\"adam\",\n",
        "    loss=\"binary_crossentropy\",\n",
        "    metrics=[\"accuracy\"]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PjTyupnRRFo4"
      },
      "source": [
        "Plot the model, to see where the data goes:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NTNb_i5knbs0"
      },
      "outputs": [],
      "source": [
        "keras.utils.plot_model(model, show_shapes=True, rankdir=\"TD\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "do9QTaiHRI6U"
      },
      "source": [
        "## Run the model\n",
        "\n",
        "Run it on the processed dataset.\n",
        "\n",
        "Now we haven't created a separate validation dataset, but this you can fix yourself :)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pC9kX28xkkbm"
      },
      "outputs": [],
      "source": [
        "model.fit(processed_ds, epochs=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rep8ejzcSBBN"
      },
      "source": [
        "### <span style=\"color: red; font-weight: bold;\">Exercise:<span>\n",
        "\n",
        "Study the class implementation of the network (`MyModel`) above. Try to improve it by adding\n",
        "- A second hidden `Dense` layer\n",
        "- Batch normalisation\n",
        "- `Dropout` between the `Dense` layers."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}