notebooks/04_fcnn_mnist_shuffled.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tbTYFRhJoaBu"
   },
   "source": [
    "# MNIST digit classification on shuffled pixels with and without hidden layers\n",
    "\n",
    "\n",
    "In this notebook you will use the MNIST dataset for a classification task. You will compare a the performace of a fully connected neural network with and without hidden layers.\n",
    "\n",
    "\n",
    "**Dataset:** You work with the MNIST dataset. We have 60'000 28x28 pixel greyscale images of digits and want to classify them into the right label (0-9). Note that we will shuffle the pixels in an image randomly.\n",
    "\n",
    "**Content:**\n",
    "* load the original MNIST data \n",
    "* shuffle the pixels of the images in the MNIST data\n",
    "* visualize samples of the data\n",
    "* flatten the data\n",
    "* use keras to train a fcNN with and without hidden layers and compare the perfomance on new unseen test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PEIS4WvpsT5t"
   },
   "source": [
    "#### Imports\n",
    "\n",
    "In the next two cells, we load all the required libraries and functions. We download the MNIST data, normalize the pixelvalues to be between 0 and 1, and seperate it into a training and validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y6S_hQX5oaBw",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# load required libraries:\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.style.use('default')\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Convolution2D, MaxPooling2D, Flatten , Activation\n",
    "from tensorflow.keras.utils import to_categorical \n",
    "from tensorflow.keras import optimizers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4h_3TS0CtJJb"
   },
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "#### Loading and preparing the MNIST data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "colab_type": "code",
    "id": "4sZ8lqFfoaB2",
    "outputId": "a9aa1a96-4707-440b-a43f-9de241343a5d"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "# separate x_train in X_train and X_val, same for y_train\n",
    "X_train=x_train[0:50000] / 255 #divide by 255 so that they are in range 0 to 1\n",
    "Y_train=to_categorical(y_train[0:50000],10) # one-hot encoding\n",
    "\n",
    "X_val=x_train[50000:60000] / 255\n",
    "Y_val=to_categorical(y_train[50000:60000],10)\n",
    "\n",
    "X_test=x_test / 255\n",
    "Y_test=to_categorical(y_test,10)\n",
    "\n",
    "del x_train, y_train, x_test, y_test\n",
    "\n",
    "X_train=np.reshape(X_train, (X_train.shape[0],28,28,1))\n",
    "X_val=np.reshape(X_val, (X_val.shape[0],28,28,1))\n",
    "X_test=np.reshape(X_test, (X_test.shape[0],28,28,1))\n",
    "\n",
    "print(X_train.shape)\n",
    "print(X_val.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3ly7CrHtLUP9"
   },
   "source": [
    "Let's visualize the first 4 MNIST images before shuffling the pixels randomly around. It is very easy to recognise the true label of the digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 285
    },
    "colab_type": "code",
    "id": "c4gUgwGUeftF",
    "outputId": "4b0d711f-0756-4580-f59f-70f6275ae7ff"
   },
   "outputs": [],
   "source": [
    "# visualize the 4 first mnist images before shuffling the pixels\n",
    "plt.figure(figsize=(12,12))\n",
    "for i in range(0,4):\n",
    "    plt.subplot(1,4,(i+1))\n",
    "    plt.imshow((X_train[i,:,:,0]),cmap=\"gray\")\n",
    "    plt.title('true label: '+str(np.argmax(Y_train,axis=1)[i]))\n",
    "    #plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define a function to shuffle the pixels around. All images are shuffled in the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to shuffle the pixel order within an image\n",
    "# used to shuffel the pixels of all mnist images in the same manner\n",
    "\n",
    "def shuffel_pixels(idx, data):\n",
    "    data_new=np.zeros((data.shape))\n",
    "    for i,img in enumerate(data):\n",
    "        data_new[i] = img.flatten()[idx].reshape((28,28,1))\n",
    "    return data_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "shuffel_idx = np.random.permutation(np.arange(28*28))\n",
    "X_train_shuffle = shuffel_pixels(shuffel_idx, X_train)\n",
    "X_val_shuffle = shuffel_pixels(shuffel_idx, X_val)\n",
    "X_test_shuffle = shuffel_pixels(shuffel_idx, X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize the 4 first mnist images after shuffling the pixels\n",
    "plt.figure(figsize=(12,12))\n",
    "for i in range(0,4):\n",
    "    plt.subplot(1,4,(i+1))\n",
    "    plt.imshow((X_train_shuffle[i,:,:,0]),cmap=\"gray\")\n",
    "    plt.title('true label: '+str(np.argmax(Y_train,axis=1)[i]))\n",
    "    #plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "trSndjWX4HO7"
   },
   "source": [
    "## fcNN as classification model for MNIST data\n",
    "Now we want to train a fcNN to classify the shffled MNIST data. \n",
    "We use two network architectures:\n",
    "* fcnn with no hidden layers\n",
    "* fcnn with two hidden layers (100 and 50)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LB_m8HJaOMsY"
   },
   "source": [
    "Because we will use fcNN we need to flatten our inuput into a 1d vector. We do this in the next cell with reshape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CoSUhNzRfbj4"
   },
   "outputs": [],
   "source": [
    "# prepare data for fcNN - we need a vector as input\n",
    "X_train_shuffle_flat = X_train_shuffle.reshape([X_train_shuffle.shape[0], 784])\n",
    "X_val_shuffle_flat = X_val_shuffle.reshape([X_val_shuffle.shape[0], 784])\n",
    "X_test_shuffle_flat = X_test_shuffle.reshape([X_test_shuffle.shape[0], 784])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uOYYJFOh62qC"
   },
   "source": [
    "### Train the fcNN on the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "xILWGAnpC0Bt",
    "outputId": "0e34be12-36ad-4a46-afae-d5be75a01bbe"
   },
   "outputs": [],
   "source": [
    "# check the shape\n",
    "X_train_shuffle_flat.shape,Y_train.shape,X_val_shuffle_flat.shape,Y_val.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bulk0OpFOpeO"
   },
   "source": [
    "Here we define the nework. In the output we predict the probability for the 10 digits with the softmax actication function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EAispBe0oaCH"
   },
   "outputs": [],
   "source": [
    "# define fcNN\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Dense(10, batch_input_shape=(None, 784),activation=\"softmax\"))\n",
    "\n",
    "# compile model and intitialize weights\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 372
    },
    "colab_type": "code",
    "id": "7xfW6yzqoaCK",
    "outputId": "32ab1b57-ea77-4312-cea5-eea73391b7a1"
   },
   "outputs": [],
   "source": [
    "# summarize model along with number of model weights\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 390
    },
    "colab_type": "code",
    "id": "mhvTpCtfoaCQ",
    "outputId": "2810f9b9-039e-48c9-bdeb-6b1f28fba184"
   },
   "outputs": [],
   "source": [
    "# train the model\n",
    "history=model.fit(X_train_shuffle_flat, Y_train, \n",
    "                  batch_size=128, \n",
    "                  epochs=10,\n",
    "                  verbose=2, \n",
    "                  validation_data=(X_val_shuffle_flat, Y_val)\n",
    "                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 428
    },
    "colab_type": "code",
    "id": "t39HSiHcoaCT",
    "outputId": "005c699f-aa7f-4ccf-8317-2ad6c982b4ce"
   },
   "outputs": [],
   "source": [
    "# plot the development of the accuracy and loss during training\n",
    "plt.figure(figsize=(12,4))\n",
    "plt.subplot(1,2,(1))\n",
    "plt.plot(history.history['accuracy'],linestyle='-.')\n",
    "plt.plot(history.history['val_accuracy'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'valid'], loc='lower right')\n",
    "plt.subplot(1,2,(2))\n",
    "plt.plot(history.history['loss'],linestyle='-.')\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'valid'], loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iCaoy8l4oaCW"
   },
   "source": [
    "#### Prediction on the test set \n",
    "\n",
    "Now, let's use the fcNN that was trained to predict new unseen data (our testdata).\n",
    "We determine the confusion matrix and the accuracy on the testdata to evaluate the classification performance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 212
    },
    "colab_type": "code",
    "id": "z1U4S7o5oaCY",
    "outputId": "de83a545-08bf-40c7-bcd8-9d99eb855d9d"
   },
   "outputs": [],
   "source": [
    "pred=model.predict(X_test_shuffle_flat)\n",
    "print(confusion_matrix(np.argmax(Y_test,axis=1),np.argmax(pred,axis=1)))\n",
    "acc_fc = np.sum(np.argmax(Y_test,axis=1)==np.argmax(pred,axis=1))/len(pred)\n",
    "print(\"Acc = \" , acc_fc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the second fcNN on the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we define the nework with two hidden layers (100, 50). We use the sigmoid activation function on the hidden layers. In the output we predict the probability for the 10 digits with the softmax actication function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define fcNN with 2 hidden layers\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Dense(100, batch_input_shape=(None, 784)))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(50))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "# compile model and intitialize weights\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# summarize model along with number of model weights\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train the model\n",
    "history=model.fit(X_train_shuffle_flat, Y_train, \n",
    "                  batch_size=128, \n",
    "                  epochs=10,\n",
    "                  verbose=2, \n",
    "                  validation_data=(X_val_shuffle_flat, Y_val)\n",
    "                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the development of the accuracy and loss during training\n",
    "plt.figure(figsize=(12,4))\n",
    "plt.subplot(1,2,(1))\n",
    "plt.plot(history.history['accuracy'],linestyle='-.')\n",
    "plt.plot(history.history['val_accuracy'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'valid'], loc='lower right')\n",
    "plt.subplot(1,2,(2))\n",
    "plt.plot(history.history['loss'],linestyle='-.')\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'valid'], loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred=model.predict(X_test_shuffle_flat)\n",
    "print(confusion_matrix(np.argmax(Y_test,axis=1),np.argmax(pred,axis=1)))\n",
    "acc_fc = np.sum(np.argmax(Y_test,axis=1)==np.argmax(pred,axis=1))/len(pred)\n",
    "print(\"Acc = \" , acc_fc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wv898fZgjb-Z"
   },
   "source": [
    "## Exercise\n",
    "\n",
    "Compare the performace of the fully connected neural networks on the original and the shuffled Mnist data.  \n",
    "What do you observe?\n",
    "How do you explain your observation?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6qUgfXcshaZY"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "04_fcnn_mnist_shuffled.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}