Facial Recognition Competition Using FastAi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the Facial Recognition Challenge Dataset with FastAi\n",
    "\n",
    "This is my take on lesson's [one](https://course.fast.ai/videos/?lesson=1) and [two](https://course.fast.ai/videos/?lesson=2) of the fastai course. I decided to use this library and see how well it will work out of the box on the [2013 facial recognition challenge](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading 2013 facial recognition dataset\n",
    "\n",
    "To download the competition dataset, simply go to the [facial expression recognition](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge) page on Kaggle and click on `download all`.\n",
    "Then you should extract the contents in the same directory as this jupyter notebook file.\n",
    "\n",
    "Your folder structure should look like this:\n",
    "\n",
    "```\n",
    ".\n",
    "├── example_submission.csv \n",
    "├── fer2013.tar.gz \n",
    "├── icml_face_data.csv \n",
    "├── test.csv \n",
    "├── train.csv\n",
    "├── cleaned.csv\n",
    "└── Facial Recognition Competition Using FastAi.ipynb\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading relevant libraries\n",
    "\n",
    "Every notebook starts with the following three lines; they ensure that any edits to libraries you make are reloaded here automatically, and also that any charts or images displayed are shown in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We import all the necessary packages. We are going to work with the [fastai V1 library](http://www.fast.ai/2018/10/02/fastai-ai/) which sits on top of [Pytorch 1.0](https://hackernoon.com/pytorch-1-0-468332ba5163). The fastai library provides many useful functions that enable us to quickly and easily build neural networks and train our models.\n",
    "\n",
    "We will also import [fastai.widgets](https://docs.fast.ai/widgets.image_cleaner.html#Image-Cleaner-Widget) which offer several widgets to support the workflow of a deep learning practitioner. The purpose of the widgets is to help you organize, clean, and prepare your data for your model. Widgets are separated by data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.vision import *\n",
    "from fastai.widgets import *\n",
    "\n",
    "import os\n",
    "import sys\n",
    "import cv2\n",
    "import shutil  \n",
    "import tarfile\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's extract fer2013.tar.gz file in the working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Extract fer2013tar.gz \n",
    "tf = tarfile.open(\"fer2013.tar.gz\")\n",
    "tf.extractall()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The competition dataset gives us a csv file with sets of pixels rather than the images themselves, the code below taken from the [competition's discussion](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/discussion/29428) lets us convert those pixels to black and white images.\n",
    "\n",
    "Thanks to [MadScientist](https://www.kaggle.com/madmlscientist) for this code snippet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "output_path =  \"images\"\n",
    "\n",
    "if os.path.exists(output_path):\n",
    "    os.system('rm -rf {}'.format(output_path))\n",
    "\n",
    "os.system('mkdir {}'.format(output_path))\n",
    "\n",
    "label_names = ['Angry', 'Disgust', 'Fear', 'Happy', 'Sad', 'Surprise', 'Neutral']\n",
    "\n",
    "data = np.genfromtxt('fer2013/fer2013.csv',delimiter=',',dtype=None, encoding=None)\n",
    "labels = data[1:,0].astype(np.int32)\n",
    "image_buffer = data[1:,1]\n",
    "images = np.array([np.fromstring(image, np.uint8, sep=' ') for image in image_buffer])\n",
    "usage = data[1:,2]\n",
    "dataset = zip(labels, images, usage)\n",
    "usage_path = \"\"\n",
    "for i, d in enumerate(dataset):\n",
    "    if(d[-1] == \"Training\" or d[-1] == \"PrivateTest\"):\n",
    "        usage_path = os.path.join(output_path, \"Training\")\n",
    "    else:\n",
    "        usage_path = os.path.join(output_path, d[-1])\n",
    "\n",
    "    label_path = os.path.join(usage_path, label_names[d[0]])\n",
    "    img = d[1].reshape((48,48))\n",
    "    img_name = '%08d.jpg' % i\n",
    "    img_path = os.path.join(label_path, img_name)\n",
    "    if not os.path.exists(usage_path):\n",
    "        os.system('mkdir {}'.format(usage_path))\n",
    "    if not os.path.exists(label_path):\n",
    "        os.system('mkdir {}'.format(label_path))\n",
    "    cv2.imwrite(img_path, img)\n",
    "\n",
    "    #     print('Write {}'.format(img_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option A: Using cleaned.csv file to filter the training dataset\n",
    "\n",
    "Since I have run this multiple times, I have obtained the `cleaned.csv` file which you will also obtain from the *Cleaning up* section of this jupyter notebook.\n",
    "\n",
    "Run the code below (Option A) and don't run Option B if you have obtained cleaned.csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('../cleaned.csv', header='infer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change path to where we formed our images\n",
    "path = \"images\"\n",
    "\n",
    "np.random.seed(42)\n",
    "data = ImageDataBunch.from_csv(path, folder=\".\", valid_pct=0.2, csv_labels='../cleaned.csv',\n",
    "ds_tfms=get_transforms(), size=224, num_workers=8).normalize(imagenet_stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option B: Load the training dataset from folder without using cleaned.csv file\n",
    "\n",
    "Uncomment and run this code instead of *Option A* if this is your first time running the jupyter notebook and you haven't obtained a cleaned.csv file yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change path to where we formed our images\n",
    "# path = \"/images\"\n",
    "# os.chdir(path)\n",
    "\n",
    "# # bs = 64\n",
    "# tfms = get_transforms(do_flip=False)\n",
    "# data = ImageDataBunch.from_folder(path, train = \"Training\", valid_pct=0.2, ds_tfms=tfms, size=26, num_workers=0, bs = 64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(f\"Classes in our data: {data.classes}\\nNumber of classes: {data.c}\\nTraining Dataset Length: {len(data.train_ds)}\\nValidation Dataset Length: {len(data.valid_ds)}\")\n",
    "\n",
    "data.show_batch(rows=3, columns = 5, figsize=(5,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model\n",
    "\n",
    "To train a model properly, I followed [Poonam's](https://forums.fast.ai/t/why-do-we-need-to-unfreeze-the-learner-everytime-before-retarining-even-if-learn-fit-one-cycle-works-fine-without-learn-unfreeze/41614/5) advice on the fastai forum on when to freeze and unfreeze the learner during model training. I suggested reading her approach to understanding how to optimize your model better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = cnn_learner(data, models.resnet34, metrics=[accuracy,error_rate])\n",
    "learn.fit_one_cycle(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('stage-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training with the backbone frozen allows us to only train the untrained layers in the head. Once those layers have converged somewhat, we unfreeze the entire model and continue training.\n",
    "\n",
    "> Note: With the fastai library, loading the model will load it in a frozen state by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('stage-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the learning rate finder, we are looking for the strongest downward slope that's kind of sticking around for quite a while. For this case, it seems that we don't have a downward slope so let's limit our learning rate between 3e<sup>-6</sup> and 3e<sup>-3</sup>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(2, max_lr=slice(3e-6,3e-3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('stage-2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretation\n",
    "\n",
    "We can use the ClassificationInterpretation class to have a look at what's going on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('stage-2');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interp = ClassificationInterpretation.from_learner(learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning up\n",
    "\n",
    "Some of our top losses aren't due to bad performance by our model. There are images in our data set that shouldn't be there.\n",
    "\n",
    "Using the `ImageCleaner` widget from `fastai.widgets` we can prune our top losses, removing photos that don't belong."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to get the file paths from our top_losses. We can do this with `.from_toplosses`. We then feed the top losses indexes and corresponding dataset to `ImageCleaner`.\n",
    "\n",
    "Notice that the widget will not delete images directly from disk but it will create a new csv file `cleaned.csv` from where you can create a new ImageDataBunch with the corrected labels to continue training your model.\n",
    "\n",
    "Note: Please Set the Number of images to a number that you'd like to view:\n",
    "ex: ```n_imgs=100```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db = (ImageList.from_folder(\"images/Training\")\n",
    "                   .split_none()\n",
    "                   .label_from_folder()\n",
    "                   .transform(get_transforms(), size=224)\n",
    "                   .databunch()\n",
    "     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_cln = cnn_learner(db, models.resnet34, metrics=error_rate)\n",
    "\n",
    "learn_cln.load('images/models/stage-2');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds, idxs = DatasetFormatter().from_toplosses(learn_cln)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `ImageCleaner` we will get the widget running inside our jupyter notebook and we can correct the labels or delete images that don't with any of our labels. This is important to reduce the noise in our dataset and increase the performance of our learner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ImageCleaner(ds, idxs, path, batch_size=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flag photos for deletion by clicking 'Delete'. Then click 'Next Batch' to delete flagged photos and keep the rest in that row. ImageCleaner will show you a new row of images until there is no more to show. In this case, the widget will show you images until there are none left from top_losses.ImageCleaner(ds, idxs)\n",
    "\n",
    "You can also find duplicates in your dataset and delete them! To do this, you need to run .from_similars to get the potential duplicates' ids and then run ImageCleaner with duplicates=True. The API works similarly as with misclassified images: just choose the ones you want to delete and click 'Next Batch' until there are no more images left."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validating the test set\n",
    "\n",
    "Since we were given the test set with labels out of the box, we will have to approach this as if we are validating our model rather than testing an unlabeled set of images. To do that, we will simply create a `data_test` from the images we have split my folders `Training` and `PublicTest` then validate to see how well our model performed. We do this rather than submitting the results because submissions for this competition is closed. For more information check out the [fastai docs](https://docs.fast.ai/data_block.html#Add-a-test-set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('stage-2')\n",
    "# ds_tfms=get_transforms(), size=224, num_workers=8\n",
    "data_test = (ImageList.from_folder(path)\n",
    "            .split_by_folder(train='Training', valid='PublicTest')\n",
    "            .label_from_folder()\n",
    "            .transform(tfms=get_transforms(), size=224)\n",
    "            .databunch()\n",
    "            .normalize()\n",
    "        )\n",
    "\n",
    "loss, acc, err_r = learn.validate(data_test.valid_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = str(np.round(loss, 3))\n",
    "print(f\"Our final model's training loss: {loss}, with Accuracy: {round(acc.item(), 3)} and Error Rate: {round(err_r.item(), 3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting the model in production\n",
    "\n",
    "To put my model in production, I used lankinen's approach and followed his instructions in his [medium article](https://medium.com/@lankinen/fastai-model-to-production-this-is-how-you-make-web-app-that-use-your-model-57d8999450cf). You can see it implemented in a different way on my github for a [multilabel classification model](https://github.com/kais-viz/fastai-v3).\n",
    "\n",
    "### Notes on the article\n",
    "* When trying to install `torch_nightly`, the URL provided gives a 404 error, to get around it, visit the URL and manually find the latest version applicable to your environment (Ubunutu 18) and install it.\n",
    "\n",
    "```\n",
    "wget https://download.pytorch.org/whl/nightly/cpu/torch_nightly-1.2.0.dev20190805%2Bcpu-cp36-cp36m-linux_x86_64.whl\n",
    "pip3 install torch_nightly-1.2.0.dev20190805%2Bcpu-cp36-cp36m-linux_x86_64.whl\n",
    "```\n",
    "\n",
    "* You might need to use t2.large instance which provides 8GB of RAM, and increasing the volume to 15GB because you might ran out of memory and disk space while installing fastai library.\n",
    "\n",
    "* We will need export.pkl file which we get from `data.export()`\n",
    "\n",
    "* Just before loading up the server, you should update torchvision's installation by type:\n",
    "`pip3 install torchvision`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}