Lab_4_Augment/Lab sheet 4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 4 - How to improve your results?\n",
    "\n",
    "In this lab session we will look at data augmentation and debugging strategies.\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "1. Learn how to perform data augmentation in tensorflow\n",
    "2. Experiment with different types of data augmentation\n",
    "3. Learn how to debug with tensorflow "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup (REMINDER)\n",
    "\n",
    "1. Login to BC4\n",
    "\n",
    "    ```\n",
    "    ssh <your_UoB_ID>@bc4login.acrc.bris.ac.uk\n",
    "    ```\n",
    "    \n",
    "2. Clone the repository\n",
    "\n",
    "    ```\n",
    "    git clone \"https://github.com/COMSM0018-Applied-Deep-Learning/labsheets.git\" ~/labsheets\n",
    "    ```\n",
    "\n",
    "3. Change to the lab 4 directory:\n",
    "\n",
    "    ```\n",
    "    cd ~/labsheets/Lab_4_Augment/\n",
    "    ```\n",
    "    \n",
    "4. Make all ```go_interactive.sh``` and ```tensorboard_params.sh``` files executables by using the command `chmod`: \n",
    "\n",
    "    ```\n",
    "    chmod +x go_interactive.sh tensorboard_params.sh\n",
    "    ```\n",
    "   \n",
    "5. Switch to interactive mode, and note the change of the gpu login to a reserved gpu:\n",
    "\n",
    "    ```\n",
    "    ./go_interactive.sh \n",
    "    ```\n",
    "    \n",
    "6. Run the following script. It will pop up two values: `ipnport=XXXXX` and `ipnip=XX.XXX.X.X.`\n",
    "\n",
    "    ```\n",
    "    ./tensorboard_params.sh\n",
    "    ```\n",
    "    \n",
    "    **Write them down since we will use them for using TensorBoard.**\n",
    "\n",
    "7. Train the model using the command: [your new modified file - see below]\n",
    "    \n",
    "    ```\n",
    "    python cifar_augment.py\n",
    "    ```\n",
    "   \n",
    "8. Open a **new terminal window** and login using SSH like in step 1 then run:\n",
    "\n",
    "    ```\n",
    "    tensorboard --logdir=logs/ --port=<ipnport>\n",
    "    ```\n",
    "    \n",
    "9. Open a **new terminal window** on your machine and type: \n",
    "    \n",
    "    ```\n",
    "    ssh -N <USER_NAME>@bc4login.acrc.bris.ac.uk -L 6006:<ipnip>:<ipnport>\n",
    "    ```\n",
    "\n",
    "10. Open your web browser (Use Chrome; Firefox currently has issues with tensorboard) and open the port 6006 (http://localhost:6006). This should open TensorBoard, and you can navigate through the summaries that we included.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. From tf.nn to tf.layers\n",
    "\n",
    "**NOW** copy your code from Lab 3, and rename it as `cifar_augment.py`. \n",
    "\n",
    "Until now,  you have fully specified your network in details, using ops defined in the [`tf.nn`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/nn) module to build the network. Today we'll modify the code in a way using ops from the [`tf.layers`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/layers) module which is a higher level interface making it easier to try new architectures. We started with [`tf.nn`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/nn) to show you the nuts and bolts of neural networks so nothing was hidden. The [`tf.layers`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/layers) ops are defined in a very similar way to the layers you used in the previous lab, except that they are parameterised so you don't have to repeat the same code over and over again.\n",
    "\n",
    "We will show you how your previous convolutional layer can now be re-written using tf.layers.\n",
    "\n",
    "*previously (in labs 1-3):*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def deepnn(x):\n",
    "    x_image = tf.reshape(x, [-1, FLAGS.img_width, FLAGS.img_height, FLAGS.img_channels])\n",
    "    with tf.variable_scope('Conv_1'):\n",
    "        W_conv1 = weight_variable([5, 5, FLAGS.img_channels, 32])\n",
    "        b_conv1 = bias_variable([32])\n",
    "        h_conv1 = tf.nn.relu(tf.nn.conv2d(x_image, W_conv1, strides=[1, 1, 1, 1], padding='SAME', name='convolution') + b_conv1)\n",
    "\n",
    "        # Pooling layer - downsamples by 2X.\n",
    "        h_pool1 = tf.nn.max_pool(h_conv1, ksize=[1, 2, 2, 1],\n",
    "                          strides=[1, 2, 2, 1], padding='SAME', name='pooling')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "followed by your Lab 3 code on batch normalisation. We will replace all this with the following\n",
    "\n",
    "*after*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xavier_initializer = tf.contrib.layers.xavier_initializer(uniform=True)\n",
    "def deepnn(x):\n",
    "    x_image = tf.reshape(x, [-1, FLAGS.img_width, FLAGS.img_height, FLAGS.img_channels])\n",
    "    conv1 = tf.layers.conv2d(\n",
    "        inputs=x_image,\n",
    "        filters=32,\n",
    "        kernel_size=[5, 5],\n",
    "        padding='same',\n",
    "        use_bias=False,\n",
    "        kernel_initializer=xavier_initializer,\n",
    "        name='conv1'\n",
    "    )\n",
    "    conv1_bn = tf.nn.relu(tf.layers.batch_normalization(conv1))\n",
    "    pool1 = tf.layers.max_pooling2d(\n",
    "        inputs=conv1_bn,\n",
    "        pool_size=[2, 2],\n",
    "        strides=2,\n",
    "        name='pool1'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOW** Change your full architecture to use [`tf.layers`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/layers)\n",
    "\n",
    "The architecture of the network stays the same from the last lab: two convolutional layers followed by two fully connected layers with batch normalisation on the convolutional layers.\n",
    "\n",
    "You can use [`tf.layers.dense`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/layers/dense) for your fully connected layers\n",
    "\n",
    "Debug and test. Your performance should not change."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Flushing summaries periodically\n",
    "\n",
    "Train and test summaries are flushed every 120 seconds by default. Decrease this so you don't have to wait until the network finishes training to inspect the summaries in tensorboard.\n",
    "\n",
    "**NOW** Set the `flush_secs` kwarg to a reasonable value when constructing the [`tf.summary.FileWriter`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/summary/FileWriter) objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_writer = tf.summary.FileWriter(run_log_dir + \"_train\", sess.graph, flush_secs=5)\n",
    "summary_writer_validation = tf.summary.FileWriter(run_log_dir + \"_validate\", sess.graph, flush_secs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both improvements will prove useful for your project coursework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3  Data Augmentation\n",
    "\n",
    "Generally the more data a CNN (or any deep learning model) has access to the better features it learns, and therefore the better it performs. Data augmentation refers to techniques to artificially increase the amount of training data with label preserving transformations on the original training data, i.e. we want to transform CIFAR-10 images in such a way that the data becomes more varied but the object in the image remains the same.\n",
    "\n",
    "It is typical to implement data augmentation *online* where each training mini-batch is loaded and mutated stocastically in some way (e.g. rotation, translation, blurring). Recall that we don't stop training after having processed the full dataset but after some other stopping criteria and so it is possible, even probable, that we will process each training example more than once. If we mutate the inputs stochastically then each time an example is part of a mini-batch it will be mutated in some different way (e.g. rotating by an angle sampled from a random distribution)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practical 3.1: Data Augmentation with Random Flips\n",
    "\n",
    "One example of data augmentation which can improve results on the CIFAR-10 dataset is horizontal flips. By randomly flipping the image you are able to add additional training data, without making the object in the image unrecognisable. \n",
    "\n",
    "* **Q. Think of problems that are invariant to horizontal flips... then think of problems that are not invariant (i.e. a horizontal flip will change the correct label of your sample).**\n",
    "\n",
    "Implement data augmentation into your network using online data augmentation and retrain to see the improvement in your results. Be careful that you **don't** apply the data augmentation during **testing**.\n",
    "\n",
    "* **Hint**: For types of data augmentation implemented in tensorflow have a look at the [`tf.image`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/image) module.\n",
    "* **Hint**: Because the stochastic [`tf.image`](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/image) ops are defined for 3D tensors you'll need to map the op over the mini-batch using [`tf.map_fn`](https://www.tensorflow.org/api_docs/python/tf/map_fn) so it applied to each training example.\n",
    "* **Hint**: Inside your computational graph you'll need to do different things if you're training or testing, this is what [`tf.cond`](https://www.tensorflow.org/api_docs/python/tf/cond) will allow you to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practical 3.2: Your own data augmentation\n",
    "\n",
    "Think carefully about another form of data augmentation which you believe would improve the capability of your network to recognise images from the CIFAR-10 dataset. Some types of data augmentation may have a negative impact on your network. For instance, flipping the image vertically is not useful as the test data will not contain any upside down boats or cats etc.\n",
    "\n",
    "Implement your chosen data augmentation method to see if it does improve your results.\n",
    "\n",
    "**Hint:** If you're stuck for ideas revisit the practical slides or talk to the TAs.\n",
    "\n",
    "Train your model and save the relevant logs, as well as the code you've written"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Debugging Strategies\n",
    "\n",
    "In Labs 1 and 2 you have seen how to use tensorflow to view how the accuracy and loss change over time. This is one of the most valuable tools for gaining insight into what your network is doing and debugging it.\n",
    "\n",
    "One thing to bear in mind when debugging tensorflow is that you *can't* easily use a normal python `print` statement. This is because tensorflow works by first building a computational graph and then evaluating the graph. You cannot print a value until it has been evaluated in the graph. To add a print operation to a tensorflow graph you use `tf.Print`. For example:\n",
    "\n",
    "```python\n",
    "x = tf.get_variable('x', [10, 5])\n",
    "x = tf.Print(x, [x])\n",
    "\n",
    "sess = tf.InteractiveSession()\n",
    "sess.run(x)\n",
    "```\n",
    "\n",
    "However, it is generally better pratice and faster to use a debugger. We will go through how to use the tensorflow debugger in this next exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Practical 4.1: Using the tensorflow debugger\n",
    "\n",
    "Another useful tools is the [tensorflow debugger](https://www.tensorflow.org/programmers_guide/debugger).\n",
    "\n",
    "To use it you'll first need to access an interactive node on bluecrystal. To do this run the command:\n",
    "\n",
    "```bash\n",
    "./go_interactive.sh\n",
    "\n",
    "```\n",
    "\n",
    "Try running the debug example within tensorflow: \n",
    "\n",
    "```bash\n",
    "module add libs/tensorflow/1.2\n",
    "python -m tensorflow.python.debug.examples.debug_mnist\n",
    "```\n",
    "\n",
    "This code also trains a classifier for [MNIST dataset](http://yann.lecun.com/exdb/mnist/), you’ll notice that unlike the previous examples we have trained, the accuracy decreases after step 1 and does not increase. This\n",
    "is most likely due to a bad numeric value such as `inf` or `nan` being generated in the training graph. The way tfdbg\n",
    "works is to add filters for tensor values so we can find the problem. Since `inf` and `nan` are common problems this filter already exists.\n",
    "\n",
    "To debug the example with tfdbg run:\n",
    "\n",
    "```bash\n",
    "python −m tensorflow.python.debug.examples.debug_mnist −−debug\n",
    "```\n",
    "\n",
    "You will see the following screen:\n",
    "\n",
    "![tfdbg start](img/tfdbg_screenshot_run_start.png)\n",
    "\n",
    "\n",
    "You can use **PageUp** / **PageDown** / **Home** / **End** to navigate. If you lack those keys use **Fn + Up** / **Fn + Down** / **Fn + Right** / **Fn + Left**\n",
    "\n",
    "Run `help` to list the available commands, or alternatively you can refer to the [`tfdbg` cheatsheet](https://www.tensorflow.org/programmers_guide/debugger#tfdbg_cli_frequently-used_commands)\n",
    "\n",
    "```bash\n",
    "tfdbg> help\n",
    "```\n",
    "\n",
    "Run the filter `has_inf_or_nan` to determine which tensors are contain either `Inf` or `NaN` values:\n",
    "\n",
    "```bash\n",
    "tfdbg> run -f has_inf_or_nan\n",
    "```\n",
    "\n",
    "You should now see this screen:\n",
    "\n",
    "![has_inf_or_nan](img/tfdbg_screenshot_run_end_inf_nan.png)\n",
    "\n",
    "The tensors that match the filter are displayed in chronological order, the tensor at the top `cross_entropy/Log:0` is the one in which `NaN` or `Inf` first appeared, so this is a good place to start our debugging.\n",
    "\n",
    "To view the value of a tensor click on the underlined tensor name e.g. `cross_entropy/Log:0` or enter the equivalent command:\n",
    "\n",
    "```bash\n",
    "tfdbg> pt cross_entropy/Log:0\n",
    "```\n",
    "\n",
    "To perform a regex search of tensor values run:\n",
    "\n",
    "```bash\n",
    "tfdbg> /(inf|nan)\n",
    "```\n",
    "\n",
    "OK, so there are `-Inf` values present in the tensor, how do we determine where they originate from? Well let's determine how this tensor was constructed. Use `node_info --traceback cross_entropy/Log` to determine what op output this tensor.\n",
    "\n",
    "* **Q:  Which line in the stack trace corresponds to user's code that defines the cross entropy?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing Lab 4 Portfolio\n",
    "\n",
    "You should by now have the following files, which you can zip under the name `Lab_4_<username>.zip`\n",
    "\n",
    "Note that we are asking you **this lab** to submit a copy of your modified code with data augmentation\n",
    "    \n",
    "From your logs, include only the TensorBoard summaries and remove the checkpoints (model.ckpt-* files)\n",
    "\n",
    "```\n",
    " Lab_4_<username>.zip\n",
    " |----------cifar_augment.py\n",
    " |----------logs\\ \n",
    "```\n",
    "\n",
    "Store this zip safely. You will be asked to upload all your labs' portfolio to SAFE after Week 10 - check SAFE for deadline details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Resources\n",
    "\n",
    "* [`tfdbg` tutorial](https://www.tensorflow.org/programmers_guide/debugger)\n",
    "* [Using `tfdbg` for batch run scripts](https://www.tensorflow.org/programmers_guide/debugger#offline_debugging_of_remotely-running_sessions)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}