Lab_2_CNNs/Lab sheet 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 2 - Convolutional Neural Networks and TensorFlow Graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this second lab session, you will build and train a shallow convolutaional neural network for recognising objects using the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html). CIFAR-10 has 10 object classes, 60,000 examples with 6,000 examples per class.\n",
    "\n",
    "[![CIFAR10 samples](cifar10-sample.png)](https://www.cs.toronto.edu/~kriz/cifar.html)\n",
    "\n",
    "### Objectives:\n",
    "\n",
    "1. Build your first convolutional model using TensorFlow 1.2 for recognising objects using the CIFAR-10 dataset. \n",
    "\n",
    "2. Train your model on BC4 and visualize the training process\n",
    "\n",
    "3. Evaluate your model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. TensorFlow - Graphs and Sessions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 TensorFlow-1.2\n",
    "\n",
    "TensorFlow was originally developed by the Google Brain team as an internal machine learning tool and was open-sourced under the Apache 2.0 License in November 2015. Since then, it has become a popular choice among researchers and companies due to its balanced trade-off between flexibility (required in research) and production-worthiness (required in industry). Additionally, it's well documented and maintained, backed by a large community (> 10,000 commits and > 3000 TF-related repos in one year). \n",
    "\n",
    "### Graphs and sessions\n",
    "\n",
    "TensorFlow does all its computation in graphs (creators referred to them as **dataflow graphs**), [Danijar Hafner's website](https://danijar.com/what-is-a-tensorflow-session/) and [TensorFlow's documentation](https://www.tensorflow.org/programmers_guide/graphs) contains more details about the concept of computational graphs and their advantages, in this Lab we will focus only on how to build and excecute them.\n",
    "\n",
    "The **graph** will define the variables and computation. It doesn’t compute anything nor does it hold any values, it just defines the operations that we want to be performed.\n",
    "\n",
    "The excecution of the graph, referred as a **session**, allocates resources, feeds the data, computes the operations and holds the values of intermediate results and variables. The figure below shows how the data flows from the *input readers* through the *opertations* such as convolutions and activations, then the *gradients* are computed and error is *backpropagated* to the *weight* and *bias variables*. \n",
    "\n",
    "![](https://www.tensorflow.org/images/tensors_flowing.gif)\n",
    "\n",
    "\n",
    "**We encourage to use single graphs and single session** for your models in this course. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Downloading Relevant Files\n",
    "\n",
    "* First, visit the [GitHub labsheet repository](https://github.com/COMSM0018-Applied-Deep-Learning/labsheets)\n",
    "* Clone the repository `git clone \"https://github.com/COMSM0018-Applied-Deep-Learning/labsheets.git\"`\n",
    "* Copy both `CIFAR10`, `Lab_2_CNNs` into a new folder which we will refer to as `/path_to_files/`\n",
    "* Using Jupyter notebook, open the file `Lab_2_CNNs/simple_train_cifar.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Building your first Model\n",
    "\n",
    "Next, we will describe the code in the  ```simple_train_cifar.py``` file for implementing a CNN for recognising objects on the [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. Your first CNN will be formed of two *convolutional layers* and two *fully connected layers* with the following sizes and hyperparameters: \n",
    "\n",
    "* Filter 5 x 5, with stride 1 and padding [`'SAME'`](https://stackoverflow.com/questions/37674306/what-is-the-difference-between-same-and-valid-padding-in-tf-nn-max-pool-of-t) for convolutions\n",
    "* Kernel 2 x 2 , with stride 2 and padding [`'SAME'`](https://stackoverflow.com/questions/37674306/what-is-the-difference-between-same-and-valid-padding-in-tf-nn-max-pool-of-t) for max pooling \n",
    "* 1024 Neurons for the fully connected layer\n",
    "* ReLU activation functions for both convolutional layers, and fully connected layers\n",
    "* Weight and bias initialization using random values from a truncated normal distribution with $\\sigma= 0.1$  and $\\mu=0$ \n",
    "\n",
    "\n",
    "### Imports\n",
    "\n",
    "Here you can see the provided code stub with all relevant imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# %load -r 12:35 simple_train_cifar.py\n",
    "\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import sys\n",
    "import os\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'CIFAR10'))\n",
    "import cifar10 as cf\n",
    "\n",
    "FLAGS = tf.app.flags.FLAGS\n",
    "\n",
    "tf.app.flags.DEFINE_string('data-dir', os.getcwd() + '/dataset/',\n",
    "                           'Directory where the dataset will be stored and checkpoint.')\n",
    "tf.app.flags.DEFINE_integer('max-steps', 1000,\n",
    "                            'Number of mini-batches to train on.')\n",
    "tf.app.flags.DEFINE_integer('log-frequency', 100,\n",
    "                            'Number of steps between logging results to the console and saving summaries')\n",
    "tf.app.flags.DEFINE_integer('save-model', 1000,\n",
    "                            'Number of steps between model saves')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters\n",
    "\n",
    "We also defined all variables required for training globally, including batch size, learning rate, as well as the dataset's information including input size (img_height, img_channels) and the number of classes.\n",
    "\n",
    "`train_dir` refers to the location of the training files in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# %load -r 36:48 simple_train_cifar.py\n",
    "# Optimisation hyperparameters\n",
    "tf.app.flags.DEFINE_integer('batch-size', 128, 'Number of examples per mini-batch')\n",
    "tf.app.flags.DEFINE_float('learning-rate', 1e-4, 'The learning rate for our training.')\n",
    "tf.app.flags.DEFINE_integer('img-width', 32, 'Image width')\n",
    "tf.app.flags.DEFINE_integer('img-height', 32, 'Image height')\n",
    "tf.app.flags.DEFINE_integer('img-channels', 3, 'Image channels')\n",
    "tf.app.flags.DEFINE_integer('num-classes', 10, 'Number of classes')\n",
    "tf.app.flags.DEFINE_string('train-dir',\n",
    "                           '{cwd}/logs/exp_bs_{bs}_lr_{lr}'.format(cwd=os.getcwd(),\n",
    "                                                                   bs=FLAGS.batch_size,\n",
    "                                                                   lr=FLAGS.learning_rate),\n",
    "                           'Directory where to write event logs and checkpoint.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weight and Bias Variables\n",
    "Next we also defined functions that will create weight and bias variables of given shapes, and the correct truncated normal distribution initialisation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def weight_variable(shape):\n",
    "    '''weight_variable generates a weight variable of a given shape.'''\n",
    "    initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "    return tf.Variable(initial, name='weights')\n",
    "\n",
    "\n",
    "def bias_variable(shape):\n",
    "    '''bias_variable generates a bias variable of a given shape.'''\n",
    "    initial = tf.constant(0.1, shape=shape)\n",
    "    return tf.Variable(initial, name='biases')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining Input Size\n",
    "\n",
    "Note in the file, the input size is defined as 32X32X3, reflecting 3 colour channels - R,G,B\n",
    "\n",
    "![](./images/cnn_ex_1.png)\n",
    "\n",
    "You can check this in the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, [None, FLAGS.img_width * FLAGS.img_height * FLAGS.img_channels])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional and Pooling Layers\n",
    "In your file, you will find the definition of the first convolutional filter shown here []\n",
    "\n",
    "![](./images/cnn_ex_2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "    with tf.variable_scope('Conv_1'):\n",
    "        W_conv1 = weight_variable([5, 5, FLAGS.img_channels, 32])\n",
    "        b_conv1 = bias_variable([32])\n",
    "        h_conv1 = tf.nn.relu(tf.nn.conv2d(x_image, W_conv1, strides=[1, 1, 1, 1], padding='SAME', \n",
    "                                          name='convolution') + b_conv1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Followed by the definition of a max-pooling layer:\n",
    "![](./images/cnn_ex_3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "        # Pooling layer - downsamples by 2X.\n",
    "        h_pool1 = tf.nn.max_pool(h_conv1, ksize=[1, 2, 2, 1],\n",
    "                          strides=[1, 2, 2, 1], padding='SAME', name='pooling')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the rest of your CNN (Your Code)\n",
    "\n",
    "**NOW** Introduce the next convolutional layer, and label it as `Conv_2` based on the following figure\n",
    "\n",
    "![](./images/cnn_ex_4.png)\n",
    "\n",
    "**THEN** follow it by the same max-pooling, again using 2x2 kernel as before\n",
    "\n",
    "![](./images/cnn_ex_5.png)\n",
    "\n",
    "**NOW** reshape the output of your max pooling to be one dimensional\n",
    "\n",
    "![](./images/cnn_ex_6.png)\n",
    "\n",
    "**NOW** remind yourself from the last lab how to introduce fully connected layers, with RELU as your activation function as follows\n",
    "\n",
    "![](./images/cnn_ex_7.png)\n",
    "\n",
    "**FINALLY** Add another fully connected layer of the same size and define the output accordingly. The number of classes in CIFAR-10 is defined in the flag: FLAGS.num_classes\n",
    "\n",
    "![](./images/cnn_ex_8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the above should be defined with the function `def deepnn(x):`\n",
    "\n",
    "We can now declare our main function, which uses a tensoflow session that initialises the CNN we defined, and allows us to check that all the conections and data feeding are in the right place through [TensorBoard](https://www.tensorflow.org/get_started/summaries_and_tensorboard)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimisation and Gradient Descent\n",
    "\n",
    "**Now** that we have a CNN we want to actually train it! Time to include a loss function. We're using the standard cross entropy loss between the logits and the labels from the ground truth. Replace the code *in main*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define your loss function - softmax_cross_entropy\n",
    "    cross_entropy = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope('x_entropy'):\n",
    "        cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOW** replace the code with the right optimiser and the test accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "    # Define your AdamOptimiser, using FLAGS.learning_rate to minimise the loss function\n",
    "    \n",
    "    \n",
    "    # calculate the prediction and the accuracy\n",
    "    correct_prediction = 0\n",
    "    accuracy = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find the main training loop inside of the main function inside the tensorflow session. We run the loop 10,000 times however we only bother printing out the training information every 100 steps. Note we do not have to write any derivatives explicitely due to TensorFlow's [automatic differentiation](http://www.columbia.edu/~ahd2125/post/2015/12/5/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for step in range(FLAGS.max_steps):\n",
    "    # Training: Backpropagation using train set\n",
    "    (trainImages, trainLabels) = cifar.getTrainBatch()\n",
    "    (testImages, testLabels) = cifar.getTestBatch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have defined everything your TF model needs to be trained, so we only need to include some code for visualising the training progess and saving checkpoints.\n",
    "\n",
    "**NOW** uncomment ALL lines in your code that start with **`##`**, making sure you correctly defined the `optimiser`.\n",
    "\n",
    "Study the code as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summaries and Tensorboard\n",
    "Tensorboard allows for the visualisation of training and testing statistics in addition to a graphical output of the CNN that was trained. To do this we can run tensorboard on Blue Crystal and, via the use of port forwarding, view the results on the lab machines. \n",
    "\n",
    "First, we need to indicate what we want to be save on the summaries, for now we will save some images that are feed in to model, the loss and accuracy for every batch. \n",
    "\n",
    "**Important** study the code to ensure you understand what summaries it writes to the tensorboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Saving checkpoints\n",
    "\n",
    "Lastly, we include a saver for saving checkpoints so you can use them as a backup of your training and later for evaluation of your model.\n",
    "\n",
    "Through the flags, we have set the the code to save the model every 1000 steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# %load -r 186:189 simple_train_cifar.py\n",
    "# Save the model checkpoint periodically.\n",
    "if step % FLAGS.save_model == 0 or (step + 1) == FLAGS.max_steps:\n",
    "    checkpoint_path = os.path.join(FLAGS.train_dir + '_train', 'model.ckpt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validating and evaluating results\n",
    "\n",
    "Finally, below we show how to use the CIFAR-10's *test set*, in order to see how well the model performs for classifing unseen examples. In this Lab sessions will use the test set primarly for hyperparameters selection. Bare in mind that for this task, usually a subsampling of the training set (commonly detoned as *validation set*) is used for this task. Using validation and test sets helps to identify cases of under and overfitting, as well as for benchmarking the performance among different algorithms. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "    # Testing\n",
    "\n",
    "    # resetting the internal batch indexes\n",
    "    cifar.reset()\n",
    "    evaluatedImages = 0\n",
    "    test_accuracy = 0\n",
    "    nRuns = 0\n",
    "\n",
    "    while evaluatedImages != cifar.nTestSamples:\n",
    "        # don't loop back when we reach the end of the test set\n",
    "        (testImages, testLabels) = cifar.getTestBatch(allowSmallerBatches=True)\n",
    "        test_accuracy_temp, _ = sess.run([accuracy, test_summary], feed_dict={x: testImages, y_: testLabels})\n",
    "\n",
    "        nRuns = nRuns + 1\n",
    "        test_accuracy = test_accuracy + test_accuracy_temp\n",
    "        evaluatedImages = evaluatedImages + testLabels.shape[0]\n",
    "\n",
    "    test_accuracy = test_accuracy / nRuns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Complete File\n",
    "\n",
    "We now have a complete file for defining a CNN, training, validation and checkpoints. We will next move onto training the model using Blue Crystal 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Blue Crystal Phase 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BlueCrystal Phase 4 (BC4) is the latest update on the University's High Performance Computing (HPC) machine. It includes 32 GPU-accelerated nodes, each of them with two NVIDIA Tesla P100 GPU accelerators and also a visualization node equipped with NVIDIA GRID GPUs; what matters to us are the Tesla P100 GPU accelerators that we will use for training your Deep Learing algorithms. \n",
    "\n",
    "Further information on BC4 and the support we have for it are available at: https://www.acrc.bris.ac.uk/acrc/phase4.htm\n",
    "\n",
    "**NOTE**: You may try to debug and run programs on your own machine, but sadly we are unable to offer assistance for installing and/or set-up of the dependencies on personal machines. \n",
    "\n",
    "\n",
    "There are two *modes* for using BC4: *Interactive* and *Batch*. We will use *Interactive* as much as possible during lab sessions, since it allows the immediate excution of your program and you can see outputs directy on the terminal window (great for debugging); while the *Batch* method queues your job and generates files related with the excecution of your file. You will use *Batch* as part of CW2, so we will revisit that later.\n",
    "\n",
    "## 2.1   Copying Lab-1 files between your machine and BC4\n",
    "\n",
    "You need to copy the provided folders `CIFAR10` and `Lab_1_intro` (which contains `simple_train_cifar.py`, `submit_job.sh`, `tensorboard_params.sh`, `go_interactive.sh`) to your account in BC4. For copying individual files from your machine to your home directory on BC4 use the next example with go_interactive.sh:\n",
    "\n",
    "For copying individual files from your machine to your home directory on BC4 use the next example with `go_interactive.sh`:\n",
    "\n",
    "```bash\n",
    "scp  /path_to_files/Lab_1_intro/go_interactive.sh <your_UoB_ID>@bc4login.acrc.bris.ac.uk:\n",
    "```\n",
    "\n",
    "or all files at once by using: \n",
    "\n",
    "```bash\n",
    "scp -r /path_to_files/*  <your_UoB_ID>@bc4login.acrc.bris.ac.uk:\n",
    "```\n",
    "\n",
    "For copying back files from BC4 to your machine use the  command ```scp``` from a terminal on your machine, you can copy individual files, as well as directories:\n",
    "\n",
    "```bash\n",
    "scp  <your_UoB_ID>@bc4login.acrc.bris.ac.uk:/path_on_bc4/foo.foo   /path_in_your_machine/\n",
    "```\n",
    " \n",
    " Alternatively, you may wish to use SSHFS to mount a directory on BC4 to a directory using:\n",
    " \n",
    "```bash\n",
    "mkdir -p ~/bc4 && sshfs <your_UOB_ID>@bc4login.acrc.bris.ac.uk:/dir_on_bc4/ ~/bc4\n",
    "```\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2  Logging in, running scripts  and managing your directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replicate the next steps for logging-in BC4. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### **Logging in** \n",
    "\n",
    "The connection to BC4 is done via SSH, thus open a **new** Terminal window and type: \n",
    "\n",
    "```bash\n",
    "ssh <your_UoB_ID>@bc4login.acrc.bris.ac.uk\n",
    "```\n",
    "\n",
    "You should see something like this in your home directory:\n",
    " \n",
    " ```\n",
    " CIFAR10\n",
    " |----------cifar10.py\n",
    " Lab_2_CNNs\n",
    " |----------simple_train_cifar.py \n",
    " |----------submit_job.sh \n",
    " |----------tensorboard_params.sh \n",
    " |----------go_interactive.sh```\n",
    " \n",
    "**NOTE: If you cannot see the file structure above, you have not copied the files correctly**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Training your first CNN.\n",
    "\n",
    "\n",
    "**It's finally here, the moment you've been waiting for!** \n",
    "\n",
    "Follow the next steps for running the training script:\n",
    "\n",
    "1. Using the blue crystal ssh login change to the lab 2 directory:\n",
    "\n",
    "    ```cd Lab_2_CNNs/```\n",
    "    \n",
    "2. make all .sh files executables by using the command `chmod`: \n",
    "\n",
    "   ```chmod +x go_interactive.sh submit_job.sh tensorboard_params.sh```\n",
    "   \n",
    "3. Switch to interactive mode, and note the change of the gpu login to a reserved gpu:\n",
    "\n",
    "    ```./go_interactive.sh ```\n",
    "    \n",
    "4. Run the following script. It will pop up two values: `ipnport=XXXXX` and `ipnip=XX.XXX.X.X.`\n",
    "\n",
    "    ```./tensorboard_params.sh```\n",
    "    \n",
    "    **Write them down since we will use them for using TensorBoard.**\n",
    "\n",
    "5. Train the model using the command:\n",
    "    \n",
    "    ```bash\n",
    "    python simple_train_cifar.py &\n",
    "    tensorboard --logdir=logs/ --port=<ipnport>\n",
    "    ```\n",
    "   \n",
    "   where `ipnport` comes from the previous step. It might take a minute or two before you start seeing the accuracy on the validation batch at every step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 Visualising and Monitoring Your Training\n",
    "\n",
    "\n",
    "1. Open a **new Terminal window** on your machine and type: \n",
    "    \n",
    "    ``` ssh  <USER_NAME>@bc4login.acrc.bris.ac.uk -L 6006:<ipnip>:<ipnport>```</mark> \n",
    "    \n",
    "    where `ipnip` and `ipnport` comes from step 2 in **2.3**.\n",
    "\n",
    "2. Open your web browser (Use Chrome; Firefox currently has issues with tensorboard) and open the port 6006 (http://localhost:6006). This should open TensorBoard, and you can navigate through the summaries that we included.\n",
    "\n",
    "3. Go to the **GRAPH** tab and navigate to your CNN, identifying the two convolutional layers `CONV_1`, `CONV_2`, and their hyperparameters as well as the two fully-connected layers `FC_1` and `FC_2`\n",
    "\n",
    "4. Go to the **SCALARS** tab \n",
    "\n",
    "5. Tick the box *Show data download links*\n",
    "\n",
    "6. Click on **Loss**\n",
    "\n",
    "7. Download the csv file `run_exp_bs_128_lr_0.0001_train,tag_Loss.csv`\n",
    "\n",
    "** Keep the csv file for your portfolio of Lab_1 **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using TensorBoard you can monitor the training process. In the following labs you will perform experiments by varying hyperparemeters such as learning rate, batch size, epochs, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.6 Saving your trained model\n",
    "\n",
    "You should copy your log files back from BC4, and save them for your first lab portfolio\n",
    "\n",
    "```bash\n",
    "scp -r <your_UoB_ID>@bc4login.acrc.bris.ac.uk:/Lab_2_CNNs/logs   /path_in_your_machine/\n",
    "\n",
    "```\n",
    "\n",
    "Both your directory `logs/` and your `csv` file should be submitted as part of your Lab_2 portfolio (see [**section 5**](#5.-Preparing-Lab_2-Portfolio))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Training your Second CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is now time to train your own modification to the CNN above.\n",
    "\n",
    "Choose one hyperparameter to change in your built CNN. You might add/remove layers, change layer sizes, ...\n",
    "\n",
    "** Discuss your choice with a TA **\n",
    "\n",
    "1. Duplicate and rename your file ```simple_train_cifar.py``` into ```second_train_cifar.py```\n",
    "2. Change the code to reflect **your chosen hyperparameter change**\n",
    "3. Copy ```second_train_cifar.py``` to BC4 as you've done previously\n",
    "4. Train the model using the following command. \n",
    "   ```bash\n",
    "   python second_train_cifar.py &\n",
    "   tensorboard --logdir=logs_second/ --port=<ipnport>\n",
    "   ```\n",
    "   **NOTE: the change in the logs directory**\n",
    "\n",
    "5. Load your tensorboard to read from the *new logs* directory\n",
    "6. Save *logs_second* folder and the new *csv* back to your machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 4. Closing all sessions\n",
    "\n",
    "Once the training has finished, **close all sessions** by typing `exit`. You need to do this twice for an **interactive session.** \n",
    "\n",
    "**Please make sure closing your session in order to release the gpu node.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Preparing Lab_2 Portfolio\n",
    "\n",
    "You should by now have the following files, which you can zip under the name `Lab_1_<username>.zip` \n",
    "\n",
    "**From your logs, include only the TensorBoard summaries and remove the checkpoints (model.ckpt-* files)**\n",
    "\n",
    " ```\n",
    " Lab_2_<username>.zip\n",
    " |----------logs\\ \n",
    "            |----------exp_bs_128_lr_0.0001_train\n",
    "                       |----------events.out.tfevents.xxxxxxxxxx.gpuxx.bc4.acrc.priv\n",
    "            |----------exp_bs_128_lr_0.0001_validate\n",
    "                       |----------events.out.tfevents.xxxxxxxxxx.gpuxx.bc4.acrc.priv\n",
    " |----------run_exp_bs_128_lr_0.0001_train,tag_Loss.csv\n",
    " |----------logs_second\\\n",
    "            |----------<your second training>_train\n",
    "                       |----------events.out.tfevents.xxxxxxxxxx.gpuxx.bc4.acrc.priv\n",
    "            |----------<your second training>_validate\n",
    "                       |----------events.out.tfevents.xxxxxxxxxx.gpuxx.bc4.acrc.priv\n",
    " |----------<your second training.csv>\n",
    " ```\n",
    " \n",
    " Store this zip safely. You will be asked to upload all your labs' portfolio to ** SAFE during Week 9 ** - check SAFE for deadline details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# NOTE: Using the batch method \n",
    "\n",
    "During the course it may occur that very few GPUs are available due maintenance, other people using them, unforseen events, etc. making it not possible to use the *Interactive Session* previously described. Should this happen you will have to use the *batch method*. To do this establish a connection to BC4 as described before. Open the file \"submit_job.sh\" using emacs, vim or your favourite CLI text editor, and **modify line #10** to include your email (Blue Crystal will send you notifications about the jobs you're submitting) and **modify line #14** for the filename you are running. Now, run the next command for submitting a job to the BC4 queueing system:\n",
    "\n",
    "```sbatch submit_job.sh```\n",
    "\n",
    "And you should see a generated files (`hostname_<job_number>.err` and `hostname_<job_number>.out`) that shows the outputs after running your python script.\n",
    "In the `hostname_<job_number>.out` you should see output that you normally see on the terminal window when using the interactive mode,  `hostname_<job_number>.err` shows the output that occurred during the runing of script that exited with a non-zero exit status.\n",
    "\n",
    "**NOW** try your batch mode to get used to it for upcoming labs, and primarily for your group project"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}