2_intermediate/5.1-Kaggle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Credits\n",
    "\n",
    "See main readme for credits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import glob\n",
    "import os\n",
    "\n",
    "from skimage.io import imread\n",
    "from skimage.transform import resize\n",
    "\n",
    "import data_utils\n",
    "\n",
    "#import sys\n",
    "#sys.path.append(os.path.join('.', '..')) # Allow us to import shared custom \n",
    "#                                         # libraries, like utils.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tying everything together\n",
    "\n",
    "Now that you have learned about the three most used network architectures: FFNs, CNNs and RNNs. It is time to combine these network types into a more advanced model. \n",
    "\n",
    "It often happens that you have a combination of data that cannot fully be modeled by one of these three types of network. Knowing how to divide the data into the right subsets, and then build a network that handles each subset efficiently can mean the difference between a great model and an unusable one. \n",
    "\n",
    "## Kaggle challenge\n",
    "\n",
    "In this lab we will work on a data science challenge from [`kaggle.com`](kaggle.com).\n",
    "Kaggle is a website to participate in real life challenges. Early 2017 it was bought by Google, who wanted access to the global community of data scientists it has created over the last 7 years. Since then Google have sponsored its expansion and now the prizes of the competitions and the amount of public datasets are bigger than ever. Most competitions on Kaggle have a dataset, an accuracy metric and a leaderboard to compare submissions.\n",
    "You can read more about Kaggle [here](https://www.kaggle.com/about) and access a great amount of public datasets [here](https://www.kaggle.com/datasets).\n",
    "\n",
    "**NB**: You will need a Kaggle account for this exercise!\n",
    "\n",
    "The challenge we will pursue is the [_Leaf Classification_](https://www.kaggle.com/c/leaf-classification) challenge.\n",
    "The dataset consists approximately 1,584 images of leaf specimens (16 samples each of 99 species) which have been converted to binary black leaves against white backgrounds. Three sets of features are also provided per image: a shape contiguous descriptor, an interior texture histogram, and a ﬁne-scale margin histogram. For each feature, a 64-attribute vector is given per leaf sample.\n",
    "\n",
    "We will primarily look into the type of neural network best suited for handling this type of data. \n",
    " * For images, usually the convolutional neural network does a pretty good job, \n",
    " * For timeseries (like the shape) usually the RNN is the network of choice\n",
    " * For the describing features a FFN is often a great option\n",
    "\n",
    "Lastly, we will train the model and put the outputs in a submission file that we can submit to kaggle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get set up\n",
    "\n",
    "Go to kaggle, create a user, download the dataset.\n",
    "Unpack the dataset in the current directory.\n",
    "You should now be good to go."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the data\n",
    "\n",
    "First we start out by looking at the images. You need to download them first!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_paths = glob.glob(\"images/*.jpg\")\n",
    "print(\"Amount of images =\", len(image_paths))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we load in the training data, which is in CSV format. For this, we use [pandas](https://pandas.pydata.org/). Pandas is a horrible tool, but useful for data analysis. Please refrain from using pandas in any production code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now loading the train.csv to find features for each training point\n",
    "train = pd.read_csv('train.csv')\n",
    "train_images = ['images/{}.jpg'.format(i) for i in train.id.values]\n",
    "# notice how we \"only\" have 990 (989+0 elem) images for training, the rest is for testing\n",
    "train.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our training data and images loaded into memory. It is time to take a look at the data. Trying to classify leaves does not sound like a particularly difficult or interesting problem. We have probably all had teachers forcing us to do it on field trips as children.\n",
    "\n",
    "But try to take a look at the 99 different categories and come up with a system that discern all 99 types of leaves from each other. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# First we find an example of each species in order to visualize it\n",
    "species = np.array(sorted(train.species.unique()))\n",
    "species_examples = [np.random.choice(train[train.species == s].id.values) for s in species]\n",
    "\n",
    "# Then we gather its' index in our list of images in order to find the correct image\n",
    "indexes = [image_paths.index('images/{}.jpg'.format(i)) for i in species_examples]\n",
    "\n",
    "# now plot 1 image from each category\n",
    "fig = plt.figure(figsize=(50, 50))\n",
    "for i, idx in enumerate(indexes):\n",
    "    plt.subplot(10, 10, i + 1)\n",
    "    image = imread(image_paths[idx], as_grey=True)\n",
    "    plt.imshow(image, cmap='gray')\n",
    "    plt.title(\"%s\" % (species[i]))\n",
    "    plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, classifying leaves is actually a very tough problem. What makes it even worse, is that we cannot use all the image data we have available. In order to decrease the amount of computations needed, we need to reduce the size of the images as much as possible. On top of that our neural network only accepts fixed size input tensors. This means we will have to resize the images to ensure they all have the same sizes.\n",
    "\n",
    "This resizing is problematic because it alters the shape of the leaves, and for some of them, this is their most distinctive feature. Take a look at `Salix_Intergra` in the bottom left corner. Describing this leaf without taking its' shape into account seems extremely difficult."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(18, 6))\n",
    "amount = 5\n",
    "image_sample = np.random.choice(train_images, amount)\n",
    "\n",
    "ax = plt.subplot(2, amount + 1, 1)\n",
    "txt = ax.text(0.4, 0.5, 'Original', fontsize=20)\n",
    "txt.set_clip_on(False)\n",
    "plt.axis('off')\n",
    "    \n",
    "for i, path in enumerate(image_sample):\n",
    "    plt.subplot(2, amount + 1, i + 2)\n",
    "    image = imread(path, as_grey=True)\n",
    "    plt.imshow(image, cmap='gray')\n",
    "    _id = int(path.split('/')[-1].split('.')[0])\n",
    "    plt.title(\"{0}\\nshape: {1}\".format(train[train.id == _id].species.values[0], image.shape))\n",
    "    plt.axis('off')\n",
    "\n",
    "ax = plt.subplot(2, amount + 1, len(image_sample) + 2)\n",
    "txt = ax.text(0.4, 0.5, 'Resized', fontsize=20)\n",
    "txt.set_clip_on(False)\n",
    "plt.axis('off')\n",
    "    \n",
    "for i, path in enumerate(image_sample):\n",
    "    i += len(image_sample) + 3\n",
    "    plt.subplot(2, amount + 1, i)\n",
    "    image = imread(path, as_grey=True)\n",
    "    \n",
    "    image = resize(image, output_shape=(300, 300)) # <-- This is the method that resizes the image\n",
    "    \n",
    "    plt.imshow(image, cmap='gray')\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now do similar as in train example above for test.csv\n",
    "test = pd.read_csv('test.csv')\n",
    "# notice that we do not have species here, we need to predict that ..!\n",
    "test.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and now do similar as in train example above for test.csv\n",
    "sample_submission = pd.read_csv('sample_submission.csv')\n",
    "# accordingly to these IDs we need to provide the probability of a given plant being present\n",
    "sample_submission.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Investigating the other features\n",
    "\n",
    "Now that we have looked at the image data we have available, it is time to take a look at the other available features. Below we choose a random subset of the training data, and visualize the 3 types of available features:\n",
    "* margin\n",
    "* shape\n",
    "* texture\n",
    "\n",
    "Try to run it a few times to try and get an understanding of how the features differ from species to species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# try and extract and plot columns\n",
    "X = train.as_matrix()\n",
    "species = X[:, 1:2]\n",
    "margin = X[:, 2:66]\n",
    "shape = X[:, 66:130]\n",
    "texture = X[:, 130:]\n",
    "\n",
    "# let us plot some of the features\n",
    "plt.figure(figsize=(21,7)) # Set the plot size\n",
    "amount = 5 # Choose the amount of images we want to show at a time\n",
    "for i, idx in enumerate(np.random.choice(range(len(train)), amount)):\n",
    "    ax = plt.subplot(amount,4,1+i*4)\n",
    "    txt = ax.text(0.2, 0.2, species[idx][0], fontsize=20)\n",
    "    txt.set_clip_on(False)\n",
    "    plt.axis('off')\n",
    "    if i == 0:\n",
    "        plt.title('Species', fontsize=20)\n",
    "    plt.subplot(amount,4,2+i*4)\n",
    "    plt.plot(margin[idx])\n",
    "    if i == 0:\n",
    "        plt.title('Margin', fontsize=20)\n",
    "    plt.axis('off')\n",
    "    plt.subplot(amount,4,3+i*4)\n",
    "    plt.plot(shape[idx])\n",
    "    if i == 0:\n",
    "        plt.title('Shape', fontsize=20)\n",
    "    plt.axis('off')\n",
    "    plt.subplot(amount,4,4+i*4)\n",
    "    plt.plot(texture[idx])\n",
    "    if i == 0:\n",
    "        plt.title('Texture', fontsize=20)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1\n",
    "\n",
    "1. Test various resizings of the image until you have found the smallest resizing of the image where you can still see differentiate between the images. Use `IMAGE_SHAPE=(?, ?, 1)` to reflect your choices.\n",
    "\n",
    "So far we have learned about the feed forward neural network, the convolutional neural network and the recurrent neural network.\n",
    "Given margin and texture are histograms, shape is a contigious value over a \"time\" dimension \n",
    "\n",
    "* How could Margin, Shape and Texture be represented for classification?\n",
    "\n",
    "* Describe what network you would build and how you would represent the data points (image, margin, shape and texture)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# dims = [128, 64, 32]\n",
    "# _samples = np.random.choice(train_images, 10)\n",
    "# print(len(_samples))\n",
    "# for img in _samples:\n",
    "#     for dim in dims:\n",
    "#         print('{0}x{0}'.format(dim))\n",
    "\n",
    "#         plt.imshow(resize(imread(img, as_gray=True), output_shape=(dim, dim)), cmap='gray')\n",
    "#         plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Defining the data loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# loading data and setting up constants\n",
    "TRAIN_PATH = \"train.csv\"\n",
    "TEST_PATH = \"test.csv\"\n",
    "IMAGE_PATHS = glob.glob(\"images/*.jpg\")\n",
    "NUM_CLASSES = 99\n",
    "IMAGE_SHAPE = (64, 64, 1)\n",
    "NUM_FEATURES = 64 # for all three features, margin, shape and texture\n",
    "# train holds both X (input) and t (target/truth)\n",
    "data = data_utils.load_data(train_path=TRAIN_PATH, \n",
    "                            test_path=TEST_PATH,\n",
    "                            image_paths=IMAGE_PATHS,\n",
    "                            image_shape=IMAGE_SHAPE[:2])\n",
    "# to visualize the size of the dimensions of the data\n",
    "# print\n",
    "print(\"@@@Shape checking of data sets@@@\")\n",
    "# print\n",
    "print(\"TRAIN\")\n",
    "print(\"\\timages\\t%s%f\" % (data.train['images'].shape, data.train['images'].mean()))\n",
    "print(\"\\tmargins\\t%s\\t%f\" % (data.train['margins'].shape, data.train['margins'].mean()))\n",
    "print(\"\\tshapes\\t%s\\t%f\" % (data.train['shapes'].shape, data.train['shapes'].mean()))\n",
    "print(\"\\ttextures%s\\t%f\" % (data.train['textures'].shape, data.train['textures'].mean()))\n",
    "print(\"\\tts\\t %s\" % (data.train['ts'].shape))\n",
    "print(\"\\twhile training, batch_generator will onehot encode ts to (batch_size, num_classes)\")\n",
    "# print()\n",
    "print(\"TEST\")\n",
    "print(\"\\timages\\t%s\\t%f\" % (data.test['images'].shape, data.test['images'].mean())) \n",
    "print(\"\\tmargins\\t%s\\t%f\" % (data.test['margins'].shape, data.test['margins'].mean()))\n",
    "print(\"\\tshapes\\t%s\\t%f\" % (data.test['shapes'].shape, data.test['shapes'].mean()))\n",
    "print(\"\\ttextures%s\\t%f\" % (data.test['textures'].shape, data.test['textures'].mean()))\n",
    "print(\"\\tids\\t%s\" % (data.test['ids'].shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Generator\n",
    "\n",
    "While training, we will not directly access the entire dataset, instead we have a `batch_generator` function to give us inputs aligned with their targets/ids in a size that our model can handle in memory (batch\\_size).\n",
    "\n",
    "Furthermore, the `batch_generator` also handles validation splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "dummy_batch_gen = data_utils.batch_generator(data, batch_size=64, num_classes=99, num_iterations=5e3, seed=42)\n",
    "train_batch = next(dummy_batch_gen.gen_train())\n",
    "valid_batch, i = next(dummy_batch_gen.gen_valid())\n",
    "test_batch, i = next(dummy_batch_gen.gen_test())\n",
    "\n",
    "print(\"TRAIN\")\n",
    "print(\"\\timages,\", train_batch['images'].shape)\n",
    "print(\"\\tmargins,\", train_batch['margins'].shape)\n",
    "print(\"\\tshapes,\", train_batch['shapes'].shape)\n",
    "print(\"\\ttextures,\", train_batch['textures'].shape)\n",
    "print(\"\\tts,\", train_batch['ts'].shape)\n",
    "print()\n",
    "print(\"VALID\")\n",
    "print(\"\\timages,\", valid_batch['images'].shape)\n",
    "print(\"\\tmargins,\", valid_batch['margins'].shape)\n",
    "print(\"\\tshapes,\", valid_batch['shapes'].shape)\n",
    "print(\"\\ttextures,\", valid_batch['textures'].shape)\n",
    "print(\"\\tts,\", valid_batch['ts'].shape)\n",
    "print()\n",
    "print(\"TEST\")\n",
    "print(\"\\timages,\", test_batch['images'].shape)\n",
    "print(\"\\tmargins,\", test_batch['margins'].shape)\n",
    "print(\"\\tshapes,\", test_batch['shapes'].shape)\n",
    "print(\"\\ttextures,\", test_batch['textures'].shape)\n",
    "print(\"\\tids,\", len(test_batch['ids']))\n",
    "# notice that mean is very different, which is why we use batch_norm in all input data in model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.nn import Linear, GRU, Conv2d, Dropout, MaxPool2d, BatchNorm1d\n",
    "from torch.nn.functional import relu, elu, relu6, sigmoid, tanh, softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "use_cuda = torch.cuda.is_available()\n",
    "\n",
    "def get_variable(x):\n",
    "    \"\"\" Converts tensors to cuda, if available. \"\"\"\n",
    "    if use_cuda:\n",
    "        return x.cuda()\n",
    "    return x\n",
    "\n",
    "def get_numpy(x):\n",
    "    \"\"\" Get numpy array for both cuda and not. \"\"\"\n",
    "    if use_cuda:\n",
    "        return x.cpu().data.numpy()\n",
    "    return x.data.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "height, width, channels = IMAGE_SHAPE\n",
    "conv_out_channels = 16\n",
    "conv_stride = 1\n",
    "pool_stride = 2\n",
    "\n",
    "features_cat_size = 3 * NUM_FEATURES\n",
    "cnn_final_size = 144\n",
    "features_rnn_size = NUM_FEATURES+NUM_FEATURES+100\n",
    "\n",
    "class Net(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        # note 3x, because we concat the three feature inputs\n",
    "        self.batchnorm_features = BatchNorm1d(num_features=features_cat_size)\n",
    "        \n",
    "#         self.dropout = Dropout(p=0.2)\n",
    "#         # conv layer with (5, 5) kernel, stride of 1, and padding 2 (gives same dim output)\n",
    "#         self.conv_1 = Conv2d(in_channels=channels,\n",
    "#                              out_channels=conv_out_channels,\n",
    "#                              kernel_size=5,\n",
    "#                              stride=conv_stride,\n",
    "#                              padding=2)\n",
    "#         self.conv_2 = Conv2d(in_channels=conv_out_channels,\n",
    "#                              out_channels=conv_out_channels,\n",
    "#                              kernel_size=5,\n",
    "#                              stride=conv_stride,\n",
    "#                              padding=2)\n",
    "#         self.conv_3 = Conv2d(in_channels=conv_out_channels,\n",
    "#                              out_channels=conv_out_channels,\n",
    "#                              kernel_size=5,\n",
    "#                              stride=conv_stride,\n",
    "#                              padding=2)\n",
    "#         self.conv_4 = Conv2d(in_channels=conv_out_channels,\n",
    "#                              out_channels=conv_out_channels,\n",
    "#                              kernel_size=5,\n",
    "#                              stride=conv_stride,\n",
    "#                              padding=2)\n",
    "#         self.pool_1 = MaxPool2d(kernel_size=3,\n",
    "#                                 stride=pool_stride,\n",
    "#                                 padding=0)\n",
    "#         self.pool_2 = MaxPool2d(kernel_size=3,\n",
    "#                                 stride=pool_stride,\n",
    "#                                 padding=0)\n",
    "#         self.pool_3 = MaxPool2d(kernel_size=3,\n",
    "#                                 stride=pool_stride,\n",
    "#                                 padding=0)\n",
    "#         self.pool_4 = MaxPool2d(kernel_size=3,\n",
    "#                                 stride=pool_stride,\n",
    "#                                 padding=0)\n",
    "#         self.batchnorm_image_cnn = BatchNorm1d(num_features=cnn_final_size)\n",
    "        \n",
    "#         self.rnn_1 = GRU(input_size=1,\n",
    "#                          hidden_size=100,\n",
    "#                          num_layers=1,\n",
    "#                          batch_first=True)\n",
    "#         self.batchnorm_shape = BatchNorm1d(num_features=features_rnn_size)\n",
    "        \n",
    "        self.l_out = Linear(in_features=features_cat_size,\n",
    "                            out_features=NUM_CLASSES,\n",
    "                            bias=False)\n",
    "        \n",
    "    def forward(self, x_img, x_margin, x_shape, x_texture):\n",
    "        out = {}\n",
    "        ## use concatenated leaf features only\n",
    "        # concat the features\n",
    "        x = torch.cat((x_margin, x_shape, x_texture), dim=1)\n",
    "        features = self.batchnorm_features(x)\n",
    "        \n",
    "        ## image features only\n",
    "        # we permute the dimensions, such that we have NCHW (num, channels, heigh, width)\n",
    "#         x = x_img.permute(0, 3, 1, 2)\n",
    "#         x = relu(self.pool_1(self.conv_1(x)))\n",
    "#         x = relu(self.pool_2(self.conv_2(x)))\n",
    "#         x = relu(self.pool_3(self.conv_3(x)))\n",
    "#         x = self.dropout(x)\n",
    "#         x = relu(self.pool_4(self.conv_4(x)))\n",
    "#         features = x.view(BATCH_SIZE, -1)\n",
    "#         features = self.batchnorm_image_cnn(features)\n",
    "        \n",
    "        ## use concatenated leaf features only, where shape has been through an rnn\n",
    "        # use last hidden state of rnn\n",
    "        #x_shape = self.batchnorm_shape(x_shape)\n",
    "#         x_shape.unsqueeze_(-1)\n",
    "#         _, hs = self.rnn_1(x_shape)\n",
    "#         # note, hs gives (num_layers * num_directions, batch, hidden_size)\n",
    "#         # so, we need to move batch dim to first element\n",
    "#         hs = hs.permute(1, 0, 2)\n",
    "#         hs = hs.view(BATCH_SIZE, -1)\n",
    "#         x = torch.cat((x_margin, hs, x_texture), dim=1)\n",
    "#         features = self.batchnorm_shape(x)\n",
    "    \n",
    "        out['out'] = softmax(self.l_out(features), dim=1)\n",
    "        return out\n",
    "\n",
    "net = Net()\n",
    "if use_cuda:\n",
    "    net.cuda()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the cost function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 70\n",
    "LEARNING_RATE = 0.001\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "# weight_decay is equal to L2 regularization\n",
    "optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)\n",
    "\n",
    "def accuracy(ys, ts):\n",
    "    predictions = torch.max(ys, 1)[1]\n",
    "    correct_prediction = torch.eq(predictions, ts)\n",
    "    return torch.mean(correct_prediction.float())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_img_shape = tuple([BATCH_SIZE] + list(IMAGE_SHAPE))\n",
    "_feature_shape = (BATCH_SIZE, NUM_FEATURES)\n",
    "\n",
    "def randnorm(size):\n",
    "    return np.random.normal(0, 1, size).astype('float32')\n",
    "\n",
    "# dummy data\n",
    "_x_image = get_variable(Variable(torch.from_numpy(randnorm(_img_shape))))\n",
    "_x_margin = get_variable(Variable(torch.from_numpy(randnorm(_feature_shape))))\n",
    "_x_shape = get_variable(Variable(torch.from_numpy(randnorm(_feature_shape))))\n",
    "_x_texture = get_variable(Variable(torch.from_numpy(randnorm(_feature_shape))))\n",
    "\n",
    "# test the forward pass\n",
    "output = net(x_img=_x_image, x_margin=_x_margin, x_shape=_x_shape, x_texture=_x_texture)\n",
    "output['out']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "max_iter = 500\n",
    "log_every = 10\n",
    "VALIDATION_SIZE = 0.1 # 0.1 is ~ 100 samples for valition\n",
    "eval_every = 20\n",
    "\n",
    "def get_labels(batch):\n",
    "    return get_variable(Variable(torch.from_numpy(batch['ts']).long()))\n",
    "\n",
    "def get_input(batch):\n",
    "    return {\n",
    "        'x_img': get_variable(Variable(torch.from_numpy(batch['images']))),\n",
    "        'x_margin': get_variable(Variable(torch.from_numpy(batch['margins']))),\n",
    "        'x_shape': get_variable(Variable(torch.from_numpy(batch['shapes']))),\n",
    "        'x_texture': get_variable(Variable(torch.from_numpy(batch['textures'])))\n",
    "    }\n",
    "\n",
    "train_loss, train_accs = [], []\n",
    "batch_gen = data_utils.batch_generator(data,\n",
    "                                       batch_size=BATCH_SIZE,\n",
    "                                       num_classes=NUM_CLASSES,\n",
    "                                       num_iterations=max_iter,\n",
    "                                       seed=42,\n",
    "                                       val_size=VALIDATION_SIZE)\n",
    "\n",
    "net.train()\n",
    "for i, batch_train in enumerate(batch_gen.gen_train()):\n",
    "    if i % eval_every == 0:\n",
    "        net.eval()\n",
    "        val_losses, val_accs, val_lengths = 0, 0, 0\n",
    "        for batch_valid, num in batch_gen.gen_valid():\n",
    "            output = net(**get_input(batch_valid))\n",
    "            labels_argmax = torch.max(get_labels(batch_valid), 1)[1]\n",
    "            val_losses += criterion(output['out'], labels_argmax) * num\n",
    "            val_accs += accuracy(output['out'], labels_argmax) * num\n",
    "            val_lengths += num\n",
    "        # divide by the total accumulated batch sizes\n",
    "        val_losses /= val_lengths\n",
    "        val_accs /= val_lengths\n",
    "        print(\"### EVAL loss: {:.2f} accs: {:.2f}\".format(get_numpy(val_losses)[0],\n",
    "                                                          get_numpy(val_accs)[0]))\n",
    "        net.train()\n",
    "    \n",
    "    output = net(**get_input(batch_train))\n",
    "    labels_argmax = torch.max(get_labels(batch_train), 1)[1]\n",
    "    batch_loss = criterion(output['out'], labels_argmax)\n",
    "    \n",
    "    train_loss.append(get_numpy(batch_loss))\n",
    "    train_accs.append(get_numpy(accuracy(output['out'], labels_argmax)))\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    batch_loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if i % log_every == 0:        \n",
    "        print(\"train, it: {} loss: {:.2f} accs: {:.2f}\".format(i, \n",
    "                                                               np.mean(train_loss), \n",
    "                                                               np.mean(train_accs)))\n",
    "        # reset\n",
    "        train_loss, train_accs = [], []\n",
    "        \n",
    "    if max_iter < i:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submission to Kaggle\n",
    "\n",
    "First we have to make testset predictions, then we have to place it in the submission file and the upload to kaggle for our score! You can upload at max 5 submissions a day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET PREDICTIONS\n",
    "# containers to collect ids and predictions\n",
    "ids_test, preds_test = [], []\n",
    "net.eval()\n",
    "# run like with validation\n",
    "for batch_test, num in batch_gen.gen_test():\n",
    "    output = net(**get_input(batch_test))\n",
    "    y_out = output['out'].data\n",
    "\n",
    "    ids_test += batch_test['ids']\n",
    "    if num!=len(y_out):\n",
    "        # in case of the last batch, num will be less than batch_size\n",
    "        y_out = y_out[:num]\n",
    "    preds_test.append(y_out)\n",
    "preds_test = np.concatenate(preds_test, axis=0)\n",
    "assert len(ids_test) == len(preds_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make submission file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "preds_df = pd.DataFrame(preds_test, columns=data.le.classes_)\n",
    "ids_test_df = pd.DataFrame(ids_test, columns=[\"id\"])\n",
    "submission = pd.concat([ids_test_df, preds_df], axis=1)\n",
    "submission.to_csv('submission_mlp.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# below prints the submission, can be removed and replaced with code block below\n",
    "submission.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upload submission\n",
    "\n",
    "1. Go to [`https://www.kaggle.com/c/leaf-classification/`](https://www.kaggle.com/c/leaf-classification/)\n",
    "2. Make a submission\n",
    "3. Click or drop your submission here (writing a description is good practice)\n",
    "4. Submit\n",
    "\n",
    "Success! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2\n",
    "\n",
    "1. Try to get the best score on Kaggle for this dataset as you can. (You can upload your solution multiple times as you progress)\n",
    "\n",
    "A good implementation would get a score between $0.04$ to $0.06$ (the smaller the better), try and see if you can get there, and explain what might have gone wrong if you cant. \n",
    "\n",
    "When trying to improve the network nothing is sacred, you can change learning rate, try testing various learning rates, batch sizes, validation sizes, etc. And most importantly, the validation set is very small (only 1 sample per class), etc.\n",
    "\n",
    "To get you of to a good start we have created a list of thing you might want to try:\n",
    "* Include a second fully connected layer with dropout layer\n",
    "* Try with L1 regularization\n",
    "* Include L2 regularization (weigth decay)\n",
    "* Use only the image for training (with CNN) - comment on the increased time between iterations.\n",
    "* Use dropout between the convolutional layers\n",
    "* Include the RNN part\n",
    "* Increase or decrease the batch size \n",
    "* Change the image size to be rectangular, bigger or smaller\n",
    "* Try to combine the FFN, CNN, RNN parts in various ways\n",
    "\n",
    "If your network is not performing as well as you would like it to, [here](http://theorangeduck.com/page/neural-network-not-working) is a great explanation of what might have gone wrong.\n",
    "\n",
    "## Insights?\n",
    "\n",
    "What worked, and what didn't? Sometimes the fancy tricks might not do any good. Keep that in mind, and always start out simple."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}