Deep Learning with PyTorch/DL-with-pytorch - 4 [CNN].ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 4 [CNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Convolutional Neural Networks\n","- CIFAR-10 image classification with CNN\n","  - Last time, we have attempted to classify images in the CIFAR-10 dataset with MLP and deeper MLP. The final results were accuracy scores of 0.369 and 0.504, respectively.\n","  - Let's try to improve the classification performance by implementing simple CNN"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"0d78cd9a-4aa1-4b43-e240-f16e706115c7","executionInfo":{"status":"ok","timestamp":1545889909282,"user_tz":420,"elapsed":64924,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K    100% |████████████████████████████████| 591.8MB 29kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x618be000 @  0x7f36e05d62a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K    100% |████████████████████████████████| 61kB 18.0MB/s \n","\u001b[?25hCollecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K    100% |████████████████████████████████| 2.0MB 3.8MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n","  Found existing installation: Pillow 4.0.0\n","    Uninstalling Pillow-4.0.0:\n","      Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"59fd3412-a818-47e9-b910-d23da6d76900","executionInfo":{"status":"ok","timestamp":1545889919108,"user_tz":420,"elapsed":1028,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":2}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- CIFAR10 dataset can be downloaded by ```torchvision```\n","  - [torchvision.datasets](https://pytorch.org/docs/stable/torchvision/datasets.html)"]},{"metadata":{"id":"W5anlYa01w3w","colab_type":"code","outputId":"e48610fb-2c14-46a5-c1e5-1053e125fbe9","executionInfo":{"status":"ok","timestamp":1545889971425,"user_tz":420,"elapsed":50510,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from torchvision import datasets\n","import torchvision.transforms as transforms\n","\n","train_dataset = datasets.CIFAR10(root = \"/\", train = True, download = True, transform = transforms.ToTensor())\n","test_dataset = datasets.CIFAR10(root = \"/\", train = False, download = True, transform = transforms.ToTensor())"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /cifar-10-python.tar.gz\n","Files already downloaded and verified\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating CNN model and training\n","\n","- Create and train simple CNN model\n","\n","![](https://www.researchgate.net/profile/Qianzhou_Du2/publication/322477802/figure/fig3/AS:582461356511232@1515881017676/Illustration-of-Convolutional-Neural-Network-CNN-Architecture.png)"]},{"metadata":{"id":"YEoNQr10kGUW","colab_type":"code","colab":{}},"cell_type":"code","source":["# create data loaders \n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create CNN with one convolution/pooling layer\n","class net(nn.Module):\n","  def __init__(self, input_dim, num_filters, kernel_size, stride, padding, num_classes):\n","    super(net, self).__init__()\n","    self.input_dim = input_dim\n","    conv_output_size = int((input_dim - kernel_size + 2 * padding)/stride) + 1   # conv layer output size\n","    pool_output_size = int((conv_output_size - kernel_size)/stride) + 1          # pooling layer output size\n","    \n","    self.conv = nn.Conv2d(3, num_filters, kernel_size = kernel_size, stride = stride, padding = padding)     \n","    self.pool = nn.MaxPool2d(kernel_size = kernel_size, stride = stride)\n","    self.relu = nn.ReLU()\n","    self.dense = nn.Linear(pool_output_size * pool_output_size * num_filters, num_classes)     \n","    \n","  def forward(self, x):\n","    x = self.conv(x)\n","    x = self.relu(x)\n","    x = self.pool(x)\n","    x = x.view(x.size(0), -1)   # resize to fit into final dense layer\n","    x = self.dense(x)\n","    return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = 32\n","NUM_FILTERS = 32\n","KERNEL_SIZE = 3\n","STRIDE = 1\n","PADDING = 1\n","NUM_CLASSES = 10\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 30              "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_FILTERS, KERNEL_SIZE, STRIDE, PADDING, NUM_CLASSES).to(DEVICE)\n","criterion = nn.CrossEntropyLoss()   # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"3a8c705c-c46e-443d-82ba-af9fa858f9ec","executionInfo":{"status":"ok","timestamp":1545890996474,"user_tz":420,"elapsed":295268,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":545}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n","  temp_loss = []\n","  for (x, y) in train_loader:\n","    x, y = x.float().to(DEVICE), y.to(DEVICE)\n","    outputs = model(x)\n","    loss = criterion(outputs, y)\n","    temp_loss.append(loss.item())\n","    \n","    optimizer.zero_grad()\n","    loss.backward()\n","    optimizer.step()\n","    \n","  print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":19,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 1.5467755730499697\n","Loss at 1th epoch: 1.1874774668527686\n","Loss at 2th epoch: 1.0879874247724137\n","Loss at 3th epoch: 1.0216767588234923\n","Loss at 4th epoch: 0.9694403696547994\n","Loss at 5th epoch: 0.9249361718402189\n","Loss at 6th epoch: 0.8892614440539913\n","Loss at 7th epoch: 0.8543041116746185\n","Loss at 8th epoch: 0.8303252603391857\n","Loss at 9th epoch: 0.8046949812213479\n","Loss at 10th epoch: 0.7781740239514109\n","Loss at 11th epoch: 0.7531528044539644\n","Loss at 12th epoch: 0.7326688110218633\n","Loss at 13th epoch: 0.7106734506614373\n","Loss at 14th epoch: 0.680794730210853\n","Loss at 15th epoch: 0.6707794649521713\n","Loss at 16th epoch: 0.6504438420390839\n","Loss at 17th epoch: 0.6231620132618243\n","Loss at 18th epoch: 0.6014589527836236\n","Loss at 19th epoch: 0.5822262440038763\n","Loss at 20th epoch: 0.5729258664886056\n","Loss at 21th epoch: 0.5556111552983599\n","Loss at 22th epoch: 0.5367484169695383\n","Loss at 23th epoch: 0.5206924786653055\n","Loss at 24th epoch: 0.5027579848876085\n","Loss at 25th epoch: 0.4926028959738934\n","Loss at 26th epoch: 0.47675407138626896\n","Loss at 27th epoch: 0.4579864634424829\n","Loss at 28th epoch: 0.44510160489460393\n","Loss at 29th epoch: 0.4307325435873797\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained CNN model with accuracy score \n","  - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"d1db27e2-f9ca-4642-971e-98ab60a352f3","executionInfo":{"status":"ok","timestamp":1545891014673,"user_tz":420,"elapsed":3350,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n","  for x, y in test_loader:\n","    x, y = x.float().to(DEVICE), y.to(DEVICE)\n","    outputs = F.softmax(model(x)).max(1)[-1]       # predicted label\n","    y_true += list(y.cpu().numpy())                # true label\n","    y_pred += list(outputs.cpu().numpy())   "],"execution_count":20,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n","  \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"8c2be308-ea59-46d7-879f-dee1bde8f264","executionInfo":{"status":"ok","timestamp":1545891017606,"user_tz":420,"elapsed":1795,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":21,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.6454"]},"metadata":{"tags":[]},"execution_count":21}]}]}