-
Notifications
You must be signed in to change notification settings - Fork 172
Walkthrough: AlexNet
Contents:
- About Minerva owl.net
- About ImageNet
- About AlexNet
- Training AlexNet using Minerva
- Multi-view classification using AlexNet
- Using AlexNet to extract feature
owl.net is a DNN training framework build on Minerva's python interface owl. The main purposes of this package are: 1) Providing a simple way for Minerva users to train deep neural network for computer vision problems. 2) Providing a prototype about how to build user applications utilizing the advantages of Minerva.
We borrow Caffe's well-defined network and solver configure file format but the execution is conducted in Minerva engine. It's a showcase of Minerva's flexibile interface (building Caffe's main functionality in several hundreds of lines) and computation efficiency (Multi-GPU training).
If you are not familiar with ImageNet Large Scale Visual Recognition Challenge, please see here. The classification task contains 1.28 million images belong to 1000 classes.
To make IO efficient, we recommend transfer the original image into LMDB after you download the dataset. We could use the tool provided by Caffe to do the convertion. After converting the images, we need to compute the mean value of each pixel among the dataset. When training, mean values are subtracted from the image to produce a zero-mean input. Mean_file for ILSVRC12 can be downloaded by the script provided by Caffe.
In ILSVRC2012, AlexNet was proposed. It's the winning model of ILSVRC2012 classification task and it achieved a large accuracy margin compared with the non-DNN methods. It contains 5 convolutional layers and 3 fully-connected layers. During training, some randomness is introduced in the data augmentation process and dropout layer. Those details are defined in the configure file provided by Caffe. Note that currently we don't support convolutions with more than one group, for one GPU released recently has enough RAM to hold the whole model.
We implemented the DNN training logic in trainer.py, user need to define the training parameters in solver file.
The information need to be defined through solver are:
- network configuration file
- snapshot saving directory
- max iteration
- testing interval
- test iteration
- snapshot saving interval
- learning rate tuning strategy
- momentum
- weight decay AlexNet usually need traverse the training set 70-90 times before converging and the learning rate should be tune smaller several times. The standard solver for AlexNet could be found here.
User could use following command to start training given Caffe's solver under the scripts folder
./net_trainer.py <solver_file> [--snapshot SNAPSHOT] [-n NUM_GPU] solver_file is the file name in Caffe's solver format. SNAPSHOT is the index of the snapshot to start with (default: 0). NUM_GPU is the number of gpu to use.
If we set NUM_GPU greater than 1, our code will slice a training batch into NUM_GPU pieces and FF/BP in parallel. The update is executed synchronously, so the training result using one gpu or NUM_GPU gpus will be the same.
The SNAPSHOT parameter will guide the system find the saved model under "snapshot saving directory" with that index. If the model can be found and the weight dimension is matched with the network configure file, OwlNet will load the model and continue training. Otherwise, it will initialize the weight according to the weight_filler parameter in configure file.
| Layer | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| Type | conv+max+norm | conv+max+norm | conv | conv | conv+max | full | full | full |
| Channels | 96 | 256 | 384 | 384 | 256 | 4096 | 4096 | 1000 |
| Filter Size | 11*11 | 5*5 | 3*3 | 3*3 | 3*3 | - | - | - |
| Convolution Stride | 4*4 | 1*1 | 1*1 | 1*1 | 1*1 | - | - | - |
| Pooling Size | 3*3 | 3*3 | - | - | 3*3 | - | - | - |
| Pooling Stride | 2*2 | 2*2 | - | - | 2*2 | - | - | - |
| Padding Size | 2*2 | 1*1 | 1*1 | 1*1 | 1*1 | - | - | - |
| Training speed (images/second) |
AlexNet | VGGNet | GoogLeNet |
|---|---|---|---|
| 1 card | 189.63 | 14.37 | 82.47 |
| 2 cards | 371.01 | 29.58 | 160.53 |
| 4 cards | 632.09 | 50.26 | 309.27 |