training at increasing depth

[hukkai]

Hi, I wanted to reproduce the results in your fixup paper. (Figure 3: Depth of residual networks versus test accuracy at the first epoch for various methods on CIFAR-10 with the default BatchNorm learning rate. )

The figure shows that fixup resnet can achieve 50% test accuracy on cifar10 when the depth < 1000. But I can only get about 40% after multiple runs.

I use google's colab P100 GPU, pytorch 1.3.0 to do the experiments. Here is my codes (I used the same script in the readme):
!rm -rf *
!git clone https://github.com/hongyi-zhang/Fixup.git
!mv Fixup/cifar/* .
!rm -rf Fixup
!python cifar_train.py -a fixup_resnet32 --sess benchmark_a0d5e4lr01 --seed 11111 --alpha 0. --decay 5e-4 --n_epoch=1

Here is my colab link: https://colab.research.google.com/drive/10aj0-vEGHlqxZ95oS5RLaCwIdQgDepjM

Could you help me look into this? Thanks!