Lower accuracy than expected on VOC 2007 model & data

[mbuckler]

Thank you for providing your code. I have installed and run the test provided but unfortunately I am seeing lower accuracy on the VOC 2007 benchmark than I expected.

On the Readme I see that the model achieves 71.2 but when I run ./experiments/scripts/test_vgg16.sh 0 pascal_voc with VOC 2007 data and your model I see a result of Mean AP = 0.4955. If I am right this is meant to be interpreted as an mAP of 49.55. Should I be using a different testing script or different model than the one downloaded by ./data/scripts/fetch_faster_rcnn_models.sh ? Here are the full results:

AP for aeroplane = 0.5898 AP for bicycle = 0.5308 AP for bird = 0.4317 AP for boat = 0.3876 AP for bottle = 0.2347 AP for bus = 0.6052 AP for car = 0.5414 AP for cat = 0.6908 AP for chair = 0.2789 AP for cow = 0.5222 AP for diningtable = 0.5555 AP for dog = 0.6149 AP for horse = 0.7065 AP for motorbike = 0.5160 AP for person = 0.4421 AP for pottedplant = 0.2304 AP for sheep = 0.4441 AP for sofa = 0.5538 AP for train = 0.6770 AP for tvmonitor = 0.3559 Mean AP = 0.4955

[mbuckler]

I should also note that after completion of the testing script I receive a notification about a segfault which I am unsure how to trace. Perhaps this may be a part of the problem?

./experiments/scripts/test_vgg16.sh: line 60: 338 Segmentation fault (core dumped) CUDA_VISIBLE_DEVICES=${GPU_ID} python ./tools/test_vgg16_net.py --imdb ${TEST_IMDB} --weight data/imagenet_weights/vgg16.weights --model ${NET_FINAL} --cfg experiments/cfgs/vgg16.yml --set ${EXTRA_ARGS}

[endernewton]

Not sure what happened in your case. Can you provide your hardware and software setup? I don't have segfault here.

[mbuckler]

Hey! Thank you very much for replying. As it turns out the reason for the incorrect accuracy was due to the wrong GPU architecture being specified in the extra_compile_args in the setup.py script. I needed -arch=sm+52 because I am using a Titan X. Interestingly the segfault still appears at the end of testing, but since the testing is functional I have all that I need to continue.

I'll close this issue for sure since everything is working, but I should also mention that a few things might want to be changed in the code to help Linux users like myself. I've made a pull request that you might be interested in. Thanks again!

[caunion]

Suffered from similar problem! I have a little old GPU, gtx 780, the third generation architecture (Kepler). If using the default sm_52, the accuracy is incorrect. Change it to 'sm_35' will get the reported mAP...

It is so tricky. I wonder why the compilation of nms was successful when using incorrect -arch parameter..

[caunion]

@mbuckler After correct the sm_xx, did you get extact 71.2 mAP? I correct the sm_xx for my GTX780, but tested with the pretrained model voc07 from the authors page, I got the mAP 70.89; while with the pretrained model on voc07+voc12, I tested with the mAP 74.95. Is these correct numbers? @endernewton

[endernewton]

TEST.MODE top? default is a little bit lower TEST.MODE nms.

…

--------------------- Ender X. Chen Language Technologies Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA

On Sat, Apr 29, 2017 at 6:40 PM, Xiaoke Jia ***@***.***> wrote: @mbuckler <https://github.com/mbuckler> After correct the sm_xx, did you get extact 71.2 mAP? I correct the sm_xx for my GTX780, but tested with the pretrained model voc07 from the authors page, I got the mAP 70.89; while with the pretrained model on voc07+voc12, I tested with the mAP 74.95. Is these correct numbers? @endernewton <https://github.com/endernewton> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACTwEWmz0_yiND8-Y2pcNgXWtv-G3piGks5r07x5gaJpZM4L9-wq> .

[cjcchen]

Hi, I also got 0.5206 mAP after running the test script. I am using Telsa P100 and I have set the arch to be sm_60 according to "http://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/". Is it anything wrong?