PointRCNN (and also the pretrained model) doesn't perform properly on the KITTI dataset

[vahdat-ab]

Hi,
I installed 1.0.0.rc2 and run the dataset generation for the KITTI dataset.
I evaluated the provided pre-trained model for PointRCNN over the KITTI dataset and it doesn't show the performance reported on the paper. The results are:

----------- AP11 Results ------------                                                                                                                                                                      
                                                                                                                                                                                                           
Car AP11@0.70, 0.70, 0.70:                                                                                                                                                                                 
bbox AP11:0.0000, 0.0000, 0.0339                                                                                                                                                                           
bev  AP11:0.0000, 0.0866, 0.0866                                                                                                                                                                           
3d   AP11:0.0000, 0.0000, 0.0000                                                                                                                                                                           
aos  AP11:0.00, 0.00, 0.00                                                                                                                                                                                 
Car AP11@0.70, 0.50, 0.50:                                                                                                                                                                                 
bbox AP11:0.0000, 0.0000, 0.0339                                                                                                                                                                           
bev  AP11:0.0000, 0.0866, 0.0866                                                                                                                                                                           
3d   AP11:0.0000, 0.0000, 0.0000                                                                                                                                                                           
aos  AP11:0.00, 0.00, 0.00                                                                                                                                                                                 
Pedestrian AP11@0.50, 0.50, 0.50:                                                                                                                                                                          
bbox AP11:6.6845, 6.2422, 6.1098                                                                                                                                                                           
bev  AP11:12.7065, 12.0848, 11.5359                                                                                                                                                                        
3d   AP11:0.0000, 0.0000, 0.0019                                                                                                                                                                           
aos  AP11:6.58, 6.15, 5.98                                                                                                                                                                                 
Pedestrian AP11@0.50, 0.25, 0.25:                                                                                                                                                                          
bbox AP11:6.6845, 6.2422, 6.1098                                                                                                                                                                           
bev  AP11:42.9215, 39.5759, 34.6241                                                                                                                                                                        
3d   AP11:35.1524, 32.6993, 28.4828                                                                                                                                                                        
aos  AP11:6.58, 6.15, 5.98                                                                                                                                                                                 
Cyclist AP11@0.50, 0.50, 0.50:                                                                                                                                                                             
bbox AP11:3.7611, 3.0747, 2.5931                                                                                                                                                                           
bev  AP11:0.5722, 1.5152, 1.5152                                                                                                                                                                           
3d   AP11:0.0000, 0.0000, 0.0000                                                                                                                                                                           
aos  AP11:3.76, 3.07, 2.59                                                                                                                                                                                 
Cyclist AP11@0.50, 0.25, 0.25:                                                                                                                                                                             
bbox AP11:3.7611, 3.0747, 2.5931                                                                                                                                                                           
bev  AP11:48.6239, 43.3577, 40.6218                                                                                                                                                                        
3d   AP11:23.8051, 19.6509, 19.2659                                                                                                                                                                        
aos  AP11:3.76, 3.07, 2.59                                                                                                                                                                                 
                                                                                                                                                                                                           
Overall AP11@easy, moderate, hard:                                                                                                                                                                         
bbox AP11:3.4818, 3.1056, 2.9123                                                                                                                                                                           
bev  AP11:4.4262, 4.5622, 4.3792                                                                                                                                                                           
3d   AP11:0.0000, 0.0000, 0.0006                                                                                                                                                                           
aos  AP11:3.45, 3.07, 2.86             
----------- AP40 Results ------------                                                                                                                                                                      
                                                                                                                                                                                                           
Car AP40@0.70, 0.70, 0.70:                                                                                                                                                                                 
bbox AP40:0.0000, 0.0000, 0.0000                                                                                                                                                                           
bev  AP40:0.0000, 0.0000, 0.0000                                                                                                                                                                           
3d   AP40:0.0000, 0.0000, 0.0000                                                                                                                                                                           
aos  AP40:0.00, 0.00, 0.00                                                                                                                                                                                 
Car AP40@0.70, 0.50, 0.50:                                                                                                                                                                                 
bbox AP40:0.0000, 0.0000, 0.0000
bev  AP40:0.0000, 0.0198, 0.0198
3d   AP40:0.0000, 0.0000, 0.0000
aos  AP40:0.00, 0.00, 0.00
Pedestrian AP40@0.50, 0.50, 0.50:
bbox AP40:2.8530, 2.6450, 2.2946
bev  AP40:5.2249, 4.6793, 3.7824
3d   AP40:0.0000, 0.0000, 0.0000
aos  AP40:2.74, 2.53, 2.18
Pedestrian AP40@0.50, 0.25, 0.25:
bbox AP40:2.8530, 2.6450, 2.2946
bev  AP40:40.2142, 36.3923, 31.3368
3d   AP40:31.1603, 28.6751, 24.0590
aos  AP40:2.74, 2.53, 2.18
Cyclist AP40@0.50, 0.50, 0.50:
bbox AP40:3.2933, 2.2465, 2.1000
bev  AP40:0.2561, 0.1185, 0.1182
3d   AP40:0.0000, 0.0000, 0.0000
aos  AP40:3.29, 2.24, 2.10
Cyclist AP40@0.50, 0.25, 0.25:
bbox AP40:3.2933, 2.2465, 2.1000
bev  AP40:46.9033, 40.4245, 38.5530
3d   AP40:18.5216, 14.2241, 13.6172
aos  AP40:3.29, 2.24, 2.10

Overall AP40@easy, moderate, hard:
bbox AP40:2.0488, 1.6305, 1.4649
bev  AP40:1.8270, 1.5993, 1.3002
3d   AP40:0.0000, 0.0000, 0.0000
aos  AP40:2.01, 1.59, 1.42

I also trained the model from scratch with the default config file and couldn't get the reported performance.
I assume it might be related to the refactoring of coordinate systems.

[Tai-Wang]

It should not be related to the refactoring of coordinate systems because PointRCNN is merged after the release of 1.0.0rc.
"couldn't get the reported performance" with training from scratch means similar performance with the one shown above? We can first try to reproduce your problem with the given pretrained model.

[vahdat-ab]

Yes. Even if I train the default config file over 8 GPUs, the reported mAP is similar to what I reported above. I tried it with the dataset generated with different rc0, rc1, and 0.18.1. I was not able to get the reported mAP? You're right it may not be related to the coordinate systems.

[VVsssssk]

Hi, I have to fix it in #1874. And I will update PointRCNN checkpoints in the model zoo later.