[Enhance] Speed up evaluation on waymo

[JingweiZhang12]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Speed up evaluation on Waymo dataset through the three aspects:

1. not converting to kitti-format
2. reading metainfo from the preprocessed file rather than from original files online.
3. using multi-thread waymo evaluation toolkit: https://github.com/Abyssaledge/waymo-open-dataset-master

The ablation of evaluation time on Waymo for LiDAR-based methods:

Not to KITTI format	Offline metainfo	Multi-thread eval toolkit	time(min)
			30
√			25.5
√	√		18.2
√	√	√	5

BC-breaking (Optional)

Does the modification introduce changes that break the back-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[xyupeng]

Hi @JingweiZhang12 ,

Thanks for your fast evaluation code. Can you kindly share the file idx2metainfo.pkl required for fast evaluation? I didn't find it in this commit.

[JingweiZhang12]

Hi @JingweiZhang12 ,

Thanks for your fast evaluation code. Can you kindly share the file idx2metainfo.pkl required for fast evaluation? I didn't find it in this commit.

Hi, @xyupeng, download link

[xyupeng]

I have a question: is Prediction2Waymo based on the new coordinate system in mmdet3d>=1.0 ? e.g., the new coordinate in LiDARInstance3DBoxes. Because I'm working on mmdet3d==0.17, but I still want to use fast waymo evalution.

[xyupeng]

Thank so much!

[JingweiZhang12]

I have a question: is Prediction2Waymo based on the new coordinate system in mmdet3d>=1.0 ? e.g., the new coordinate in LiDARInstance3DBoxes. Because I'm working on mmdet3d==0.17, but I still want to use fast waymo evalution.

[JingweiZhang12]

I have a question: is Prediction2Waymo based on the new coordinate system in mmdet3d>=1.0 ? e.g., the new coordinate in LiDARInstance3DBoxes. Because I'm working on mmdet3d==0.17, but I still want to use fast waymo evalution.

Yes, it's baed on the new coordinate system. If you use the old coordinate system, you should note that:

Our LiDAR coordinate system has two changes:
The yaw angle is defined to be right-handed instead of left-handed for consistency;
The box dimensions are (l, w, h)  instead of (w, l , h), since w corresponds to dy and l corresponds to dx in KITTI.

Details can be seen https://mmdetection3d.readthedocs.io/en/latest/tutorials/coord_sys_tutorial.html

[xyupeng]

Sorry for choosing the wrong place to comment. But I went through the data pipeline for waymo in mmdet3d 0.17.
I find that the dimensions in gt_bboxes_3d[:, 3:6] are wlh (width, length, height), but not lwh as defined in waymo label proto (i.e., waymo label proto defines xyz correspond to lwh).

The pipeline is:

1. kitti_format/training/label_all/0000000.txt: hwl
2. kitti_format/training/waymo_infos_train.pkl: info['annos']['dimensions']: lhw
3. WaymoDataset.get_ann_info(): ann_results['gt_bboxes_3d'][:, 3:6]: wlh (from CameraInstance3DBoxes to LiDARInstance3DBoxes )

So the final input gt_bboxes_3d has dimensions wlh? Is this the expected behavior instead of lwh?

[JingweiZhang12]

Sorry for choosing the wrong place to comment. But I went through the data pipeline for waymo in mmdet3d 0.17. I find that the dimensions in gt_bboxes_3d[:, 3:6] are wlh (width, length, height), but not lwh as defined in waymo label proto (i.e., waymo label proto defines xyz correspond to lwh).

The pipeline is:

1. kitti_format/training/label_all/0000000.txt: hwl
2. kitti_format/training/waymo_infos_train.pkl: info['annos']['dimensions']: lhw
3. WaymoDataset.get_ann_info(): ann_results['gt_bboxes_3d'][:, 3:6]: wlh (from CameraInstance3DBoxes to LiDARInstance3DBoxes )

So the final input gt_bboxes_3d has dimensions wlh? Is this the expected behavior instead of lwh?

The old lidar coordinate system (before version 1.0) has dimention wlh. https://mmdetection3d.readthedocs.io/en/latest/tutorials/coord_sys_tutorial.html

[xyupeng]

Sorry for choosing the wrong place to comment. But I went through the data pipeline for waymo in mmdet3d 0.17. I find that the dimensions in gt_bboxes_3d[:, 3:6] are wlh (width, length, height), but not lwh as defined in waymo label proto (i.e., waymo label proto defines xyz correspond to lwh).
The pipeline is:

1. kitti_format/training/label_all/0000000.txt: hwl
2. kitti_format/training/waymo_infos_train.pkl: info['annos']['dimensions']: lhw
3. WaymoDataset.get_ann_info(): ann_results['gt_bboxes_3d'][:, 3:6]: wlh (from CameraInstance3DBoxes to LiDARInstance3DBoxes )

So the final input gt_bboxes_3d has dimensions wlh? Is this the expected behavior instead of lwh?

The old lidar coordinate system (before version 1.0) has dimention wlh. https://mmdetection3d.readthedocs.io/en/latest/tutorials/coord_sys_tutorial.html

Got it. Thanks so much.

[ZCMax]

Please merge the latest dev-1.x branch and resolve conflicts

[Tai-Wang]

Convert to KITTI format for camera-only models while not for LiDAR-based models by default. Please provide a basic benchmark table for reference.