Skip to content

[NeurIPS 2022] DeepInteraction: 3D Object Detection via Modality Interaction

License

Notifications You must be signed in to change notification settings

fudan-zvg/DeepInteraction

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DeepInteraction & DeepInteraction++

DeepInteraction: 3D Object Detection via Modality Interaction,
Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, Li Zhang
NeurIPS 2022

DeepInteraction++: Multi-Modality Interaction for Autonomous Driving,
Zeyu Yang, Nan Song, Wei Li, Xiatian Zhu, Li Zhang, Philip H.S. Torr
Arxiv 2024

News

  • (2022/6/27) DeepInteraction-e ranks first on nuScenes among all solutions.
  • (2022/6/26) DeepInteraction-large ranks first on nuScenes among all non-ensemble solutions.
  • (2022/5/18) DeepInteraction-base ranks first on nuScenes among all solutions that do not use test-time augmentation and model ensemble.

Results

3D Object Detection (on nuScenes test)

Model Modality mAP NDS
DeepInteraction-e C+L 75.74 76.34
DeepInteraction-large C+L 74.12 75.52
DeepInteraction-base C+L 70.78 73.43

3D Object Detection (on nuScenes val)

Model Modality mAP NDS Checkpoint
DeepInteraction C+L 69.85 72.63 Fusion_0075_refactor.pth
DeepInteraction++ C+L 70.63 73.27 Fusion_0075_plusplus.pth

Get Started

Environment

This implementation is build upon mmdetection3d, please follow the steps in install.md to prepare the environment.

Data

Please follow the official instructions of mmdetection3d to process the nuScenes dataset.(https://mmdetection3d.readthedocs.io/en/latest/datasets/nuscenes_det.html)

Pretrained

Downloads the pretrained backbone weights to pretrained/

Train & Test

# train DeepInteraction with 8 GPUs
tools/dist_train.sh projects/configs/nuscenes/Fusion_0075_refactor.py 8
# test DeepInteraction with 8 GPUs
tools/dist_test.sh projects/configs/nuscenes/Fusion_0075_refactor.py ${CHECKPOINT_FILE} 8 --eval=bbox

Acknowledgement

Many thanks to the following open-source projects:

Reference

@inproceedings{yang2022deepinteraction,
  title={DeepInteraction: 3D Object Detection via Modality Interaction},
  author={Yang, Zeyu and Chen, Jiaqi and Miao, Zhenwei and Li, Wei and Zhu, Xiatian and Zhang, Li},
  booktitle={NeurIPS},
  year={2022}
}
@inproceedings{yang2024deepinteractionpp,
  title={DeepInteraction++: Multi-Modality Interaction for Autonomous Driving},
  author={Yang, Zeyu and Song, Nan and Li, Wei and Zhu, Xiatian and Zhang, Li and Philip H.S.},
  booktitle={Arxiv},
  year={2024}
}

About

[NeurIPS 2022] DeepInteraction: 3D Object Detection via Modality Interaction

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages