HGL: Hierarchical Geometry Learning for Test-time Adaptation in 3D Point Cloud Segmentation [ECCV2024 Oral]

The official implementation of our work "HGL: Hierarchical Geometry Learning for Test-time Adaptation in 3D Point Cloud Segmentation".

Introduction

3D point cloud segmentation has received significant interest for its growing applications. However, the generalization ability of models suffers in dynamic scenarios due to the distribution shift between test and training data. To promote robustness and adaptability across diverse scenarios, test-time adaptation (TTA) has recently been introduced. Nevertheless, most existing TTA methods are developed for images, and limited approaches applicable to point clouds ignore the inherent hierarchical geometric structures in point cloud streams, i.e., local (point-level), global (object-level), and temporal (frame-level) structures. In this paper, we delve into TTA in 3D point cloud segmentation and propose a novel Hierarchical Geometry Learning (HGL) framework. HGL comprises three complementary modules from local, global to temporal learning in a bottom-up manner. Technically, we first construct a local geometry learning module for pseudo-label generation. Next, we build prototypes from the global geometry perspective for pseudo-label fine-tuning. Furthermore, we introduce a temporal consistency regularization module to mitigate negative transfer. Extensive experiments on four datasets demonstrate the effectiveness and superiority of our HGL. Remarkably, on the SynLiDAR to SemanticKITTI task, HGL achieves an overall mIoU of 46.91%, improving GIPSO by 3.0% and significantly reducing the required adaptation time by 80%.

Environment

• MinkowskiEnginge.
• open3d 0.13.0
• KNN-CUDA
• pytorch-lighting 1.4.1
• wandb
• nuscenes-devkit
• tqdm
• pickle

NOTE ！！！:

• This code is the initial version and I haven't had enough time to trim and optimize it. I will update it after the CVPR2025 deadline.. The reason for publishing it in advance is to facilitate further exploration of the 3D TTA problem by other researchers. The code's mainly based on the GIPSO framework and our main change points are use_pseudo_new(LGL), use_prototype(GFG), score_weight(TGR) and SoftDICELoss.
• I will be working on optimizing the code, in the meantime feel free to contact me if you have any questions!

Source training

To train the source model on SynLiDAR

python train_lighting.py --config_file configs/source/synlidar_source.yaml

For Synth4D --config_file configs/source/synth4dkitti_source.yaml.

For nuScenes --config_file configs/source/synth4dnusc_source.yaml

Pretrained models

We use the pretrained models on Synth4D-KITTI, Synth4D-nuScenes and SynLIDAR provided by GIPSO. You can find the models here. For the model performance please refer to the main paper.

After downloading the pretrained models decompress them in /pretrained_models.

Adaptation to target

To adapt the source model SynLiDAR to the target domain SemanticKITTI

sh train.sh

If you want to save point cloud for future visualization you will need to add --save_predictions and they will be saved in pipeline.save_dir.

Thanks

We thanks the open source projects Minkowski-Engine and GIPSO.