UMERegRobust - Universal Manifold Embedding Compatible Features for Robust Point Cloud Registration (ECCV 2024)

---

In this work, we adopt the Universal Manifold Embedding (UME) framework for the estimation of rigid transformations and extend it, so that it can accommodate scenarios involving partial overlap and differently sampled point clouds. UME is a methodology designed for mapping observations of the same object, related by rigid transformations, into a single low-dimensional linear subspace. This process yields a transformation-invariant representation of the observations, with its matrix form representation being covariant (i.e. equivariant) with the transformation. We extend the UME framework by introducing a UME-compatible feature extraction method augmented with a unique UME contrastive loss and a sampling equalizer. These components are integrated into a comprehensive and robust registration pipeline, named UMERegRobust. We propose the RotKITTI registration benchmark, specifically tailored to evaluate registration methods for scenarios involving large rotations. UMERegRobust achieves better than state-of-the-art performance on the KITTI benchmark, especially when strict precision of $(1^\circ, 10cm)$ is considered (with an average gain of +9%), and notably outperform SOTA methods on the RotKITTI benchmark (with +45% gain compared the most recent SOTA method).

Paper Link: https://www.arxiv.org/abs/2408.12380

---

Method Overview

---

Environment Setup

Code was tested on:

• Ubuntu 20.04
• Python 3.8
• Cuda 11.7
• Pytorch 1.13.0+cu117

Special Packages Used:

• MinkowskiEngine
• Pytorch3D
• PyTorch Scatter
• NKSR

Create Env:

# Create Conda Env
conda create umereg_conda_env python=3.8

# Install CUDA Toolkit 11.7
conda install nvidia/label/cuda-11.7.0::cuda-toolkit
conda install conda-forge::cudatoolkit-dev

# Git for Conda
conda install git

# Install Pytorch 1.13.0+cu117
pip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu117

# Install MinkowskiEngine 
pip install -U git+https://github.com/NVIDIA/MinkowskiEngine -v --no-deps --config-settings="--blas_include_dirs=${CONDA_PREFIX}/include" --config-settings="--blas=openblas"

# Install Pytorch3D + torch_scatter
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
pip install torch-scatter -f https://data.pyg.org/whl/torch-1.13.0+cu117.html

# NKSR
pip install -U nksr -f https://nksr.huangjh.tech/whl/torch-1.13.0+cu117.html

# Other Relevant Packages
pip install open3d
pip install tensorboard

Clone UMERegRobust Repository:

git clone https://github.com/yuvalH9/UMERegRobust.git

---

Datasets

You can evaluate or train UMERegRobust on both the KITTI dataset and the nuScenes dataset.

Please refer to the detailed datasets guidelines:

• KITTI Guideline
• nuScenes Guideline

---

Sampling Equalizer Module (SEM) Preprocessing

To use the SEM to preprocess the input point cloud please use:

python datasets/sem_preprocessing.py --dataset_mode [kitti\nuscenes] --split [train\val] --data_path path_to_input_data --output_path path_to_output

We also supply download links to the SEM already preprocessed data for both KITTI (test, lokitt, rotkitti) and nuScenes (test, lonuscenes, rotnuscens) registration benchmarks.

• Download KITTI registration benchmark preprocessed data
• Download nuScenes registration benchmark preprocessed data

---

RotKITTI & RotNuscenes Registration Benchmarks

We suggest new registration benchmarks RotKITTI and RotNuscenes, these benchmarks focus on point cloud pairs with big relative rotations in the wild (not synthetic rotations). Each benchmark contains registration problems with relative rotations ranging between 30-180 degrees. We encourage the comunity to test thier method on those benchmakrs.

To use the benchmarks, first download the KITTI \ nuScenes datasets as described in section Datasets. Next, the registration problems (source-target pairs) are saved in the files rotkitti_metadata.npy and rotnuscenes_metadata.npy, along with there corresponding GT transformations in the files rotkitti_gt_tforms.npy and rotnuscenes_metadata.npy, respectively.

---

Usage

Eval

1. Download the original data as described in section Datasets to data_path.
2. Download the SEM preprocessed data as described in section SEM Preprocessing to cache_data_path.
3. Update paths in relevant benchmark config files.
4. Evaluate KITTI benchmarks:

   python evaluate.py --benchmark [kitti_test\lokitti\rotkitti]

5. Evaluate nuScenes benchmarks:

   python evaluate.py --benchmark [nuscenes_test\lonuscenes\rotnuscenes]

Train

1. Download the original data as described in section Datasets to data_path.
2. Run the SEM preprocessing for train and val splits as described in section SEM Preprocessing output data to cache_data_path.
3. Update paths in relevant train config files.
4. Train KITTI:

   python train_coloring.py --config kitti

5. Train nuScenes benchmarks:

   python train_coloring.py --config nuscenes

---

Results - KITTI Benchmarks

KITTI Test

Method	Normal Precision (1.5°, 30 cm)	Strict Precision (1°, 10 cm)
FCGF	75.1	73.1
Predetor	88.2	58.7
CoFiNet	83.2	56.4
GeoTrans	66.3	62.6
GCL	93.9	78.6
UMERegRobust	94.3	87.8

Table1: KITTI Benchmark - Registration Recall [%]

RotKITTI

Method	Normal Precision (1.5°, 30 cm)	Strict Precision (1°, 10 cm)
FCGF	11.6	3.6
Predetor	41.6	35.0
CoFiNet	62.5	30.1
GeoTrans	78.5	50.1
GCL	40.1	28.8
UMERegRobust	81.1	73.3

Table2: RotKITTI Benchmark - Registration Recall [%]

LoKITTI

Method	Normal Precision (1.5°, 30 cm)	Strict Precision (1°, 10 cm)
FCGF	17.2	6.9
Predetor	33.7	28.4
CoFiNet	11.2	1.0
GeoTrans	37.8	7.2
GCL	72.3	26.9
UMERegRobust	59.3	30.2

Table3: LoKITTI Benchmark - Registration Recall [%]

---

Results - nuScenes Benchmarks

nuScenes Test

Method	Normal Precision (1.5°, 30 cm)	Strict Precision (1°, 10 cm)
FCGF	58.2	37.8
Predetor	53.9	48.1
CoFiNet	62.3	56.1
GeoTrans	70.7	37.9
GCL	82.0	67.5
UMERegRobust	85.5	76.0

Table4: nuScenes Benchmark - Registration Recall [%]

RotNuscenes

Method	Normal Precision (1.5°, 30 cm)	Strict Precision (1°, 10 cm)
FCGF	5.5	5.2
Predetor	16.5	15.7
CoFiNet	27.0	23.6
GeoTrans	34.3	13.1
GCL	21.0	19.6
UMERegRobust	51.9	39.7

Table5: RotNuScenes Benchmark - Registration Recall [%]

LoNuscenes

Method	Normal Precision (1.5°, 30 cm)	Strict Precision (1°, 10 cm)
FCGF	1.9	0.0
Predetor	35.6	4.2
CoFiNet	30.3	23.5
GeoTrans	48.1	17.3
GCL	62.3	5.6
UMERegRobust	70.8	56.3

Table6: LoNuScenes Benchmark - Registration Recall [%]

---

Citation

If you find this work useful, please cite:

@misc{haitman2024umeregrobust,
      title={UMERegRobust - Universal Manifold Embedding Compatible Features for Robust Point Cloud Registration}, 
      author={Yuval Haitman and Amit Efraim and Joseph M. Francos},
      year={2024},
      eprint={2408.12380},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.12380}, 
}

---