RuntimeError: CUDA error: an illegal memory access was encountered

[negiabhinit]

Checklist

• I have searched for similar issues.
• I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for main branch).

My Question

running the following script
(o3dml) abhnegi@tghfg~/o3dml/Open3D-ML$ python scripts/run_pipeline.py torch -c ml3d/configs/pointpillars_kitti.yml --split test --dataset.dataset_path "/pfs/rdi/cea/rdicea_vru/01_Datasets/Kitti/" --pipeline ObjectDetection --dataset.use_cache True

gives me this error :
Traceback (most recent call last):
File "/pfs/rdi/cea/home/abhnegi/o3dml/Open3D-ML/scripts/run_pipeline.py", line 261, in
sys.exit(main())
File "/pfs/rdi/cea/home/abhnegi/o3dml/Open3D-ML/scripts/run_pipeline.py", line 190, in main
pipeline.run_test()
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 114, in run_test
self.run_valid()
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 191, in run_valid
results = model(data)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 132, in forward
x = self.extract_feats(inputs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 104, in extract_feats
voxels, num_points, coors = self.voxelize(points)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 117, in voxelize
res_voxels, res_coors, res_num_points = self.voxel_layer(res)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 361, in forward
[torch.zeros_like(points_feats[0:1, :]), points_feats])
RuntimeError: CUDA error: an illegal memory access was encountered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.