[Bug] Fix for Point RCNN memory usage increases during training is not merged after several major releases

[APassbyDreg]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

same as #1907

Reproduces the problem - code sample

same as #1907

Reproduces the problem - command or script

same as #1907

Reproduces the problem - error message

same as #1907

Additional information

The problem mentioned in #1907 is still not merged to the main branch.