Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Maybe a small bug about test progress bar in multi_gpu_test(). #921

Open
MisakaMikt opened this issue Dec 8, 2023 · 1 comment
Open

Comments

@MisakaMikt
Copy link

During distributed training, sot model use multi_gpu_test() func to test all videos on several GPUs. But the progress bar is updated on rank 0.

https://github.com/open-mmlab/mmtracking/blob/e79491ec8f0b8c86fda947fbaaa824c66ab2a991/mmtrack/apis/test.py#L160C34-L160C34

The number of frames on each rank is not equal, in my run, the number of frames on rank0 is less than total_frames / world_size, so the progress bar will be less than the actual total number of frames. For example, I used 3 gpu, and total 21007 frames(got10k val), and here is some out info. the final progress only 5565*3=16695.

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ] 16695/21007, 99.7 task/s, elapsed: 167s, ETA: 43s
rank 0 world_size 3 cnt 5565.
rank 1 world_size 3 cnt 7072.
rank 2 world_size 3 cnt 8370.

@tericalpha
Copy link

tericalpha commented Dec 18, 2023

Hello @MisakaMikt
Haven't you got this MMCV compatibility error?

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants