Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Strange training behaviour #9

Open
Vadim2S opened this issue Feb 20, 2024 · 1 comment
Open

Strange training behaviour #9

Vadim2S opened this issue Feb 20, 2024 · 1 comment

Comments

@Vadim2S
Copy link

Vadim2S commented Feb 20, 2024

I am run training and observe strange behaviour.

Last model.pt created at 19.02.2024 16:51. I am have model_0.pt ... model_60000.pt files. train.log says:

изображение

Now 20.02.2024 8:30 am. and htop says:

изображение

I.e. there a lots of CPU consumptions. Almost none GPU consumptions and last log\models changes was half-day ago. What is it?
@wxzwxzwxz
Copy link
Collaborator

Hi @Vadim2S

Could you provide additional information, such as nvidia-smi? The references to PNSR and training time, as indicated in the log file, appear to be normal. During the training process, it takes approximately 30 hours.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wxzwxzwxz @Vadim2S