Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Training hangs indefinitely #71

Open
julkaztwittera opened this issue Jan 7, 2023 · 0 comments
Open

Training hangs indefinitely #71

julkaztwittera opened this issue Jan 7, 2023 · 0 comments

Comments

@julkaztwittera
Copy link

I am trying to train a nerfie using 8 GPUs, but training hangs at some early steps (about 1-5 k) every time. I tried to decrease batch size, but it only delays the freezing moment. So there must be a kind of memory leak that causes the problem. It does not happen if I use only CPU, of course. I did not change your code. Did anyone report similar problem?

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant