Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

fp16_run: true causes NaNs #60

Open
CrackerHax opened this issue Aug 23, 2022 · 0 comments
Open

fp16_run: true causes NaNs #60

CrackerHax opened this issue Aug 23, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@CrackerHax
Copy link

CrackerHax commented Aug 23, 2022 •
edited
Loading

Posting this as a new issue in case anybody else has the problem.

There might be hacky workarounds for this but it seems to be a pytorch issue: pytorch/pytorch#40497

So I guess the answer is to train with fp16_run: false for now. Might want to note this in the README to save people time. Also the config file default is set to true, could probably set that to false.
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@CrackerHax @yl4579