Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Maths domain error #28

Open
adnana-dev opened this issue Jul 26, 2022 · 1 comment
Open

Maths domain error #28

adnana-dev opened this issue Jul 26, 2022 · 1 comment

Comments

@adnana-dev
Copy link

I am using Koila to solve an OOM error during my training. But the following error occurs :
``Traceback (most recent call last):
File "/mnt/sdb2/Adama/configure_docker_for_transvw/pytorch/train.py", line 92, in
loss.backward()
File "/home/nanaa/.local/lib/python3.10/site-packages/koila/lazy.py", line 435, in backward
for mini_batch_size in gpus.split_batch(
File "/home/nanaa/.local/lib/python3.10/site-packages/koila/gpus.py", line 100, in split_batch
batch_size = 2 ** (math.floor(math.log2(max_batch)))
ValueError: math domain error```
Probably due to the value of max_batch ?

@rentruewang
Copy link
Owner

Hi, thanks for the bug report! I'll look into what went wrong.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants