CUDNN_STATUS_NOT_SUPPORTED

[edebrouwer]

Hi, I'm having this error while trying to train the model on CelebA dataset :

Traceback (most recent call last):
File "train.py", line 347, in
train(args, dataset, generator, discriminator, device)
File "train.py", line 167, in train
fake_predict.backward()
File "/home/edward/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/edward/anaconda3/lib/python3.6/site-packages/torch/autograd/init.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

I'm running Python 3.6.4 with torch 1.4.0. Nvidia-driver 418.56

Any idea where I could look for some fix ?

Thanks so much !

Edward

[rosinality]

Could you give me the error logs with CUDA_LAUNCH_BLOCKING=1?

[edebrouwer]

I have exactly the same error logs with CUDA_LAUNCH_BLOKCING=1

[rosinality]

Could you check output dimensions? Sometimes this kind of error happens when tensor is too large.

[edebrouwer]

I tried with smaller batch sizes and it still fails on the lowest resolution.

…

On 17 Feb 2020, at 02:10, Kim Seonghyeon ***@***.***> wrote: Could you check output dimensions? Sometimes this kind of error happens when tensor is too large. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#83?email_source=notifications&email_token=AFM3FN6C6JLQOOJRO3GZPF3RDHPZPA5CNFSM4KVNFF3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4YY5Q#issuecomment-586779766>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFM3FNZ7LDXZNCZK4HD65XDRDHPZPANCNFSM4KVNFF3A>.

[rosinality]

Sorry but it is hard to know why...

[Dimbl4]

@edebrouwer try to upgrade torch 1.4.0 to nightly, it helped me. (cuda 10.0, torch 1.5.0.dev20200313+cu100)

[KelestZ]

Same issue. Did anyone fix it?

[seongeunso]

I had same issue in here, so I downgrade the torch version.
For me, torch 1.2.0 + cuda 10.0 works well (link)