Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Could you please share system CPU and GPU requirements? #52

Open
csrvictory opened this issue Jul 22, 2020 · 2 comments
Open

Could you please share system CPU and GPU requirements? #52

csrvictory opened this issue Jul 22, 2020 · 2 comments

Comments

@csrvictory
Copy link

Sir

I tried to run it on 8GB GPU(Active 7 GB) V100D, with 64 GB RAM,
But i got unknow error: during model.fit
Loading Convolution Algorithm is failed due to memory constraints even in the case of batch_size=1.

kindly suggest me in this regard

UnknownError Traceback (most recent call last)
in
----> 1 model.fit(data, [labels, data], batch_size=1, epochs=1)

~.conda\envs\Gandhiji\lib\site-packages\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
1176 steps_per_epoch=steps_per_epoch,
1177 validation_steps=validation_steps,
-> 1178 validation_freq=validation_freq)
1179
1180 def evaluate(self,

~.conda\envs\Gandhiji\lib\site-packages\keras\engine\training_arrays.py in fit_loop(model, fit_function, fit_inputs, out_labels, batch_size, epochs, verbose, callbacks, val_function, val_inputs, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps, validation_freq)
202 ins_batch[i] = ins_batch[i].toarray()
203
--> 204 outs = fit_function(ins_batch)
205 outs = to_list(outs)
206 for l, o in zip(out_labels, outs):

~.conda\envs\Gandhiji\lib\site-packages\keras\backend\tensorflow_backend.py in call(self, inputs)
2977 return self._legacy_call(inputs)
2978
-> 2979 return self._call(inputs)
2980 else:
2981 if py_any(is_tensor(x) for x in inputs):

~.conda\envs\Gandhiji\lib\site-packages\keras\backend\tensorflow_backend.py in _call(self, inputs)
2935 fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)
2936 else:
-> 2937 fetched = self._callable_fn(*array_vals)
2938 return fetched[:len(self.outputs)]
2939

~.conda\envs\Gandhiji\lib\site-packages\tensorflow\python\client\session.py in call(self, *args, **kwargs)
1456 ret = tf_session.TF_SessionRunCallable(self._session._session,
1457 self._handle, args,
-> 1458 run_metadata_ptr)
1459 if run_metadata:
1460 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node Input_x1/convolution}}]]
[[loss/Dec_VAE_Output_loss/Mean_3/_2353]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node Input_x1/convolution}}]]
0 successful operations.
0 derived errors ignored.

@jayurbain
Copy link

What size image or patch (volume) are you using?
What is your batch size?
Based on your volume size and batch size, you probably do not have enough GPU memory.

@locdoan12121997
Copy link

Look more like a failed cudnn installation to me

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants