You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I tried to run it on 8GB GPU(Active 7 GB) V100D, with 64 GB RAM,
But i got unknow error: during model.fit
Loading Convolution Algorithm is failed due to memory constraints even in the case of batch_size=1.
~.conda\envs\Gandhiji\lib\site-packages\keras\engine\training_arrays.py in fit_loop(model, fit_function, fit_inputs, out_labels, batch_size, epochs, verbose, callbacks, val_function, val_inputs, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps, validation_freq)
202 ins_batch[i] = ins_batch[i].toarray()
203
--> 204 outs = fit_function(ins_batch)
205 outs = to_list(outs)
206 for l, o in zip(out_labels, outs):
~.conda\envs\Gandhiji\lib\site-packages\keras\backend\tensorflow_backend.py in call(self, inputs)
2977 return self._legacy_call(inputs)
2978
-> 2979 return self._call(inputs)
2980 else:
2981 if py_any(is_tensor(x) for x in inputs):
~.conda\envs\Gandhiji\lib\site-packages\tensorflow\python\client\session.py in call(self, *args, **kwargs)
1456 ret = tf_session.TF_SessionRunCallable(self._session._session,
1457 self._handle, args,
-> 1458 run_metadata_ptr)
1459 if run_metadata:
1460 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node Input_x1/convolution}}]]
[[loss/Dec_VAE_Output_loss/Mean_3/_2353]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node Input_x1/convolution}}]]
0 successful operations.
0 derived errors ignored.
The text was updated successfully, but these errors were encountered:
What size image or patch (volume) are you using?
What is your batch size?
Based on your volume size and batch size, you probably do not have enough GPU memory.
Sir
I tried to run it on 8GB GPU(Active 7 GB) V100D, with 64 GB RAM,
But i got unknow error: during model.fit
Loading Convolution Algorithm is failed due to memory constraints even in the case of batch_size=1.
kindly suggest me in this regard
UnknownError Traceback (most recent call last)
in
----> 1 model.fit(data, [labels, data], batch_size=1, epochs=1)
~.conda\envs\Gandhiji\lib\site-packages\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
1176 steps_per_epoch=steps_per_epoch,
1177 validation_steps=validation_steps,
-> 1178 validation_freq=validation_freq)
1179
1180 def evaluate(self,
~.conda\envs\Gandhiji\lib\site-packages\keras\engine\training_arrays.py in fit_loop(model, fit_function, fit_inputs, out_labels, batch_size, epochs, verbose, callbacks, val_function, val_inputs, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps, validation_freq)
202 ins_batch[i] = ins_batch[i].toarray()
203
--> 204 outs = fit_function(ins_batch)
205 outs = to_list(outs)
206 for l, o in zip(out_labels, outs):
~.conda\envs\Gandhiji\lib\site-packages\keras\backend\tensorflow_backend.py in call(self, inputs)
2977 return self._legacy_call(inputs)
2978
-> 2979 return self._call(inputs)
2980 else:
2981 if py_any(is_tensor(x) for x in inputs):
~.conda\envs\Gandhiji\lib\site-packages\keras\backend\tensorflow_backend.py in _call(self, inputs)
2935 fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)
2936 else:
-> 2937 fetched = self._callable_fn(*array_vals)
2938 return fetched[:len(self.outputs)]
2939
~.conda\envs\Gandhiji\lib\site-packages\tensorflow\python\client\session.py in call(self, *args, **kwargs)
1456 ret = tf_session.TF_SessionRunCallable(self._session._session,
1457 self._handle, args,
-> 1458 run_metadata_ptr)
1459 if run_metadata:
1460 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node Input_x1/convolution}}]]
[[loss/Dec_VAE_Output_loss/Mean_3/_2353]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node Input_x1/convolution}}]]
0 successful operations.
0 derived errors ignored.
The text was updated successfully, but these errors were encountered: