在nvidia A4000显卡上无法训练

[kawais]

使用命令python train.py --dataset Hayao --epoch 101 --init_epoch 10
训练过程中提示错误 failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED。
这个要怎么解决？

软件版本：

packages in environment at C:\ProgramData\Anaconda3\envs\py36:

Name Version Build Channel

absl-py 1.0.0 pypi_0 pypi
astor 0.8.1 pypi_0 pypi
cached-property 1.5.2 pypi_0 pypi
certifi 2021.5.30 py36haa95532_0
colorama 0.4.4 pypi_0 pypi
cudatoolkit 10.0.130 0
cudnn 7.6.0 cuda10.0_0
dataclasses 0.8 pypi_0 pypi
gast 0.2.2 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.44.0 pypi_0 pypi
h5py 3.1.0 pypi_0 pypi
importlib-metadata 4.8.3 pypi_0 pypi
keras-applications 1.0.8 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
markdown 3.3.6 pypi_0 pypi
numpy 1.19.5 pypi_0 pypi
opencv-python 4.5.5.62 pypi_0 pypi
opt-einsum 3.3.0 pypi_0 pypi
pip 21.2.2 py36haa95532_0
protobuf 3.19.4 pypi_0 pypi
python 3.6.2 h09676a0_15
setuptools 58.0.4 py36haa95532_0
six 1.16.0 pypi_0 pypi
tensorboard 1.15.0 pypi_0 pypi
tensorflow-estimator 1.15.1 pypi_0 pypi
tensorflow-gpu 1.15.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
tqdm 4.62.3 pypi_0 pypi
typing-extensions 4.1.1 pypi_0 pypi
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 2.0.3 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
wincertstore 0.2 py36h7fe50ca_0
wrapt 1.13.3 pypi_0 pypi
zipp 3.6.0 pypi_0 pypi

错误日志：
2022-02-23 11:58:11.480753: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
2022-02-23 11:58:11.483857: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
2022-02-23 11:58:11.483905: I tensorflow/stream_executor/stream.cc:4976] [stream=000001E1AF327560,impl=000001E1A391B2F0] did not memset GPU location; source: 000000B3A7DCBCE8; size: 8388608; pattern: ffffffff
2022-02-23 11:58:11.486156: I tensorflow/stream_executor/stream.cc:4976] [stream=000001E1AF327560,impl=000001E1A391B2F0] did not memset GPU location; source: 000000B3A760C038; size: 8388608; pattern: ffffffff
2022-02-23 11:58:11.490415: I tensorflow/stream_executor/stream.cc:4976] [stream=000001E1AF327560,impl=000001E1A391B2F0] did not memset GPU location; source: 000000B3A760C058; size: 8388608; pattern: ffffffff
2022-02-23 11:58:11.488390: I tensorflow/stream_executor/stream.cc:4976] [stream=000001E1AF327560,impl=000001E1A391B2F0] did not memset GPU location; source: 000000B3A7DCBD08; size: 8388608; pattern: ffffffff
2022-02-23 11:58:11.493417: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at conv_ops.cc:1006 : Not found: No algorithm worked!
2022-02-23 11:58:11.496159: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at conv_ops.cc:1006 : Not found: No algorithm worked!
2022-02-23 11:58:11.497784: I tensorflow/stream_executor/stream.cc:4976] [stream=000001E1AF327560,impl=000001E1A391B2F0] did not memset GPU location; source: 000000B3A760C038; size: 8388608; pattern: ffffffff
2022-02-23 11:58:11.503459: I tensorflow/stream_executor/stream.cc:4976] [stream=000001E1AF327560,impl=000001E1A391B2F0] did not memset GPU location; source: 000000B3A760C058; size: 8388608; pattern: ffffffff
2022-02-23 11:58:11.505625: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at conv_ops.cc:1006 : Not found: No algorithm worked!
2022-02-23 11:58:11.844846: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at conv_ops.cc:1006 : Not found: No algorithm worked!
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
return fn(*args)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
target_list, run_metadata)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(786432, 3), b.shape=(3, 3), m=786432, n=3, k=3
[[{{node Tensordot/MatMul}}]]
[[mul_10/_893]]
(1) Internal: Blas GEMM launch failed : a.shape=(786432, 3), b.shape=(3, 3), m=786432, n=3, k=3
[[{{node Tensordot/MatMul}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 101, in
main()
File "train.py", line 96, in main
gan.train()
File "E:\AnimeGANv2-master\AnimeGANv2.py", line 248, in train
self.Generator_loss, self.G_loss_merge], feed_dict = train_feed_dict)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
run_metadata_ptr)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
run_metadata)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(786432, 3), b.shape=(3, 3), m=786432, n=3, k=3
[[node Tensordot/MatMul (defined at C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
[[mul_10/_893]]
(1) Internal: Blas GEMM launch failed : a.shape=(786432, 3), b.shape=(3, 3), m=786432, n=3, k=3
[[node Tensordot/MatMul (defined at C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'Tensordot/MatMul':
File "train.py", line 101, in
main()
File "train.py", line 91, in main
gan.build_model()
File "E:\AnimeGANv2-master\AnimeGANv2.py", line 155, in build_model
t_loss = self.con_weight * c_loss + self.sty_weight * s_loss + color_loss(self.real,self.generated) * self.color_weight + tv_loss
File "E:\AnimeGANv2-master\tools\ops.py", line 280, in color_loss
con = rgb2yuv(con)
File "E:\AnimeGANv2-master\tools\ops.py", line 309, in rgb2yuv
return tf.image.rgb_to_yuv(rgb)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\ops\image_ops_impl.py", line 2930, in rgb_to_yuv
return math_ops.tensordot(images, kernel, axes=[[ndims - 1], [0]])
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\ops\math_ops.py", line 4071, in tensordot
ab_matmul = matmul(a_reshape, b_reshape)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\util\dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\ops\math_ops.py", line 2754, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\ops\gen_math_ops.py", line 6136, in mat_mul
name=name)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()