I have just changed my laptop, and I have been reporting errors when training models with TensorFlow-gpu, so I am writing down my solution here. Because I tried to replace the cuda version, replace the cudnn version, replace the tensorflow-gpu and keras version when solving the gpu can not run, so the reported errors are also a mess.
Error 1:
2021-08-09 21:04:53.637764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2021-08-09 21:04:58.598447: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-08-09 21:17:47.603456: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows
Relying on driver to perform ptx compilation. This message will be only logged once.
2021-08-09 21:17:47.675868: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
2021-08-09 21:17:47.676730: I tensorflow/stream_executor/stream.cc:4963] [stream=000001774007A1F0,impl=00000177393F7250] did not memzero GPU location; source: 000000726209DF28
2021-08-09 21:17:47.676867: I tensorflow/stream_executor/stream.cc:316] did not allocate timer: 000000726209DED0
2021-08-09 21:17:47.676954: I tensorflow/stream_executor/stream.cc:1964] [stream=000001774007A1F0,impl=00000177393F7250] did not enqueue ‘start timer’: 000000726209DED0
2021-08-09 21:17:47.677084: I tensorflow/stream_executor/stream.cc:1976] [stream=000001774007A1F0,impl=00000177393F7250] did not enqueue ‘stop timer’: 000000726209DED0
2021-08-09 21:17:47.677201: F tensorflow/stream_executor/gpu/gpu_timer.cc:65] Check failed: start_event_ != nullptr && stop_event_ != nullptr
2021-08-09 21:04:53.637764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2021-08-09 21:04:58.598447: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-08-09 21:17:47.603456: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows
Relying on driver to perform ptx compilation. This message will be only logged once.
2021-08-09 21:17:47.675868: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
2021-08-09 21:17:47.676730: I tensorflow/stream_executor/stream.cc:4963] [stream=000001774007A1F0,impl=00000177393F7250] did not memzero GPU location; source: 000000726209DF28
2021-08-09 21:17:47.676867: I tensorflow/stream_executor/stream.cc:316] did not allocate timer: 000000726209DED0
2021-08-09 21:17:47.676954: I tensorflow/stream_executor/stream.cc:1964] [stream=000001774007A1F0,impl=00000177393F7250] did not enqueue ‘start timer’: 000000726209DED0
2021-08-09 21:17:47.677084: I tensorflow/stream_executor/stream.cc:1976] [stream=000001774007A1F0,impl=00000177393F7250] did not enqueue ‘stop timer’: 000000726209DED0
2021-08-09 21:17:47.677201: F tensorflow/stream_executor/gpu/gpu_timer.cc:65] Check failed: start_event_ != nullptr && stop_event_ != nullptr
Error 2:
Error 3:
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
Error 4:
CuDNN library: 7.4.1 but source was compiled with: 7.6.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration
In fact, these errors are caused by a problem: my computer graphics card is 3050ti, belongs to the 30 series, can only install cuda11 version or higher, so I reinstalled cuda11.3.1 and the corresponding cudnn8.2.0 version (cudnn8.2.1 is reported as an error)
CuDNN library: 7.4.1 but source was compiled with: 7.6.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration
In fact, these errors are caused by a problem: my computer graphics card is 3050ti, belongs to the 30 series, can only install cuda11 version or higher, so I reinstalled cuda11.3.1 and the corresponding cudnn8.2.0 version (cudnn8.2.1 is reported as an error)
Read More:
- Error using tensorflow GPU: could not create cudnn handle: cudnn_STATUS_NOT_INITIALIZED
- [Solved] MindSpore Error: Select GPU kernel op * fail! Incompatible data type
- TensorFlow issue: Expected int32, got list containing Tensors of type ‘_Message’ instead.
- [Solved] Tensorflow-gpu 2.0.0 Run Error: ModuleNotFoundError: No module named ‘tensorflow_core.estimator‘
- Opencv Can not Find opencv2/gpu/gpu.hpp [How to Solve]
- [Solved] Tensorflow Error: Failed to load the native TensorFlow runtime.
- Tensorflow error: attributeerror: module ‘tensorflow_ API. V1. Train ‘has no attribute’ summarywriter ‘
- TensorFlow error: AttributeError: module ‘tensorflow_core._api.v2.train’ has no attribute ‘Optimizer‘
- [Solved] Tensorflow Error: failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
- Tensorflow Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS
- Internalerror: GPU sync failed error (How to Solve)
- Pytorch failed to specify GPU resolution
- [Solved] Tensorflow Win10: ImportError: DLL load failed
- [Solved] Pytorch loading model specified GPU card number error or failed to specify
- [Solved] TensorFlow severing Container Creat Error: failed: Out of range: Read less bytes than requested
- How to Solve error: command ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe‘ failed
- Error 1 error MSB3721: Command ““C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\nvcc.exe” -genco
- [Solved] Pytorch3d Error: RuntimeError: Not compiled with GPU support.
- [Solved] The method getContextPath() from the type HttpServletRequest refers to the missing type String
- [ubuntu14.04]Tensorflow: Could not find any downloads that satisfy the requirement