problem description
when training MNIST data set with tensorflow-gpu version, error:
InternalError: Blas SGEMM launch failed : a.shape=(100, 784), b.shape=(784, 10), m=100, n=10, k=784
[[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](_recv_Placeholder_0/_4, Variable/read)]]
cause
(1) because other pythonx programs use GPU resources, existing programs cannot allocate enough resources to execute the current program.
(2). If you are using GPU TensorFlow, and you want to train the model under high graphics card usage (such as playing a game), you should be careful to allocate a fixed amount of video memory when initializing the Session, otherwise you may report an error and exit directly at the beginning of the training.
solution
(1) : determines the current Session()
if 'session' in locals() and session is not None:
print('Close interactive session')
session.close()
(2) : assigns video memory
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))
(3) : if the first two methods do not solve the problem
restart the machine
Read More:
- TensorFlow: InternalError: Blas SGEMM launch failed
- tensorflow.python.framework.errors_impl.InternalError: Blas xGEMM launch failed
- python MemoryError: Unable to allocate 165. MiB for an array with shape
- AttributeError: ‘NoneType‘ object has no attribute ‘shape‘
- Copying a param with shape torch. Size ([262, 2048]), parameter size does not match
- The method of constructing even order magic square (n = 4 * m)
- tensorflow.python.framework.errors_impl.InternalError: Failed to create session.
- InternalError: Failed to create session. Error and solution
- “Failed to load session” Ubuntu “problem solving summary
- Solve CONDA’s “solving environment: failed” problem
- Remove M Characters “Control M” or “^M” or “CTRL M” Characters in a UNIX file
- The underlying provider failed on open problem solving
- mysql problem solving: mysqladmin: connect to server at’localhost’ failed
- VirtualBox problem solving set -[drm:vmw_host_log [vmwgfx]] *ERROR* Failed to send host log message
- Solving environment: failed solution to the problem encountered when updating Anaconda
- SQL Server “login failed for user ‘domain account”. [sqlstate 28000] (error 18456). “Problem solving
- Solve the problem that M1 Mac can’t debug when using GoLand
- Problem solving of failed to read candidate component class in Java
- Failed to load resource: net::ERR_ INSECURE_ Response problem solving record
- Yarn: runtime.ContainerExecutionException : launch container failed