problem description

when training MNIST data set with tensorflow-gpu version, error:

  InternalError: Blas SGEMM launch failed : a.shape=(100, 784), b.shape=(784, 10), m=100, n=10, k=784
     [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](_recv_Placeholder_0/_4, Variable/read)]]  

cause

(1) because other pythonx programs use GPU resources, existing programs cannot allocate enough resources to execute the current program.
(2). If you are using GPU TensorFlow, and you want to train the model under high graphics card usage (such as playing a game), you should be careful to allocate a fixed amount of video memory when initializing the Session, otherwise you may report an error and exit directly at the beginning of the training.

solution

(1) : determines the current Session()

if 'session' in locals() and session is not None:
    print('Close interactive session')
    session.close()

(2) : assigns video memory

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

(3) : if the first two methods do not solve the problem
restart the machine