[Python/Pytorch – Bug] –UnknownError (see above for traceback): Failed to get convolution algorithm.
Question
Problem: TensorFlow reports an error: unknown error (see above for traceback): failed to get revolution algorithm
analysis
Analysis: the reason is that the memory of the graphics card is not enough. Selecting the appropriate memory of the graphics card can solve the problem.
Solution:
1. Gpustat checks the usage of the graphics card
2. Select a graphics card with enough memory;
Error code:
import tensorflow as tf
net = layers.Dense(10)
net.build((4, 10))
net.kernel
NameError: name ‘layers’ is not defined
Error reason: TensorFlow does not load layers
Solution:
import tensorflow as tf
from tensorflow.keras import datasets, layers,optimizers
net = layers.Dense(10)
net.build((4, 10))
net.kernel
Operation results:
<tf.Variable 'kernel:0' shape=(10, 10) dtype=float32, numpy=
array([[ 0.22973484, 0.00857711, -0.21515384, -0.5346802 , -0.2584985 ,
0.03767496, 0.22262502, 0.10832614, 0.12043941, 0.3197981 ],
[ 0.12034583, 0.01719284, -0.37415868, 0.22801459, 0.49012756,
-0.01656079, -0.02581853, 0.22888458, -0.3193212 , -0.23586014],
[-0.50331104, -0.18943703, 0.47028244, -0.33412236, 0.04251152,
-0.54133296, 0.23136115, 0.02571291, -0.36819634, 0.5134926 ],
[-0.06907243, 0.33713734, 0.34277046, 0.24761981, 0.50419617,
-0.20183799, -0.27459818, -0.34057558, -0.23564544, 0.34107167],
[-0.51874346, 0.30625004, 0.07017416, 0.4792788 , -0.08462432,
0.1762883 , 0.47576356, -0.08242992, 0.0560475 , 0.5385151 ],
[-0.02134383, 0.02438915, -0.11708987, 0.26330394, -0.4951692 ,
0.19778156, -0.1931901 , -0.41975048, 0.0376184 , 0.23603398],
[-0.20051709, -0.46164495, 0.15974921, -0.05227134, 0.14756906,
0.12185448, -0.5285519 , -0.5298273 , 0.14063555, 0.02481627],
[-0.35953748, 0.30639488, -0.02970898, -0.5232449 , -0.10309196,
-0.3557127 , -0.19765031, 0.3171267 , 0.34930962, -0.15071085],
[ 0.20013565, 0.11569405, -0.46884173, -0.40876222, 0.36319625,
0.33609563, 0.2721032 , -0.04006624, 0.09699225, 0.20260221],
[-0.03152204, -0.48894358, 0.3079273 , -0.5283493 , -0.44822672,
-0.34838638, 0.41896552, -0.34962398, -0.24334553, 0.38500214]],
dtype=float32)>
Problem solved.
Copy the placeholder this way:
input_placeholder = tf.placeholder(tf.int32,
[batch_size, sequence_len],
name="input")
copy_input = tf.Variable(initial_value=input_placeholder,
trainable=False)
Then this sentence reports an error:
sess.run(tf.global_variables_initializer())
Solution:
copy the placeholder in this way:
copy_input = tf.get_variable(
initializer=tf.constant(0, shape=[batch_size, sequence_len]),
name="copy_placeholder",
dtype=tf.int32, trainable=False)
copy_input.assign(input_placeholder)
the_tensor = tf.get_variable(
initializer=tf.constant(0, shape=[batch_size, sequence_len]),
name="a_name",
dtype=tf.int32)
This assignment:
the_tensor[0][0].assign(0)
If the above error is reported,
it should:
the_tensor[0, 0].assign(0)
Server environment:
Recently, when I was running demo pointasnl of point cloud classification, when batch_ When the size setting is relatively large, the following errors will appear during the training:
2020-06-12 00:14:01.824110: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-06-12 00:14:01.824142: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1At first, it was thought that there was something wrong with the GPU Programming code, but after repeated checking, it was found that there was no error.
After collecting information from the Internet, I vaguely realized that it should be the environmental version.
After reducing cudnn 7.4.5 to cudnn 7.3.1, this problem seems to be solved. I hope there will be no more problems.
1. Confirm whether tensorflow corresponds to CUDA and cudnn. Check here. 2. Modify settings
os.environ['CUDA_VISIBLE_DEVICES']='0,1'
# tf 1.13
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)
sess.run(tf.global_variables_initializer())
# tf 2.0
# gpu = tf.config.experimental.list_physical_devices('GPU')
# tf.config.experimental.set_memory_growth(gpu[0], True)
# gpu = tf.config.experimental.list_physical_devices('GPU')
# tf.config.experimental.set_memory_growth(gpu[1], True)
It is tf.nn.embedding_ The sentence "look up reports an error, and the sentence” look up reports an error_ Value is out of bounds,
is data_ Value is not in [0, Max]_ embedding_ Size) is in the range.
Tensorflow error:
AttributeError: module ‘tensorflow_core._api.v2.train’ has no attribute 'Optimizer‘
The solution:
tf.train.Optimizer
Change to:
tf.compat.v1.train.Optimizer
success
import tensorflow as tf
a = tf.constant(tf.random_normal([2, 2])) # Wrong
print(a)
An error occurred when passing tf.random_normal to tf.constant:
TypeError: List of Tensors when single Tensor expected
See how tf.constant() and tf.random_normal() are used
def random_normal(shape,
mean=0.0,
stddev=1.0,
dtype=dtypes.float32,
seed=None,
name=None):
def constant(value, dtype=None, shape=None, name="Const", verify_shape=False)
dtype.Conclusion: Because the return value of tf.random_normal is a Tensor, but the parameter passed in by tf.constat is that the two types of the list do not match, an error occurs.
Use NumPy to generate the random value and put it in a tf.constant()
some_test = tf.constant(
np.random.normal(loc=0.0, scale=1.0, size=(2, 2)).astype(np.float32))
Potentially faster, as it can use the GPU to generate the random numbers,Use TensorFlow to generate the random value and put it in a tf.Variable
some_test = tf.Variable(
tf.random_normal([2, 2], mean=0.0, stddev=1.0, dtype=tf.float32)
sess.run(some_test.initializer)
View the definition of the tf.Variable function:
def __init__(self,
initial_value=None,
trainable=True,
collections=None,
validate_shape=True,
caching_device=None,
name=None,
variable_def=None,
dtype=None,
expected_shape=None,
import_scope=None,
constraint=None):
...
)
args: initial_value: A Tensor, or Python object convertible to a Tensor,which is the initial value for the Variable.
Next time you encounter this kind of function parameter problem, you must check how the function is used, and what the formal parameters and return values are.