Tensorflow Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS

  Server environment:

    Ubuntu 16.04.4tensorflow 1.13.1cuda-10.0cudnn 7.4.5

Recently, when I was running demo pointasnl of point cloud classification, when batch_ When the size setting is relatively large, the following errors will appear during the training:

2020-06-12 00:14:01.824110: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-06-12 00:14:01.824142: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1

At first, it was thought that there was something wrong with the GPU Programming code, but after repeated checking, it was found that there was no error.

After collecting information from the Internet, I vaguely realized that it should be the environmental version.

After reducing cudnn 7.4.5 to cudnn 7.3.1, this problem seems to be solved. I hope there will be no more problems.

Read More: