pytorch: RuntimeError CUDA error device-side assert triggered

Training network error reporting: RuntimeError: cuda runtime error (710) : Device – side assert triggered the at/pytorch aten/SRC/THC/generic/THCTensorScatterGather cu: 380
the terminate called after throwing an instance of ‘c10: : Error’
I () : CUDA Error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:569)
Reason: The label is out of line
Method: Input

CUDA_LAUNCH_BLOCKING=1 python train.py

An error generates specific information

/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [72,0,0], thread: [32,0,0] Assertion `indexValue >= 0 && indexValue < tensor.sizes[dim]` failed.

Can be seen to be Assertion ‘indexValue & GT; = 0 & & indexValue < The predicate error is tensor. Sizes [dim], which means the label is more than zero or more than the total number and crosses the line. After debugging, I found that there was a setting that was larger than the preset total number of categories when the category was labeled. I modified this label, and the problem was solved.


Read More: