When running deep learning programs, sometimes the program is forced to terminate, but the GPU resources occupied by the program are still not released. After being trapped for a long time, it is thought that the GPU has been occupied by others. As a result, the GPU resources are leaked.

You can use this command to view the usage of GPU in Linux system

nvidia-smi

The result is as shown in the figure

At this time, you can manually kill the process that occupies the GPU to release the GPU resources

kill -9 49461

If the screen command is used, the program running in the background stops and occupies the GPU, you can also close all screen windows to release the GPU

killall screen

Of course, it’s OK to kill the process directly