RuntimeError: CUDA out of memory. Tried to allocate 600.00 MiB (GPU 0; 23.69 GiB total capacity; 21.82 GiB already allocated; 115.25 MiB free; 21.87 GiB reserved in total by PyTorch)

Runtime error: CUDA out of memory. Attempt to allocate 600.00 MIB (GPU 0; 23.69 gib total capacity; 21.82 gib allocated; 115.25 MIB free; pytorch reserves a total of 21.87 GIB)

reason

A similar bug is due to insufficient video memory

Solution 1: release the video memory

First fuser - V/dev/NVIDIA * or sudo fuser - V/dev/NVIDIA * , view the processes running on the GPU recently, and then sudo kill the relevant process number.

Of which:

Fuser: it can display which program is currently using a file, mount point, or even network port on the disk, and give the details of the program process – V: detailed mode/dev/NVIDIA *: all NVIDIA related interfaces (such as GPU)

Solution 2: turn down the batch size

If it is not enough after releasing part of the video memory, you can reduce the batch size

Reference:
after NVIDIA GPU kill process, the video memory is still unclear