The method used in this paper
The versions of pytorch, cudatoolkit and CUDA driver should be consistent
Problem description
When training the stylegan3 model with multi GPU:
python train.py --outdir=training-runs --cfg=stylegan3-r \
--data=datastes/your_data.zip \
--cfg=stylegan3-r --gpus=4 --batch=32 --gamma=8 --kimg=1800 --snap=50 --tick=2
Error Messages:
torch.multiprocessing.spawn.ProcessRaisedException:
……
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1631630841592/work/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled cuda error, NCCL version 2.7.8
ncclUnhandledCudaError: Call to CUDA function failed.
![]()
Local Environment
4xTeslaV100 graphics card drivers and CUDA version 11.0
stylegan3 Default Environment
Solution:
Go to the pytorch official website and search the corresponding version of Cudatookit
conda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=11.0 -c pytorch
Tried Method:
Method 1: install nccl (this article is useless)
Method 2: the versions of pytorch, CUDA toolkit and CUDA driver are the same
https://github.com/ultralytics/yolov5/issues/4530
Read More:
- [Solved] CUDA error:-UserWarning: CUDA initialization: CUDA unknown error
- [Solved] UserWarning: CUDA initialization: CUDA unknown error
- Cuda Runtime error (38) : no CUDA-capable device is detected
- unhandled system error, NCCL version 2.7.8 [How to Solve]
- [Solved] Pytorch loading model specified GPU card number error or failed to specify
- Pytorch failed to specify GPU resolution
- [Solved] RuntimeError: CUDA error: out of memory
- Vitis-AI Generate a Quantitative Model: NotImplementedError
- [Solved] RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
- [Solved] volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
- [Solved] import mxnet Error: OSError: libcudart.so.8.0: cannot open shared object file: No such file or directory
- [Solved] NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA drive
- How to Solve error: command ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe‘ failed
- [Samtools] Run error: error while loading shared libraries: libcrypto.so.1.0.0 or libncurses.so.5 or libtinfow.so.5
- CUDA_ERROR_SYSTEM_DRIVER_MISMATCH [How to Solve]
- Tensorflow GPU error (4 Type Error and their Solutions)
- [Solved] removeerror: ‘requests’ is a dependency of CONDA and cannot be removed from
- [Solved] paddle:FatalError: `Segmentation fault` is detected by the operating system.
- [Solved] jetson Compile pytorch Error: internal compiler error: Segmentation fault
- Pytorch error: `module ‘torch‘ has no attribute ‘__version___‘`