The method used in this paper

The versions of pytorch, cudatoolkit and CUDA driver should be consistent

Problem description

When training the stylegan3 model with multi GPU:

python train.py --outdir=training-runs --cfg=stylegan3-r \
--data=datastes/your_data.zip \
--cfg=stylegan3-r --gpus=4 --batch=32 --gamma=8 --kimg=1800 --snap=50  --tick=2  

Error Messages:

torch.multiprocessing.spawn.ProcessRaisedException:
……
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1631630841592/work/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled cuda error, NCCL version 2.7.8
ncclUnhandledCudaError: Call to CUDA function failed.

Local Environment
4xTeslaV100 graphics card drivers and CUDA version 11.0

stylegan3 Default Environment

Solution:
Go to the pytorch official website and search the corresponding version of  Cudatookit

conda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=11.0 -c pytorch

Tried Method:

Method 1: install nccl (this article is useless)

Method 2: the versions of pytorch, CUDA toolkit and CUDA driver are the same

https://github.com/ultralytics/yolov5/issues/4530

Read More:

• [Solved] CUDA error：-UserWarning: CUDA initialization: CUDA unknown error
• [Solved] UserWarning: CUDA initialization: CUDA unknown error
• Cuda Runtime error (38) : no CUDA-capable device is detected
• unhandled system error, NCCL version 2.7.8 [How to Solve]
• [Solved] Pytorch loading model specified GPU card number error or failed to specify
• Pytorch failed to specify GPU resolution
• [Solved] RuntimeError: CUDA error: out of memory
• Vitis-AI Generate a Quantitative Model: NotImplementedError
• [Solved] RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
• [Solved] volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
• [Solved] import mxnet Error: OSError: libcudart.so.8.0: cannot open shared object file: No such file or directory
• [Solved] NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA drive
• How to Solve error: command ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe‘ failed
• [Samtools] Run error: error while loading shared libraries: libcrypto.so.1.0.0 or libncurses.so.5 or libtinfow.so.5
• CUDA_ERROR_SYSTEM_DRIVER_MISMATCH [How to Solve]
• Tensorflow GPU error (4 Type Error and their Solutions)
• [Solved] removeerror: ‘requests’ is a dependency of CONDA and cannot be removed from
• [Solved] paddle:FatalError: `Segmentation fault` is detected by the operating system.
• [Solved] jetson Compile pytorch Error: internal compiler error: Segmentation fault
• Pytorch error: `module ‘torch‘ has no attribute ‘__version___‘`