[Solved] MindSpore Error: “GeneratorDataset’s num_workers=8, this value is …”

1 Error description

1.1 System Environment

Hardware Environment(Ascend/GPU/CPU): CPU
Software Environment:
– MindSpore version (source or binary): 1.2.0
– Python version (eg, Python 3.7.5): 3.7.5
– OS platform and distribution (eg, Linux Ubuntu 16.04): Ubuntu 4.15.0-74-generic
– GCC/Compiler version (if compiled from source):

1.2 Basic information

1.2.1 Script

This case runs the linear function fitting example on the official website, and mindspore has been successfully installed before.

1.2.2 Error reporting

Error message: RuntimeError: Thread ID 140706176251712 Unexpected error. GeneratorDataset’s num_workers=8, this value is not within the required range of [1, cpu_thread_cnt=2].
Line of code : 639
File : /home/jenkins/agent-working-dir /workspace/Compile_CPU_X86_Ubuntu/mindspore/mindspore/ccsrc/minddata/dataset/engine/ir/datasetops/dataset_node.cc

2 Reason analysis

The number of CPU cores when the user is running is less than the number of cores used by default when the dataset module generates data. Mindspore does not perform adaptive configuration for the number of CPU cores in the hardware in 1.2.0. It is required when the configuration of the PC is not high. Manually configure the number of CPU cores.

3 Solutions

1. Add code to manually configure the number of CPU cores:
2. Use a higher version of mindspore, the current mindspore-1.6.0 will be adaptively configured according to the number of CPU cores in the hardware to avoid the occurrence of CPU cores If the number is too low, an error will be reported.

4 Summary

1. You can locate the problem according to the prompt of the error message. In this case, it is a problem of the number of CPU cores. You can search for the method of setting the number of CPU cores in the official website tutorial and the open source MindSpore documentation.
2. At present, MindSpore provides an automatic data tuning tool – Dataset AutoTune, which is used to automatically adjust the parallelism of the data processing pipeline according to the environment resources during the training process. During this process, the CPU cores in the hardware will be automatically detected. The number of adaptive configuration.
3. The config module in MindSpore can set or obtain the global configuration parameters of data processing.

Read More:

Leave a Reply

Your email address will not be published. Required fields are marked *