Recently, Python is used to process the public image database. Due to the large amount of data, it takes too long to process the images one by one in serial. Therefore, it is decided to adopt the parallel mode to make full use of the CPU on the host to speed up the processing process, which can greatly reduce the total processing time.
Here we use the concurrent.futures Module, which can use multiprocessing to achieve real parallel computing.
The core principle is: concurrent.futures It will run multiple Python interpreters in parallel in the form of sub processes, so that Python programs can use multi-core CPU to improve the execution speed. Since the subprocess is separated from the main interpreter, their global interpreter locks are also independent of each other. Each subprocess can use a CPU core completely.
The specific implementation is also very simple, the code is as follows. How many CPU cores the host has will start how many Python processes for parallel processing.
import concurrent.futures def function(files): # To do what you want # files: file list that you want to process if __name__ == '__main__': with concurrent.futures.ProcessPoolExecutor() as executor: executor.map(function, files)
After changing to parallel processing, my 12 CPUs run at full load, and the processing speed is significantly faster.
- How to solve oserror: [errno 98] address already in use and kill the python3 process
- The method of getting shell command output in Python
- PHP in Windows combined with bat batch processing to achieve multi process verification proxy server function
- Python packaged *. Exe is running os.popen (cmd)/ subprocess.Popen *. Exe crashes with invalid [winerror 6] handle
- Use subprocess to execute the command line, and the pipeline is blocked
- [How to Fix] dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
- Apache [error] server reached MaxClients setting, consider raising the MaxClients setting
- Several calculation methods of Python execution time
- Comparison of ITK and VTK
- How to Fix Error in python3 | PIP install Dlib
- Windows git review reported an error. UnicodeDecodeError
- Unable to call numpy in pychar, module notfounderror: no module named ‘numpy’
- Detailed explanation of UART, SPI and IIC and their differences and relations
- tomcat Issues org.apache.catalina.LifecycleException: Failed to start component
- onnx-tensorrt/builtin_op_importers.cpp:766:12: error: ‘class nvinfer1::IDeconvolutionLayer’ has no m
- Failed to start component [standardengine [Catalina]. Standardhost [localhost]. Standardcontext
- [SSL: CERTIFICATE_VERIFY_FAILED] Certificate Verify Failed (_ssl.C:579)
- A child container failed during start Tomcat error prompt
- Learning notes of Python 3: debugger speedups using Python not found
- Caused by: org.apache.catalina.LifecycleException: A child container failed during start