in my last blog, I mentioned that when I used multithreading + coroutine crawler to grab data, when my coroutine number × thread number was very large, it would prompt [Error 24: too many open files] and a series of other errors. This blog is a solution to this problem.
report this error?You only have a few files open?
this should be the first reaction of most people. When I met this error was also very meng, then wanted to think, it should be me every collaborators process to obtain access to the file handle, so even though you only took up to save data to a file, but when the program runs, there may be many threads or processes have the file handle, the program looks like opened a lot of files.
so the only way to deal with this situation is to change the system default maximum number of files.
ubuntu, you can enter
ulimit-n in the terminal to see the current system default maximum number of files, which is usually 1024 by default. Obviously, this value is small for the crawler.
there are two ways to change this default value. Both methods have worked on different machines. The first is simple, but it doesn’t guarantee success. If the first fails, try the second, and you’ll basically solve the problem.
the first method:
directly in the terminal input:
ulimit -n 10000
I tried this on my own computer without success, only on the server.
the second method:
input in terminal sudo vim/etc/security/limits the conf code> to open the file, to the end of the file, the keyboard I insert content: p>
* soft nofile 10000 * hard nofile 10000
after the above two lines have been written, press Esc to exit edit mode, then press shift and at the same time; Key, type
wq! save the file and exit. Then log out and log in again. If you put
ulimit -n in the terminal you will get the value you just set.
Windows has not tried, recently rarely logged in Windows, Windows do not know whether there will be such a problem, if encountered, the principle should be the same, according to this idea to find the answer should be able to solve.
above. Welcome to exchange.
- Solution to UBI partman failed with code 141 when installing Ubuntu
- Python – [encoding] in Python os.system Solution to Chinese garbled code when calling CMD command
- The solution to the error of importing CSV from Python to Python
- Solution to “error: invalid environment block” when Ubuntu starts up
- Solution to x service error when installing NVIDIA graphics driver under Ubuntu
- Solution to “the grub PC ‘package failed to install into / target / when installing Ubuntu server
- Ubuntu network configuration+Fixing “Failed to bring up eth0” in Ubuntu
- Ubuntu xshell cannot connect to VMware virtual machine solution
- Ubuntu 18.04/ubuntu 16.04 (Ubuntu kylin 18 / 16), the last grub installation failed
- Opencv2.4.9 + ffmpeg1.2.12 installation configuration and problem solution under Ubuntu 14.04
- [solution] a perfect solution to the problem of failed to load module “Canberra GTK module” in Ubuntu
- launch failed.Binary Not found in Linux / Ubuntu solution
- There is no windows solution in dual system Ubuntu grub startup
- There is a solution to the problem: severity = corrected, type = physical layer, id = 00e5 or id = 00e8 (receiver ID) under Ubuntu
- How to manually upgrade Ubuntu 16.04 LTS to Ubuntu 18.04 LTS, Part I
- Solution to abnormal errors during startup of Python’s Anaconda
- Ubuntu 16.04 Chinese installation tutorial (Graphic), Ubuntu 16.04
- Ubuntu18 “Activation of network connection failed” cannot access the Internet 【Solution】
- Solution: the network can be recovered only when the Ubuntu broadband is disconnected
- Ubuntu add apt repository command not found solution