Python — a solution to [error 24: too many open files] under Ubuntu

in my last blog, I mentioned that when I used multithreading + coroutine crawler to grab data, when my coroutine number × thread number was very large, it would prompt [Error 24: too many open files] and a series of other errors. This blog is a solution to this problem.

Why does

report this error?You only have a few files open?
this should be the first reaction of most people. When I met this error was also very meng, then wanted to think, it should be me every collaborators process to obtain access to the file handle, so even though you only took up to save data to a file, but when the program runs, there may be many threads or processes have the file handle, the program looks like opened a lot of files.
so the only way to deal with this situation is to change the system default maximum number of files.


ubuntu, you can enter ulimit-n in the terminal to see the current system default maximum number of files, which is usually 1024 by default. Obviously, this value is small for the crawler.
there are two ways to change this default value. Both methods have worked on different machines. The first is simple, but it doesn’t guarantee success. If the first fails, try the second, and you’ll basically solve the problem.

the first method:
directly in the terminal input: ulimit -n 10000
I tried this on my own computer without success, only on the server.

the second method:
input in terminal sudo vim/etc/security/limits the conf to open the file, to the end of the file, the keyboard I insert content:

* soft nofile 10000
* hard nofile 10000

after the above two lines have been written, press Esc to exit edit mode, then press shift and at the same time; Key, type wq! save the file and exit. Then log out and log in again. If you put ulimit -n in the terminal you will get the value you just set.
Windows has not tried, recently rarely logged in Windows, Windows do not know whether there will be such a problem, if encountered, the principle should be the same, according to this idea to find the answer should be able to solve.

above. Welcome to exchange.

Read More: