in my last blog, I mentioned that when I used multithreading + coroutine crawler to grab data, when my coroutine number × thread number was very large, it would prompt [Error 24: too many open files] and a series of other errors. This blog is a solution to this problem.

Why does

report this error?You only have a few files open?
this should be the first reaction of most people. When I met this error was also very meng, then wanted to think, it should be me every collaborators process to obtain access to the file handle, so even though you only took up to save data to a file, but when the program runs, there may be many threads or processes have the file handle, the program looks like opened a lot of files.
so the only way to deal with this situation is to change the system default maximum number of files.

Under

ubuntu, you can enter ulimit-n in the terminal to see the current system default maximum number of files, which is usually 1024 by default. Obviously, this value is small for the crawler.
there are two ways to change this default value. Both methods have worked on different machines. The first is simple, but it doesn’t guarantee success. If the first fails, try the second, and you’ll basically solve the problem.

the first method:
directly in the terminal input: ulimit -n 10000
I tried this on my own computer without success, only on the server.

the second method:
input in terminal sudo vim/etc/security/limits the conf to open the file, to the end of the file, the keyboard I insert content:

* soft nofile 10000
* hard nofile 10000