Tag Archives: scrapy

gerapy Run scrapy Program report timeout error [How to Solve]

Recently I’ve been developing projects using scrapy.
scrapy project, using a proxy.
Then it runs locally, everything is fine, the data can be crawled normally.
After deploying it to the online gerapy and running it, it reports an error and the logs show that:

2022-10-08 17:03:24 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.xxx.com/en/product?o=100&p=100> (failed 3 times): User timeout caused connection failure: Getting https://www.xxx.com/en/product?o=100&p=100 took longer than 180.0 seconds..

2022-10-08 17:03:24 [xxx] ERROR: <twisted.python.failure.Failure twisted.internet.error.TimeoutError: User timeout caused connection failure: Getting https://www.xxx.com/en/product?o=100&p=100 took longer than 180.0 seconds..>

2022-10-08 17:03:24 [xxx] ERROR: TimeoutError on https://www.xxx.com/en/product?o=100&p=100

I thought it was a proxy problem, then I removed the proxy and it worked fine locally
After deploying to the online gerapy and running, the error was reported again and the logs showed that:

2022-10-09 10:39:45 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.xxx.com/en/product?o=100&p=100> (failed 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]

2022-10-09 10:39:56 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.xxx.com/en/product?o=100&p=100> (failed 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]

Find a solution from stackoverflow:

Add a method in spider:

# Citation needed
from scrapy.spidermiddlewares.httperror import HttpError
from twisted.internet.error import DNSLookupError
from twisted.internet.error import TimeoutError


    # How to use
    yield scrapy.Request(url=url, meta={'dont_redirect': True, dont_filter=True, callback=self.parse_list, errback=self.errback_work)


    # Definition Method
    def errback_work(self, failure):
        self.logger.error(repr(failure))
        if failure.check(HttpError):
            response = failure.value.response
            self.logger.error('HttpError on %s', response.url)
        elif failure.check(DNSLookupError):
            request = failure.request
            self.logger.error('DNSLookupError on %s', request.url)
        elif failure.check(TimeoutError):
            request = failure.request
            self.logger.error('TimeoutError on %s', request.url)

After deploying it to the online gerapy and running it again, it still reported errors again.

Looking at the version of scrapy, I found that it was version 2.6.1, so I changed it to version 2.5.1.

pip3 install scrapy==2.5.1

Run it locally and report an error:

AttributeError: module ‘OpenSSL. SSL’ has no attribute ‘SSLv3_ METHOD’

Look at the version of the pyopenssl library:

pip3 show pyopenssl

I find that the version is incorrect.

Then change the library version to 22.0.0

pip3 install pyopenssl==22.0.0

Run it locally again, it is normal!

Then deploy it to online gerapy to run, and it is normal!

Pychart: How to Solve myspyder.items import myitem Error

Problem description

In items. Py , customize the item method.

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy

class MyspiderItem(scrapy.Item):
    # define the fields for your item here like:
    name = scrapy.Field()
    title = scrapy.Field()
    desc = scrapy.Field()
    RD_achievements = scrapy.Field()

We need to introduce this method into the sweep, which is the item method defined by ourselves in the sweep, but we need to reference it in the spider in the sweep, so we need to add a string of code to change our introduction method.

Error reported in pychart from myspyder.items import itcastitem :

Original import method:

import scrapy
from crawlAdvanced.myspider.myspider.items import MyspiderItem
"""
 scrapy crawl itcast --nolog
"""

from crawlAdvanced.myspider.myspider.items import MyspiderItem ModuleNotFoundError: No module named 'crawlAdvanced'

Solution 1

Change from myspider.items import myitem to from.. items import myitem

In a package, peers use . and parents use ..

Maybe the computer doesn’t know which file you’re talking about. After all, the two file names are the same（ I don’t understand. I hope the boss can correct it!)

scrapy startproject mySpider

scrapy crawl mySpider

scrapy crawl mySpider -o teachers.json

Solution 2

If the error is still reported, you can refer to the following methods: (my error is solved by using the above method)

# -*- coding: utf-8 -*-
 
#The original python method of introducing scrapy class is wrong, please reintroduce it with this method
from __future__ import absolute_import
# is the above code
 
import scrapy
 
from selenium import webdriver
from  scrapy_splash import SplashMiddleware
from scrapy_splash import SplashRequest
import pymysql
from ..item360 import Jc360Item

Use xx [‘xx ‘] = XX to set field value or does not support field: XXX

Demand:

After the tiem object is created, modify the written table of the original item_ Name value

My item:

class People(scrapy.Item):
    table_name = 'people'
    id = scrapy.Field()
    url_token = scrapy.Field()
    name = scrapy.Field()

Solution:

people = People()
people.__class__.table_name='people_20216'

Source code analysis:

If this is what the item says

table_name= scrapy.Field()

After creation, you can directly assign values in this way, which is normal operation

people = People()

people['table_name'] = 'people_20216'

Situation 1:

But if you write like me:

class People(scrapy.Item):
    table_name = 'people'

In addition, it can be modified as follows:

people[‘table_ name’] = ‘people_ 20216’

You will report an error:

Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "C:\Program Files\Python39\lib\site-packages\scrapy\utils\defer.py", line 150, in f
    return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
  File "E:\codedata\gitee\pySpace\study\npoms\npoms\pipelines.py", line 62, in process_item
    item['table_name'] = item.table_name + "_" + mysql_split_util.current_year_month
  File "C:\Program Files\Python39\lib\site-packages\scrapy\item.py", line 100, in __setitem__
    raise KeyError(f"{self.__class__.__name__} does not support field: {key}")

Corresponding to this section of the original code, the source code covers this:

Case 2:

If you write like this

people.table_ name’= ‘people_ 20216’

You will report an error:

Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "C:\Program Files\Python39\lib\site-packages\scrapy\utils\defer.py", line 150, in f
    return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
  File "E:\codedata\gitee\pySpace\study\npoms\npoms\pipelines.py", line 63, in process_item
    item.set_table_name(table_name)
  File "E:\codedata\gitee\pySpace\study\npoms\npoms\items.py", line 66, in set_table_name
    self.table_name = table_name
  File "C:\Program Files\Python39\lib\site-packages\scrapy\item.py", line 112, in __setattr__
    raise AttributeError(f"Use item[{name!r}] = {value!r} to set field value")
AttributeError: Use item['table_name'] = 'people_202106' to set field value

Corresponding to this section of the original code, the source code also has corresponding interception:

At this time, we need to look at the creation phase of the source code object

The object creation phase is divided into fields and classes, which we can use above__ setitem__ Method to modify the object in fields. So we need to get the class object and modify the values of the properties in the class.

command ‘gcc’ failed with exit status 1 error while installing scrapy

When I tried to install Scrapy I got this error:

    warning: no previously-included files found matching '*.py'
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages/pyOpenSSL-0.14-py2.7.egg (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages/cssselect-0.9.1-py2.7.egg (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/site-packages/six-1.6.1-py2.7.egg (from Scrapy)
Downloading/unpacking zope.interface>=3.6.0 (from Twisted>=10.0.0->Scrapy)
  Downloading zope.interface-4.1.1.tar.gz (864kB): 864kB downloaded
  Running setup.py (path:/tmp/pip_build_root/zope.interface/setup.py) egg_info for package zope.interface

    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
Downloading/unpacking cryptography>=0.2.1 (from pyOpenSSL->Scrapy)
  Downloading cryptography-0.4.tar.gz (260kB): 260kB downloaded
  Running setup.py (path:/tmp/pip_build_root/cryptography/setup.py) egg_info for package cryptography
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    c/_cffi_backend.c:14:17: error: ffi.h: No such file or directory
    In file included from c/_cffi_backend.c:21:
    c/malloc_closure.h:33: error: expected specifier-qualifier-list before 'ffi_closure'
    c/malloc_closure.h: In function 'more_core':
    c/malloc_closure.h:69: warning: division by zero
    c/malloc_closure.h:96: error: 'union mmaped_block' has no member named 'next'
    c/malloc_closure.h: At top level:
    c/malloc_closure.h:105: error: expected ')' before '*' token
    c/malloc_closure.h:113: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token
    c/_cffi_backend.c:187: error: expected specifier-qualifier-list before 'ffi_cif'
    c/_cffi_backend.c: In function 'cdataowninggc_dealloc':
    c/_cffi_backend.c:1499: error: 'ffi_closure' undeclared (first use in this function)
    c/_cffi_backend.c:1499: error: (Each undeclared identifier is reported only once
    c/_cffi_backend.c:1499: error: for each function it appears in.)
    c/_cffi_backend.c:1499: error: 'closure' undeclared (first use in this function)
    c/_cffi_backend.c:1499: error: expected expression before ')' token
    c/_cffi_backend.c:1502: warning: implicit declaration of function 'cffi_closure_free'
    c/_cffi_backend.c: In function 'cdataowninggc_traverse':
    c/_cffi_backend.c:1514: error: 'ffi_closure' undeclared (first use in this function)
    c/_cffi_backend.c:1514: error: 'closure' undeclared (first use in this function)
    c/_cffi_backend.c:1514: error: expected expression before ')' token
    c/_cffi_backend.c: In function 'cdataowninggc_clear':
    c/_cffi_backend.c:1530: error: 'ffi_closure' undeclared (first use in this function)
    c/_cffi_backend.c:1530: error: 'closure' undeclared (first use in this function)
    c/_cffi_backend.c:1530: error: expected expression before ')' token
    c/_cffi_backend.c: In function 'cdataowning_repr':
    c/_cffi_backend.c:1654: error: 'ffi_closure' undeclared (first use in this function)
    c/_cffi_backend.c:1654: error: expected expression before ')' token
    c/_cffi_backend.c: At top level:
    c/_cffi_backend.c:2200: error: expected declaration specifiers or '...' before 'ffi_abi'
    c/_cffi_backend.c: In function 'cdata_call':
    c/_cffi_backend.c:2334: error: 'ffi_abi' undeclared (first use in this function)
    c/_cffi_backend.c:2334: error: expected ';' before 'fabi'
    c/_cffi_backend.c:2376: error: 'fabi' undeclared (first use in this function)
    c/_cffi_backend.c:2380: error: too many arguments to function 'fb_prepare_cif'
    c/_cffi_backend.c:2385: error: 'cif_description_t' has no member named 'exchange_size'
    c/_cffi_backend.c:2395: error: 'cif_description_t' has no member named 'exchange_offset_arg'
    c/_cffi_backend.c:2425: error: 'cif_description_t' has no member named 'exchange_offset_arg'
    c/_cffi_backend.c:2430: warning: implicit declaration of function 'ffi_call'
    c/_cffi_backend.c:2430: error: 'cif_description_t' has no member named 'cif'
    c/_cffi_backend.c: In function 'b_new_primitive_type':
    c/_cffi_backend.c:3334: error: 'ffi_type' undeclared (first use in this function)
    c/_cffi_backend.c:3334: error: 'ffitype' undeclared (first use in this function)
    c/_cffi_backend.c:3355: error: 'ffi_type_sint8' undeclared (first use in this function)
    c/_cffi_backend.c:3356: error: 'ffi_type_sint16' undeclared (first use in this function)
    c/_cffi_backend.c:3357: error: 'ffi_type_sint32' undeclared (first use in this function)
    c/_cffi_backend.c:3358: error: 'ffi_type_sint64' undeclared (first use in this function)
    c/_cffi_backend.c:3364: error: 'ffi_type_float' undeclared (first use in this function)
    c/_cffi_backend.c:3366: error: 'ffi_type_double' undeclared (first use in this function)
    c/_cffi_backend.c:3368: error: 'ffi_type_longdouble' undeclared (first use in this function)
    c/_cffi_backend.c:3374: error: 'ffi_type_uint8' undeclared (first use in this function)
    c/_cffi_backend.c:3375: error: 'ffi_type_uint16' undeclared (first use in this function)
    c/_cffi_backend.c:3376: error: 'ffi_type_uint32' undeclared (first use in this function)
    c/_cffi_backend.c:3377: error: 'ffi_type_uint64' undeclared (first use in this function)
    c/_cffi_backend.c: At top level:
    c/_cffi_backend.c:3931: error: expected specifier-qualifier-list before 'ffi_type'
    c/_cffi_backend.c:3950: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token
    c/_cffi_backend.c: In function 'fb_build':
    c/_cffi_backend.c:4081: error: 'struct funcbuilder_s' has no member named 'atypes'
    c/_cffi_backend.c:4081: error: 'ffi_type' undeclared (first use in this function)
    c/_cffi_backend.c:4081: error: expected expression before ')' token
    c/_cffi_backend.c:4082: error: 'struct funcbuilder_s' has no member named 'nargs'
    c/_cffi_backend.c:4085: error: 'struct funcbuilder_s' has no member named 'rtype'
    c/_cffi_backend.c:4085: warning: implicit declaration of function 'fb_fill_type'
    c/_cffi_backend.c:4093: error: 'cif_description_t' has no member named 'exchange_offset_arg'
    c/_cffi_backend.c:4096: error: 'struct funcbuilder_s' has no member named 'rtype'
    c/_cffi_backend.c:4097: error: 'ffi_arg' undeclared (first use in this function)
    c/_cffi_backend.c:4107: error: 'atype' undeclared (first use in this function)
    c/_cffi_backend.c:4120: error: 'struct funcbuilder_s' has no member named 'atypes'
    c/_cffi_backend.c:4121: error: 'struct funcbuilder_s' has no member named 'atypes'
    c/_cffi_backend.c:4124: error: 'cif_description_t' has no member named 'exchange_offset_arg'
    c/_cffi_backend.c:4131: error: 'cif_description_t' has no member named 'exchange_size'
    c/_cffi_backend.c: In function 'fb_build_name':
    c/_cffi_backend.c:4153: error: 'struct funcbuilder_s' has no member named 'nargs'
    c/_cffi_backend.c:4162: error: 'struct funcbuilder_s' has no member named 'fct'
    c/_cffi_backend.c:4164: error: 'struct funcbuilder_s' has no member named 'fct'
    c/_cffi_backend.c: In function 'fb_prepare_ctype':
    c/_cffi_backend.c:4205: error: 'struct funcbuilder_s' has no member named 'fct'
    c/_cffi_backend.c:4215: error: 'struct funcbuilder_s' has no member named 'fct'
    c/_cffi_backend.c: At top level:
    c/_cffi_backend.c:4235: error: expected declaration specifiers or '...' before 'ffi_abi'
    c/_cffi_backend.c: In function 'fb_prepare_cif':
    c/_cffi_backend.c:4262: warning: implicit declaration of function 'ffi_prep_cif'
    c/_cffi_backend.c:4262: error: 'cif_description_t' has no member named 'cif'
    c/_cffi_backend.c:4262: error: 'fabi' undeclared (first use in this function)
    c/_cffi_backend.c:4262: error: 'struct funcbuilder_s' has no member named 'nargs'
    c/_cffi_backend.c:4263: error: 'struct funcbuilder_s' has no member named 'rtype'
    c/_cffi_backend.c:4263: error: 'struct funcbuilder_s' has no member named 'atypes'
    c/_cffi_backend.c:4263: error: 'FFI_OK' undeclared (first use in this function)
    c/_cffi_backend.c: In function 'b_new_function_type':
    c/_cffi_backend.c:4280: error: 'FFI_DEFAULT_ABI' undeclared (first use in this function)
    c/_cffi_backend.c:4318: error: too many arguments to function 'fb_prepare_cif'
    c/_cffi_backend.c:4326: error: 'struct funcbuilder_s' has no member named 'nargs'
    c/_cffi_backend.c:4336: error: 'struct funcbuilder_s' has no member named 'nargs'
    c/_cffi_backend.c: In function 'convert_from_object_fficallback':
    c/_cffi_backend.c:4360: error: 'ffi_arg' undeclared (first use in this function)
    c/_cffi_backend.c: At top level:
    c/_cffi_backend.c:4432: error: expected ')' before '*' token
    c/_cffi_backend.c: In function 'b_callback':
    c/_cffi_backend.c:4503: error: 'ffi_closure' undeclared (first use in this function)
    c/_cffi_backend.c:4503: error: 'closure' undeclared (first use in this function)
    c/_cffi_backend.c:4524: error: 'ffi_arg' undeclared (first use in this function)
    c/_cffi_backend.c:4542: warning: implicit declaration of function 'cffi_closure_alloc'
    c/_cffi_backend.c:4559: warning: implicit declaration of function 'ffi_prep_closure'
    c/_cffi_backend.c:4559: error: 'cif_description_t' has no member named 'cif'
    c/_cffi_backend.c:4560: error: 'invoke_callback' undeclared (first use in this function)
    c/_cffi_backend.c:4560: error: 'FFI_OK' undeclared (first use in this function)
    c/_cffi_backend.c: In function 'init_cffi_backend':
    c/_cffi_backend.c:5489: error: 'FFI_DEFAULT_ABI' undeclared (first use in this function)
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/tmp/pip_build_root/cryptography/setup.py", line 174, in <module>
        "test": PyTest,
      File "/usr/local/lib/python2.7/distutils/core.py", line 112, in setup
        _setup_distribution = dist = klass(attrs)
      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 260, in __init__
      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 285, in fetch_build_eggs
      File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 631, in resolve
      File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 871, in best_match
      File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 883, in obtain
      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 335, in fetch_build_egg
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 595, in easy_install
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 625, in install_item
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 822, in install_eggs
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1028, in build_and_install
      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1016, in run_setup
    distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1
    Complete output from command python setup.py egg_info:
    Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

c/_cffi_backend.c:14:17: error: ffi.h: No such file or directory

In file included from c/_cffi_backend.c:21:

c/malloc_closure.h:33: error: expected specifier-qualifier-list before 'ffi_closure'

c/malloc_closure.h: In function 'more_core':

c/malloc_closure.h:69: warning: division by zero

c/malloc_closure.h:96: error: 'union mmaped_block' has no member named 'next'

c/malloc_closure.h: At top level:

c/malloc_closure.h:105: error: expected ')' before '*' token

c/malloc_closure.h:113: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token

c/_cffi_backend.c:187: error: expected specifier-qualifier-list before 'ffi_cif'

c/_cffi_backend.c: In function 'cdataowninggc_dealloc':

c/_cffi_backend.c:1499: error: 'ffi_closure' undeclared (first use in this function)

c/_cffi_backend.c:1499: error: (Each undeclared identifier is reported only once

c/_cffi_backend.c:1499: error: for each function it appears in.)

c/_cffi_backend.c:1499: error: 'closure' undeclared (first use in this function)

c/_cffi_backend.c:1499: error: expected expression before ')' token

c/_cffi_backend.c:1502: warning: implicit declaration of function 'cffi_closure_free'

c/_cffi_backend.c: In function 'cdataowninggc_traverse':

c/_cffi_backend.c:1514: error: 'ffi_closure' undeclared (first use in this function)

c/_cffi_backend.c:1514: error: 'closure' undeclared (first use in this function)

c/_cffi_backend.c:1514: error: expected expression before ')' token

c/_cffi_backend.c: In function 'cdataowninggc_clear':

c/_cffi_backend.c:1530: error: 'ffi_closure' undeclared (first use in this function)

c/_cffi_backend.c:1530: error: 'closure' undeclared (first use in this function)

c/_cffi_backend.c:1530: error: expected expression before ')' token

c/_cffi_backend.c: In function 'cdataowning_repr':

c/_cffi_backend.c:1654: error: 'ffi_closure' undeclared (first use in this function)

c/_cffi_backend.c:1654: error: expected expression before ')' token

c/_cffi_backend.c: At top level:

c/_cffi_backend.c:2200: error: expected declaration specifiers or '...' before 'ffi_abi'

c/_cffi_backend.c: In function 'cdata_call':

c/_cffi_backend.c:2334: error: 'ffi_abi' undeclared (first use in this function)

c/_cffi_backend.c:2334: error: expected ';' before 'fabi'

c/_cffi_backend.c:2376: error: 'fabi' undeclared (first use in this function)

c/_cffi_backend.c:2380: error: too many arguments to function 'fb_prepare_cif'

c/_cffi_backend.c:2385: error: 'cif_description_t' has no member named 'exchange_size'

c/_cffi_backend.c:2395: error: 'cif_description_t' has no member named 'exchange_offset_arg'

c/_cffi_backend.c:2425: error: 'cif_description_t' has no member named 'exchange_offset_arg'

c/_cffi_backend.c:2430: warning: implicit declaration of function 'ffi_call'

c/_cffi_backend.c:2430: error: 'cif_description_t' has no member named 'cif'

c/_cffi_backend.c: In function 'b_new_primitive_type':

c/_cffi_backend.c:3334: error: 'ffi_type' undeclared (first use in this function)

c/_cffi_backend.c:3334: error: 'ffitype' undeclared (first use in this function)

c/_cffi_backend.c:3355: error: 'ffi_type_sint8' undeclared (first use in this function)

c/_cffi_backend.c:3356: error: 'ffi_type_sint16' undeclared (first use in this function)

c/_cffi_backend.c:3357: error: 'ffi_type_sint32' undeclared (first use in this function)

c/_cffi_backend.c:3358: error: 'ffi_type_sint64' undeclared (first use in this function)

c/_cffi_backend.c:3364: error: 'ffi_type_float' undeclared (first use in this function)

c/_cffi_backend.c:3366: error: 'ffi_type_double' undeclared (first use in this function)

c/_cffi_backend.c:3368: error: 'ffi_type_longdouble' undeclared (first use in this function)

c/_cffi_backend.c:3374: error: 'ffi_type_uint8' undeclared (first use in this function)

c/_cffi_backend.c:3375: error: 'ffi_type_uint16' undeclared (first use in this function)

c/_cffi_backend.c:3376: error: 'ffi_type_uint32' undeclared (first use in this function)

c/_cffi_backend.c:3377: error: 'ffi_type_uint64' undeclared (first use in this function)

c/_cffi_backend.c: At top level:

c/_cffi_backend.c:3931: error: expected specifier-qualifier-list before 'ffi_type'

c/_cffi_backend.c:3950: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token

c/_cffi_backend.c: In function 'fb_build':

c/_cffi_backend.c:4081: error: 'struct funcbuilder_s' has no member named 'atypes'

c/_cffi_backend.c:4081: error: 'ffi_type' undeclared (first use in this function)

c/_cffi_backend.c:4081: error: expected expression before ')' token

c/_cffi_backend.c:4082: error: 'struct funcbuilder_s' has no member named 'nargs'

c/_cffi_backend.c:4085: error: 'struct funcbuilder_s' has no member named 'rtype'

c/_cffi_backend.c:4085: warning: implicit declaration of function 'fb_fill_type'

c/_cffi_backend.c:4093: error: 'cif_description_t' has no member named 'exchange_offset_arg'

c/_cffi_backend.c:4096: error: 'struct funcbuilder_s' has no member named 'rtype'

c/_cffi_backend.c:4097: error: 'ffi_arg' undeclared (first use in this function)

c/_cffi_backend.c:4107: error: 'atype' undeclared (first use in this function)

c/_cffi_backend.c:4120: error: 'struct funcbuilder_s' has no member named 'atypes'

c/_cffi_backend.c:4121: error: 'struct funcbuilder_s' has no member named 'atypes'

c/_cffi_backend.c:4124: error: 'cif_description_t' has no member named 'exchange_offset_arg'

c/_cffi_backend.c:4131: error: 'cif_description_t' has no member named 'exchange_size'

c/_cffi_backend.c: In function 'fb_build_name':

c/_cffi_backend.c:4153: error: 'struct funcbuilder_s' has no member named 'nargs'

c/_cffi_backend.c:4162: error: 'struct funcbuilder_s' has no member named 'fct'

c/_cffi_backend.c:4164: error: 'struct funcbuilder_s' has no member named 'fct'

c/_cffi_backend.c: In function 'fb_prepare_ctype':

c/_cffi_backend.c:4205: error: 'struct funcbuilder_s' has no member named 'fct'

c/_cffi_backend.c:4215: error: 'struct funcbuilder_s' has no member named 'fct'

c/_cffi_backend.c: At top level:

c/_cffi_backend.c:4235: error: expected declaration specifiers or '...' before 'ffi_abi'

c/_cffi_backend.c: In function 'fb_prepare_cif':

c/_cffi_backend.c:4262: warning: implicit declaration of function 'ffi_prep_cif'

c/_cffi_backend.c:4262: error: 'cif_description_t' has no member named 'cif'

c/_cffi_backend.c:4262: error: 'fabi' undeclared (first use in this function)

c/_cffi_backend.c:4262: error: 'struct funcbuilder_s' has no member named 'nargs'

c/_cffi_backend.c:4263: error: 'struct funcbuilder_s' has no member named 'rtype'

c/_cffi_backend.c:4263: error: 'struct funcbuilder_s' has no member named 'atypes'

c/_cffi_backend.c:4263: error: 'FFI_OK' undeclared (first use in this function)

c/_cffi_backend.c: In function 'b_new_function_type':

c/_cffi_backend.c:4280: error: 'FFI_DEFAULT_ABI' undeclared (first use in this function)

c/_cffi_backend.c:4318: error: too many arguments to function 'fb_prepare_cif'

c/_cffi_backend.c:4326: error: 'struct funcbuilder_s' has no member named 'nargs'

c/_cffi_backend.c:4336: error: 'struct funcbuilder_s' has no member named 'nargs'

c/_cffi_backend.c: In function 'convert_from_object_fficallback':

c/_cffi_backend.c:4360: error: 'ffi_arg' undeclared (first use in this function)

c/_cffi_backend.c: At top level:

c/_cffi_backend.c:4432: error: expected ')' before '*' token

c/_cffi_backend.c: In function 'b_callback':

c/_cffi_backend.c:4503: error: 'ffi_closure' undeclared (first use in this function)

c/_cffi_backend.c:4503: error: 'closure' undeclared (first use in this function)

c/_cffi_backend.c:4524: error: 'ffi_arg' undeclared (first use in this function)

c/_cffi_backend.c:4542: warning: implicit declaration of function 'cffi_closure_alloc'

c/_cffi_backend.c:4559: warning: implicit declaration of function 'ffi_prep_closure'

c/_cffi_backend.c:4559: error: 'cif_description_t' has no member named 'cif'

c/_cffi_backend.c:4560: error: 'invoke_callback' undeclared (first use in this function)

c/_cffi_backend.c:4560: error: 'FFI_OK' undeclared (first use in this function)

c/_cffi_backend.c: In function 'init_cffi_backend':

c/_cffi_backend.c:5489: error: 'FFI_DEFAULT_ABI' undeclared (first use in this function)

Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/tmp/pip_build_root/cryptography/setup.py", line 174, in <module>

    "test": PyTest,

  File "/usr/local/lib/python2.7/distutils/core.py", line 112, in setup

    _setup_distribution = dist = klass(attrs)

  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 260, in __init__

  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 285, in fetch_build_eggs

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 631, in resolve

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 871, in best_match

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 883, in obtain

  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 335, in fetch_build_egg

  File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 595, in easy_install

  File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 625, in install_item

  File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 822, in install_eggs

  File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1028, in build_and_install

  File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1016, in run_setup

distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/cryptography
Storing debug log for failure in /root/.pip/pip.log

Solution :

c/_cffi_backend. c:truth:error:ffi. h:No such file or directory
Install the libffi-devel package
yum install libffi-devel

Error: Command ‘/ usr / bin / clang’ failed with exit status 1 solution for Mac OS installation

1、 Scene description

The error of installing scrapy (PIP3 install scrapy) in terminal on Mac OS system is as follows

error: command '/usr/bin/clang' failed with exit status 1

2、 Scene analysis

Since clang is used in the compiler for C at the bottom of Mac OS system, Xcode needs to be installed

3、 Solutions

Do the following in terminal

xcode-select --install

Then install scrapy

Note: I use Python 3, so I use PIP3 install sketch

Three ways of adding cookie by scratch

Settings

, 1. Settings
Settings file annotation Cookies_enabled=False.
Settings cookie is the easiest to configure.
the cookie added in the latter two methods is in dictionary format and needs to be deserialized with json,
and set Cookies_enabled=True

in Settings
Go to the middleware file and find the class DownloadMiddleware. Change process_request and add request.cookies={}.

3. Rewrite start_request

in the crawler main file

def start_requests(self):
    yield scrapy.Request(url,dont_filter=True,cookies={自己的cookie})