This is the error reported:
Traceback (most recent call last):
File "D:\python-professional-workplace\pycharm projest\ArticleSpider\lib\site-packages\twisted\internet\defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "D:\python-professional-workplace\pycharm projest\ArticleSpider\lib\site-packages\scrapy\crawler.py", line 81, in crawl
start_requests = iter(self.spider.start_requests())
File "D:\python--\scrapytest\Scripts\scrapypy3\Scripts\ArticleSpider\ArticleSpider\spiders\zhihu.py", line 68, in start_requests
browser = webdriver.Firefox(executable_path="D:/python--/geckodriver.exe")
File "D:\python-professional-workplace\pycharm projest\ArticleSpider\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 162, in __init__
keep_alive=True)
File "D:\python-professional-workplace\pycharm projest\ArticleSpider\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "D:\python-professional-workplace\pycharm projest\ArticleSpider\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "D:\python-professional-workplace\pycharm projest\ArticleSpider\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "D:\python-professional-workplace\pycharm projest\ArticleSpider\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: newSession
Solution:
1. The browser I use is Firefox. The main reason is that the version of geckodriver is too low, and the version of the browser is 64.0.
Update to v0.23.0: https://github.com/mozilla/geckodriver/releases
Put the downloaded geckodriver.exe into the Scripts folder under the python installation path.
Note: I set up a virtual environment, so geckodriver.exe is placed in a folder under the virtual environment, not under the scripts file.
2. Update the version of selenium (I updated the selenium first and then updated the geckodriver, so I am not sure if only updating geckodriver is useful, but only updating selenium is useless)
They are:
pip show selenium
python -m pip install --upgrade pip
pip install -U selenium
After the update, update selenium in pycharm.