scrapy使用selenium库时,报错WebDriverException: Message: Process unexpectedly closed with status 1

你在这里

scrapy使用selenium库时,报错WebDriverException: Message: Process unexpectedly closed with status 1

2018-06-20 01:03:53 [twisted] CRITICAL: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks result = g.send(result) File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 98, in crawl six.reraise(*exc_info) File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 79, in crawl self.spider = self._create_spider(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 102, in _create_spider return self.spidercls.from_crawler(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/scrapy/spiders/__init__.py", line 51, in from_crawler spider = cls(*args, **kwargs) File "/home/zhu/scrapy/properties/properties/spiders/manual.py", line 35, in __init__ self.driver = webdriver.Firefox() File "/home/zhu/.local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 170, in __init__ keep_alive=True) File "/home/zhu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 156, in __init__ self.start_session(capabilities, browser_profile) File "/home/zhu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 245, in start_session response = self.execute(Command.NEW_SESSION, parameters) File "/home/zhu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 314, in execute self.error_handler.check_response(response) File "/home/zhu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) WebDriverException: Message: Process unexpectedly closed with status 1

回答

检查scrapy工程目录下的geckodriver.log日志,如果内容如下:
Error: GDK_BACKEND does not match available displays  或
Error: no DISPLAY environment variable specified
表示没有可供显示的环境,因为是在命令行模式下(无头模式)试图启动firefox。此时需要为selenium的webdriver.Firefox设置其选项,如下:
from selenium import webdriver
from selenium.webdriver import FirefoxOptions
opts = FirefoxOptions()
opts.add_argument("--headless")
browser = webdriver.Firefox(firefox_options=opts)
browser.get('http://example.com')