Using Selenium + Scrapy

I'm trying to use scrapy with selenium to be able to interact with javascript and still have the powerful scraping framework that scrapy offers. I've written a script that visits h

Solution 1:

the response object you are assigning to your ItemLoader is the scrapy response, not Selenium's.

I would recommend creating a new Selector with the page source returned by selenium:

from scrapy import Selector
...

selenium_response_text = driver.page_source

new_selector = Selector(text=selenium_response_text)
l = ItemLoader(item=PropertiesItem(), selector=new_selector)
...

that way the add_xpath will get information from that response structure instead of scrapy (that you don't actually need).