python - scrapy unable to extract some data from website -

-

i using scrapy crawl page, able simple things visible text. there texts not visible crawler , end showing spaces.

for instance seeing page sources allow me see these fields:

https://www.dropbox.com/s/f056mffmuah6uu4/screenshot%202015-07-23%2018.23.32.png?dl=0

i've tried numerous times access field through xpath , css , not able these fields after each attempt.

when try like: 

response.xpath('//text()').extract()

i not able see these fields in text dump @ all.

would have idea why these fields not visible scrapy? website is: https://www.buzzbuzzhome.com/uc/units/houses/sapphire

in spider, need make additional xhr post request https://www.buzzbuzzhome.com/bbhajax/development/unitpricehistory endpoint price history providing necessary headers , post parameters:

import json import scrapy   class buzzspider(scrapy.spider):     name = 'buzzbuzzhome'     allowed_domains = ['buzzbuzzhome.com']     start_urls = ['https://www.buzzbuzzhome.com/uc/units/houses/sapphire']      def parse(self, response):         unit_id = response.xpath("//div[@id = 'unitdetails']/@data-unit-id").extract()[0]         development_url = "uc"         new_relic_id = response.xpath("//script[contains(., 'xpid')]").re(r'xpid:"(.*?)"')          params = {"developmenturl": development_url, "unitid": unit_id}         yield scrapy.request("https://www.buzzbuzzhome.com/bbhajax/development/unitpricehistory",                              method="post",                              body=json.dumps(params),                              callback=self.parse_history,                              headers={                                  "accept": "*/*",                                  "user-agent": "mozilla/5.0 (macintosh; intel mac os x 10_10_2) applewebkit/537.36 (khtml, gecko) chrome/43.0.2357.134 safari/537.36",                                  "x-requested-with": "xmlhttprequest",                                  "x-newrelic-id": new_relic_id,                                  "origin": "https://www.buzzbuzzhome.com",                                  "host": "www.buzzbuzzhome.com",                                  'content-type': 'application/json; charset=utf-8'                              })      def parse_history(self, response):         row in response.css("div.row"):             title = row.xpath(".//div[@class='content-title']/text()").extract()[0].strip()             text = row.xpath(".//div[@class='content-text']/text()").extract()[0].strip()              print title, text

prints:

05/25/2015 unit listed sold 12/18/2014 unit listed sale 11/24/2014 unit price increased  1.54% $461,990 11/04/2014 unit price increased  6.81% $454,990 10/02/2014 unit price increased  4.67% $425,990 01/22/2014 unit price increased  2.52% $406,990 12/06/2013 unit listed sale @ $396,990

Comments

Post a Comment

Popular posts from this blog

python - pip install -U PySide error -

-
does know how avoid following error when running pip install -u pyside per official website instructions here: https://pypi.python.org/pypi/pyside/#installing-pyside-on-a-mac-os-x-system note have done brew install qt successfully. you using pip version 7.0.3, version 7.1.2 available. should consider upgrading via 'pip install --upgrade pip' command. collecting pyside using cached pyside-1.2.4.tar.gz installing collected packages: pyside running setup.py install pyside complete output command /applications/anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-build-cpxemt/pyside/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-0r7hri-record/install-record.txt --single-version-externally-managed --compile: ...
Read more

arrays - C++ error: a brace-enclosed initializer is not allowed here before ‘{’ token -

-
simple question: i'm trying initialize array within c++ class declaration: using namespace std; #include <string> class myclass{ public: string myarray[] = {"a","b","c"}; }; and i'm getting error: error: brace-enclosed initializer not allowed here before ‘{’ token no, without complied c++11 compiler, cannot initialize member array in declaration. have initialize array member in constructor. , don't use open array if know number of elements initialize array.
Read more

cytoscape.js - How to add nodes to Dagre layout with Cytoscape -

-
i trying add nodes dagre graph through cityscape.js. problem nodes not painted. code have: cy.add({ group: "nodes", data: { id:"n0",name:"n0",weight: 75 }, }); $.ajax(url).done(function(jsontree){ var newnodes=[]; for(var i=1;i<6;i++){ //var child = jsontree.result[i]; newnodes.push( { group: "nodes", data: { id: "n"+i, name:"n"+i}}, { group: "edges", data: { id: "e"+i, source: "n0", target: "n"+i } } ); } cy.add(newnodes); any ideas? have used loaded function reload graph not work. not know if have use specific function reload graph. thank in advance. you need add positions new nodes. cytoscape.js docs : it important note positions of newly added nodes must defined when calling cy.add(). nodes can not placed in graph without valid position — otherwise not displayed. yo...
Read more