python - How can I jump to next page in Scrapy -


i'm trying scrape results here using scrapy. problem not of classes appear on page until 'load more results' tab clicked.

the problem can seen here:

enter image description here

my code looks this:

class classcentralspider(crawlspider):     name = "class_central"     allowed_domains = ["www.class-central.com"]     start_urls = (         'https://www.class-central.com/courses/recentlyadded',     )     rules = (         rule(             linkextractor(                 # allow=("index\d00\.html",),                 restrict_xpaths=('//div[@id="show-more-courses"]',)             ),             callback='parse',             follow=true         ),     )  def parse(self, response):     x = response.xpath('//span[@class="course-name-text"]/text()').extract()     item = classcentralitem()     y in x:         item['name'] = y         print item['name']      pass

the second page website seems generated via ajax call. if network tab of browser inspection tool, you'll see like:

firebug network tab

in case seems retrieving json file https://www.class-central.com/maestro/courses/recentlyadded?page=2&_=1469471093134

now seems url parameter _=1469471093134 nothing can trim away to: https://www.class-central.com/maestro/courses/recentlyadded?page=2
return json contains html code next page:

# need load  data = json.loads(response.body)  # , convert scrapy selector -  sel = selector(text=data['table'])

to replicate in code try like: 

from w3lib.url import add_or_replace_parameter  def parse(self, response):     # check if response json, if convert selector     if response.meta.get('is_json',false):         # convert json scrapy.selector here parsing         sel = selector(text=json.loads(response.body)['table'])     else:         sel = selector(response)      # parse page here items     x = sel.xpath('//span[@class="course-name-text"]/text()').extract()     item = classcentralitem()     y in x:         item['name'] = y         print(item['name'])     # next page     next_page_el = respones.xpath("//div[@id='show-more-courses']")     if next_page_el:  # there next page         next_page = response.meta.get('page',1) + 1         # make next page url         url = add_or_replace_parameter(url, 'page', next_page)         yield request(url, self.parse, meta={'page': next_page, 'is_json': true)

Comments

Post a Comment

Popular posts from this blog

jOOQ update returning clause with Oracle -

according http://www.jooq.org/doc/3.8/manual/sql-building/sql-statements/update-statement/ jooq doesn't support returning clause in update statements databases except firebird , postgres. does know, still correct oracle jooq? according oracle's documentation, oracle db supports returning clase delete, insert, or update statements. incorrect manual you're right - manual wrong , should fixed: https://github.com/jooq/jooq/issues/5470 single row update returning jooq 3.8 added support single row update .. returning statements. internally, uses callablestatement , pl/sql block of form: begin update "my_table" set "my_table"."column" = 'xyz' "my_table"."id" = 1 returning "my_table"."id", "my_table"."column" ?, ?; ? := sql%rowcount; end; the relevant issue is: https://github.com/jooq/jooq/issues/5190 multi row update returning a...
Read more

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

i have entity inherits other. on other hand, i'm using lombok project reduce boilerplate code, put @data annotation. annotation @data inheritance produces next warning: generating equals/hashcode implementation without call superclass, though class not extend java.lang.object. if intentional, add '@equalsandhashcode(callsuper=false)' type. is advisable add annotation @equalsandhashcode (callsuper = true) or @equalsandhashcode (callsuper = false) ? if not added, 1 callsuper=false or callsuper=true ? the default value false . 1 if don't specify , ignore warning. yes, recommended add @equalsandhashcode annotation on @data annotated classes extend else object. cannot tell if need true or false , depends on class hierarchy, , need examined on case-by-case basis. however, project or package, can configure in lombok.config call super methods if not direct subclass of object. lombok.equalsandhashcode.callsuper = call for configuration system ...
Read more

java - BasicPathUsageException: Cannot join to attribute of basic type -

i using java 8 jpa (hibernate 5.2.1). have clause works perfectly, until introduce clause make use foreign key table values too. i getting following error: basicpathusageexception: cannot join attribute of basic type i think problem related fact have join table, , not sure how create join using join table. employee - employee_category - category model (employee.java): @manytomany(cascade = cascadetype.all, fetch=fetchtype.eager) @jointable ( name="employee_category", joincolumns={ @joincolumn(name="emp_id", referencedcolumnname="id") }, inversejoincolumns={ @joincolumn(name="cat_id", referencedcolumnname="id") } ) private set<category> categories; model (category.java): @id private string id; jpa when introduce following code, fails error above (i think problem new piece of code): join<t, category> category = from.join("id"); predicates.add(criteriabuilder.like(category....
Read more