Scraping Table With Python/BS4 -


im trying scrape "team stats" table http://www.pro-football-reference.com/boxscores/201602070den.htm bs4 , python 2.7. im unable anywhere close it, 

url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm' page = requests.get(url) soup = beautifulsoup(page.text, "html5lib") table=soup.findall('table', {'id':"team_stats", "class":"stats_table"})   print table

i thought above code work no luck.

the problem in case "team stats" table located inside comment in html source download requests. locate comment , reparse beautifulsoup "soup" object:

import requests bs4 import beautifulsoup, navigablestring  url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm' page = requests.get(url, headers={'user-agent': 'mozilla/5.0 (macintosh; intel mac os x 10_11_4) applewebkit/537.36 (khtml, gecko) chrome/51.0.2704.103 safari/537.36'})  soup = beautifulsoup(page.content, "html5lib") comment = soup.find(text=lambda x: isinstance(x, navigablestring) , "team_stats" in x)  soup = beautifulsoup(comment, "html5lib") table = soup.find("table", id="team_stats") print(table)

and/or, can load table into, example, pandas dataframe convenient work with:

import pandas pd import requests bs4 import beautifulsoup bs4 import navigablestring  url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm' page = requests.get(url, headers={'user-agent': 'mozilla/5.0 (macintosh; intel mac os x 10_11_4) applewebkit/537.36 (khtml, gecko) chrome/51.0.2704.103 safari/537.36'})  soup = beautifulsoup(page.content, "html5lib") comment = soup.find(text=lambda x: isinstance(x, navigablestring) , "team_stats" in x)  df = pd.read_html(comment)[0] print(df)

prints:

            unnamed: 0            den            car 0          first downs             11             21 1         rush-yds-tds        28-90-1       27-118-1 2    cmp-att-yd-td-int  13-23-141-0-1  18-41-265-0-1 3         sacked-yards           5-37           7-68 4       net pass yards            104            197 5          total yards            194            315 6         fumbles-lost            3-1            4-3 7            turnovers              2              4 8      penalties-yards           6-51         12-102 9     third down conv.           1-14           3-15 10   fourth down conv.            0-0            0-0 11  time of possession          27:13          32:47

Comments

Post a Comment

Popular posts from this blog

jOOQ update returning clause with Oracle -

according http://www.jooq.org/doc/3.8/manual/sql-building/sql-statements/update-statement/ jooq doesn't support returning clause in update statements databases except firebird , postgres. does know, still correct oracle jooq? according oracle's documentation, oracle db supports returning clase delete, insert, or update statements. incorrect manual you're right - manual wrong , should fixed: https://github.com/jooq/jooq/issues/5470 single row update returning jooq 3.8 added support single row update .. returning statements. internally, uses callablestatement , pl/sql block of form: begin update "my_table" set "my_table"."column" = 'xyz' "my_table"."id" = 1 returning "my_table"."id", "my_table"."column" ?, ?; ? := sql%rowcount; end; the relevant issue is: https://github.com/jooq/jooq/issues/5190 multi row update returning a...
Read more

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

i have entity inherits other. on other hand, i'm using lombok project reduce boilerplate code, put @data annotation. annotation @data inheritance produces next warning: generating equals/hashcode implementation without call superclass, though class not extend java.lang.object. if intentional, add '@equalsandhashcode(callsuper=false)' type. is advisable add annotation @equalsandhashcode (callsuper = true) or @equalsandhashcode (callsuper = false) ? if not added, 1 callsuper=false or callsuper=true ? the default value false . 1 if don't specify , ignore warning. yes, recommended add @equalsandhashcode annotation on @data annotated classes extend else object. cannot tell if need true or false , depends on class hierarchy, , need examined on case-by-case basis. however, project or package, can configure in lombok.config call super methods if not direct subclass of object. lombok.equalsandhashcode.callsuper = call for configuration system ...
Read more

java - BasicPathUsageException: Cannot join to attribute of basic type -

i using java 8 jpa (hibernate 5.2.1). have clause works perfectly, until introduce clause make use foreign key table values too. i getting following error: basicpathusageexception: cannot join attribute of basic type i think problem related fact have join table, , not sure how create join using join table. employee - employee_category - category model (employee.java): @manytomany(cascade = cascadetype.all, fetch=fetchtype.eager) @jointable ( name="employee_category", joincolumns={ @joincolumn(name="emp_id", referencedcolumnname="id") }, inversejoincolumns={ @joincolumn(name="cat_id", referencedcolumnname="id") } ) private set<category> categories; model (category.java): @id private string id; jpa when introduce following code, fails error above (i think problem new piece of code): join<t, category> category = from.join("id"); predicates.add(criteriabuilder.like(category....
Read more