python - Join and sum on subset of rows in a dataframe -


i have pandas dataframe stores date ranges , associated colums: 

        date_start    date_end ... lots of other columns ... 1       2016-07-01  2016-07-02 2       2016-07-01  2016-07-03 3       2016-07-01  2016-07-04 4       2016-07-02  2016-07-07 5       2016-07-05  2016-07-06

and dataframe of pikachu sightings indexed date: 

              pikachu_sightings       date 2016-07-01                    2 2016-07-02                    4 2016-07-03                    6 2016-07-04                    8 2016-07-05                   10 2016-07-06                   12 2016-07-07                   14

for each row in first df i'd calculate sum of pikachu_sightings within date range (i.e., date_start date_end) , store in new column. end df (numbers left in clarity): 

        date_start    date_end    total_pikachu_sightings 1       2016-07-01  2016-07-02                      2 + 4 2       2016-07-01  2016-07-03                  2 + 4 + 6 3       2016-07-01  2016-07-04              2 + 4 + 6 + 8 4       2016-07-02  2016-07-07   4 + 6 + 8 + 10 + 12 + 14 5       2016-07-05  2016-07-06                    10 + 12

if doing iteratively i'd iterate on each row in table of date ranges, select subset of rows in table of sightings match date range , perform sum on - way slow dataset: 

for range in ranges.itertuples():     sightings_in_range = sightings[(sightings.index >= range.date_start) & (sightings.index <= range.date_end)]     sum_sightings_in_range = sightings_in_range["pikachu_sightings"].sum()     ranges.set_value(range.index, 'total_pikachu_sightings', sum_sightings_in_range)

this attempt @ using pandas, fails because length of 2 dataframes not match (and if did, there's other flaw in approach): 

range["total_pikachu_sightings"] =     sightings[(sightings.index >= range.date_start) & (sightings.index <= range.date_end)              ["pikachu_sightings"].sum()

i'm trying understand general approach/design should i'd aggregate other functions too, sum seems easiest example. sorry if obvious question - i'm new pandas!

first make sure pikachu_sightings has datetime index , sorted.

p = pikachu_sightings.squeeze() # force series p.index = pd.to_datetime(p.index) p = p.sort_index()

then make sure date_start , date_end datetime.

df.date_start = pd.to_datetime(df.date_start) df.date_end   = pd.to_datetime(df.date_end)

then simply

df.apply(lambda x: p[x.date_start:x.date_end].sum(), axis=1)  0     6 1    12 2    20 3    54 4    22 dtype: int64

Comments

Post a Comment

Popular posts from this blog

jOOQ update returning clause with Oracle -

according http://www.jooq.org/doc/3.8/manual/sql-building/sql-statements/update-statement/ jooq doesn't support returning clause in update statements databases except firebird , postgres. does know, still correct oracle jooq? according oracle's documentation, oracle db supports returning clase delete, insert, or update statements. incorrect manual you're right - manual wrong , should fixed: https://github.com/jooq/jooq/issues/5470 single row update returning jooq 3.8 added support single row update .. returning statements. internally, uses callablestatement , pl/sql block of form: begin update "my_table" set "my_table"."column" = 'xyz' "my_table"."id" = 1 returning "my_table"."id", "my_table"."column" ?, ?; ? := sql%rowcount; end; the relevant issue is: https://github.com/jooq/jooq/issues/5190 multi row update returning a...
Read more

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

i have entity inherits other. on other hand, i'm using lombok project reduce boilerplate code, put @data annotation. annotation @data inheritance produces next warning: generating equals/hashcode implementation without call superclass, though class not extend java.lang.object. if intentional, add '@equalsandhashcode(callsuper=false)' type. is advisable add annotation @equalsandhashcode (callsuper = true) or @equalsandhashcode (callsuper = false) ? if not added, 1 callsuper=false or callsuper=true ? the default value false . 1 if don't specify , ignore warning. yes, recommended add @equalsandhashcode annotation on @data annotated classes extend else object. cannot tell if need true or false , depends on class hierarchy, , need examined on case-by-case basis. however, project or package, can configure in lombok.config call super methods if not direct subclass of object. lombok.equalsandhashcode.callsuper = call for configuration system ...
Read more

java - BasicPathUsageException: Cannot join to attribute of basic type -

i using java 8 jpa (hibernate 5.2.1). have clause works perfectly, until introduce clause make use foreign key table values too. i getting following error: basicpathusageexception: cannot join attribute of basic type i think problem related fact have join table, , not sure how create join using join table. employee - employee_category - category model (employee.java): @manytomany(cascade = cascadetype.all, fetch=fetchtype.eager) @jointable ( name="employee_category", joincolumns={ @joincolumn(name="emp_id", referencedcolumnname="id") }, inversejoincolumns={ @joincolumn(name="cat_id", referencedcolumnname="id") } ) private set<category> categories; model (category.java): @id private string id; jpa when introduce following code, fails error above (i think problem new piece of code): join<t, category> category = from.join("id"); predicates.add(criteriabuilder.like(category....
Read more