scala - Spark joining DataFrames and aggregation -


this sort of contrived example, captures trying using spark/scala

pet types

val pets = array(row(1,"cat"),row(2,"dog")) val petsrdd = sc.parallelize(pets) val petschema = structtype(array(structfield("id",integertype),structfield("type",stringtype))) val petsdf = sqlcontext.createdataframe(petsrdd,petschema)

pet names

val petnames = array(row(1,1,"tigger","m"),row(2,1,"winston","m"),row(3,1,"snowball","f"),row(4,2,"spot","m"),row(5,2,"barf","m"),row(6,2,"snoppy","m")) val petnamesrdd = sc.parallelize(petnames) val petnameschema = structtype(array(structfield("id",integertype),structfield("pet_id",integertype),structfield("name",stringtype),structfield("gender",stringtype))) val petnamesdf = sqlcontext.createdataframe(petnamesrdd,petnameschema)

from here can join dataframes ...

val join = petsdf.join(petnamesdf, petsdf("id") === petnamesdf("pet_id") ), "leftouter")

results

+---+-----+---+--------+---------+------+ | id| type| id| pet_id | name    |gender +---+-----+---+--------+---------+------+ |  1|  cat| 1 | 1      |tigger   | m |  1|  cat| 2 | 1      |winston  | m |  1|  cat| 3 | 1      |snowball | f |  2|  dog| 4 | 2      |spot     | m |  3|  dog| 5 | 2      |barf     | m |  3|  dog| 6 | 2      |snoopy   | f +---+-----+---+--------+---------+------+

i flatten results looks can map results more processing.

((1,"cat"),(1,"tigger","m"),(2,"winston","m"),(3,"snowball","f")) ((2,"dog"),(1,"spot","m"),(2,"barf","m"),(3,"snoopy","f"))

i started looking @ userdefinedaggregatefunctions not work. did not try hard, seems not fit.

i looked using map transform each petdf row petdf (list of petnames), nested df not allowed.

i hoping missing built spark or idea work. new spark/scala.
thanks


Comments

Post a Comment

Popular posts from this blog

jOOQ update returning clause with Oracle -

according http://www.jooq.org/doc/3.8/manual/sql-building/sql-statements/update-statement/ jooq doesn't support returning clause in update statements databases except firebird , postgres. does know, still correct oracle jooq? according oracle's documentation, oracle db supports returning clase delete, insert, or update statements. incorrect manual you're right - manual wrong , should fixed: https://github.com/jooq/jooq/issues/5470 single row update returning jooq 3.8 added support single row update .. returning statements. internally, uses callablestatement , pl/sql block of form: begin update "my_table" set "my_table"."column" = 'xyz' "my_table"."id" = 1 returning "my_table"."id", "my_table"."column" ?, ?; ? := sql%rowcount; end; the relevant issue is: https://github.com/jooq/jooq/issues/5190 multi row update returning a...
Read more

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

i have entity inherits other. on other hand, i'm using lombok project reduce boilerplate code, put @data annotation. annotation @data inheritance produces next warning: generating equals/hashcode implementation without call superclass, though class not extend java.lang.object. if intentional, add '@equalsandhashcode(callsuper=false)' type. is advisable add annotation @equalsandhashcode (callsuper = true) or @equalsandhashcode (callsuper = false) ? if not added, 1 callsuper=false or callsuper=true ? the default value false . 1 if don't specify , ignore warning. yes, recommended add @equalsandhashcode annotation on @data annotated classes extend else object. cannot tell if need true or false , depends on class hierarchy, , need examined on case-by-case basis. however, project or package, can configure in lombok.config call super methods if not direct subclass of object. lombok.equalsandhashcode.callsuper = call for configuration system ...
Read more

java - BasicPathUsageException: Cannot join to attribute of basic type -

i using java 8 jpa (hibernate 5.2.1). have clause works perfectly, until introduce clause make use foreign key table values too. i getting following error: basicpathusageexception: cannot join attribute of basic type i think problem related fact have join table, , not sure how create join using join table. employee - employee_category - category model (employee.java): @manytomany(cascade = cascadetype.all, fetch=fetchtype.eager) @jointable ( name="employee_category", joincolumns={ @joincolumn(name="emp_id", referencedcolumnname="id") }, inversejoincolumns={ @joincolumn(name="cat_id", referencedcolumnname="id") } ) private set<category> categories; model (category.java): @id private string id; jpa when introduce following code, fails error above (i think problem new piece of code): join<t, category> category = from.join("id"); predicates.add(criteriabuilder.like(category....
Read more