Couldn't extract the data from large partition in cassandra -


i realized after sometime partition size in cassandra data model incorrect. , had designed primary key created 1 partition. so, getting read timeout in queries. have fixed issue creating new column family problem how recover data partition appreciated. have around 40gb of partition given below : 

38g     ./eventdata-7b5004e02bdc11e69ebb19dbbcf6d4b0 55m     ./eventdata-8f8317a01e5711e69ebb19dbbcf6d4b0 2.4m    ./eventdata-3c20dd401cf911e69ebb19dbbcf6d4b0 88k     ./eventdata-1866015012a911e69ebb19dbbcf6d4b0

so, read data using sstable2json convert in json gave error :

exception in thread "main" java.lang.outofmemoryerror: gc overhead limit exceeded         @ java.nio.heapbytebuffer.duplicate(heapbytebuffer.java:107)         @ org.apache.cassandra.db.composites.abstractctype.slicebytes(abstractctype.java:369)         @ org.apache.cassandra.db.composites.abstractcompoundcellnametype.frombytebuffer(abstractcompoundcellnametype.java:101)         @ org.apache.cassandra.db.composites.abstractctype$serializer.deserialize(abstractctype.java:398)         @ org.apache.cassandra.db.composites.abstractctype$serializer.deserialize(abstractctype.java:382)         @ org.apache.cassandra.io.sstable.indexhelper$indexinfo$serializer.deserialize(indexhelper.java:194)         @ org.apache.cassandra.io.sstable.indexhelper$indexinfo$serializer.deserialize(indexhelper.java:174)         @ org.apache.cassandra.db.rowindexentry$serializer.deserialize(rowindexentry.java:129)         @ org.apache.cassandra.io.sstable.sstablescanner$keyscanningiterator.computenext(sstablescanner.java:257)         @ org.apache.cassandra.io.sstable.sstablescanner$keyscanningiterator.computenext(sstablescanner.java:231)         @ com.google.common.collect.abstractiterator.trytocomputenext(abstractiterator.java:143)         @ com.google.common.collect.abstractiterator.hasnext(abstractiterator.java:138)         @ org.apache.cassandra.io.sstable.sstablescanner.hasnext(sstablescanner.java:211)         @ org.apache.cassandra.tools.sstableexport.export(sstableexport.java:332)         @ org.apache.cassandra.tools.sstableexport.export(sstableexport.java:369)         @ org.apache.cassandra.tools.sstableexport.export(sstableexport.java:382)         @ org.apache.cassandra.tools.sstableexport.main(sstableexport.java:467) error 14:02:13 leak detected: reference (org.apache.cassandra.utils.concurrent.ref$state@2835df6c) class org.apache.cassandra.io.sstable.s stablereader$instancetidier@1560150690:/mnt/eventdata/eventdata-7b5004e02bdc11e69ebb19dbbcf6d4b0/./cw-eventdata-ka-2839 not released before reference garbage collected

then trying split sstable using sstablesplit utility again got error. have cassandra 2.1.12 running on node having 8gb of ram , 8gb of swap space :

exception in thread "main" java.lang.outofmemoryerror: gc overhead limit exceeded         @ org.apache.cassandra.io.util.randomaccessreader.readbytes(randomaccessreader.java:375)         @ org.apache.cassandra.utils.bytebufferutil.read(bytebufferutil.java:352)         @ org.apache.cassandra.utils.bytebufferutil.readwithshortlength(bytebufferutil.java:331)         @ org.apache.cassandra.db.composites.abstractctype$serializer.deserialize(abstractctype.java:398)         @ org.apache.cassandra.db.composites.abstractctype$serializer.deserialize(abstractctype.java:382)         @ org.apache.cassandra.io.sstable.indexhelper$indexinfo$serializer.deserialize(indexhelper.java:193)         @ org.apache.cassandra.io.sstable.indexhelper$indexinfo$serializer.deserialize(indexhelper.java:174)         @ org.apache.cassandra.db.rowindexentry$serializer.deserialize(rowindexentry.java:129)         @ org.apache.cassandra.io.sstable.sstablescanner$keyscanningiterator.computenext(sstablescanner.java:257)         @ org.apache.cassandra.io.sstable.sstablescanner$keyscanningiterator.computenext(sstablescanner.java:231)         @ com.google.common.collect.abstractiterator.trytocomputenext(abstractiterator.java:143)         @ com.google.common.collect.abstractiterator.hasnext(abstractiterator.java:138)         @ org.apache.cassandra.io.sstable.sstablescanner.hasnext(sstablescanner.java:211)         @ org.apache.cassandra.utils.mergeiterator$onetoone.computenext(mergeiterator.java:202)         @ com.google.common.collect.abstractiterator.trytocomputenext(abstractiterator.java:143)         @ com.google.common.collect.abstractiterator.hasnext(abstractiterator.java:138)         @ org.apache.cassandra.db.compaction.compactiontask.runmaythrow(compactiontask.java:181)         @ org.apache.cassandra.utils.wrappedrunnable.run(wrappedrunnable.java:28)         @ org.apache.cassandra.db.compaction.compactiontask.executeinternal(compactiontask.java:73)         @ org.apache.cassandra.db.compaction.abstractcompactiontask.execute(abstractcompactiontask.java:59)         @ org.apache.cassandra.db.compaction.sstablesplitter.split(sstablesplitter.java:38)         @ org.apache.cassandra.tools.standalonesplitter.main(standalonesplitter.java:153)


Comments

Post a Comment

Popular posts from this blog

jOOQ update returning clause with Oracle -

according http://www.jooq.org/doc/3.8/manual/sql-building/sql-statements/update-statement/ jooq doesn't support returning clause in update statements databases except firebird , postgres. does know, still correct oracle jooq? according oracle's documentation, oracle db supports returning clase delete, insert, or update statements. incorrect manual you're right - manual wrong , should fixed: https://github.com/jooq/jooq/issues/5470 single row update returning jooq 3.8 added support single row update .. returning statements. internally, uses callablestatement , pl/sql block of form: begin update "my_table" set "my_table"."column" = 'xyz' "my_table"."id" = 1 returning "my_table"."id", "my_table"."column" ?, ?; ? := sql%rowcount; end; the relevant issue is: https://github.com/jooq/jooq/issues/5190 multi row update returning a...
Read more

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

i have entity inherits other. on other hand, i'm using lombok project reduce boilerplate code, put @data annotation. annotation @data inheritance produces next warning: generating equals/hashcode implementation without call superclass, though class not extend java.lang.object. if intentional, add '@equalsandhashcode(callsuper=false)' type. is advisable add annotation @equalsandhashcode (callsuper = true) or @equalsandhashcode (callsuper = false) ? if not added, 1 callsuper=false or callsuper=true ? the default value false . 1 if don't specify , ignore warning. yes, recommended add @equalsandhashcode annotation on @data annotated classes extend else object. cannot tell if need true or false , depends on class hierarchy, , need examined on case-by-case basis. however, project or package, can configure in lombok.config call super methods if not direct subclass of object. lombok.equalsandhashcode.callsuper = call for configuration system ...
Read more

java - BasicPathUsageException: Cannot join to attribute of basic type -

i using java 8 jpa (hibernate 5.2.1). have clause works perfectly, until introduce clause make use foreign key table values too. i getting following error: basicpathusageexception: cannot join attribute of basic type i think problem related fact have join table, , not sure how create join using join table. employee - employee_category - category model (employee.java): @manytomany(cascade = cascadetype.all, fetch=fetchtype.eager) @jointable ( name="employee_category", joincolumns={ @joincolumn(name="emp_id", referencedcolumnname="id") }, inversejoincolumns={ @joincolumn(name="cat_id", referencedcolumnname="id") } ) private set<category> categories; model (category.java): @id private string id; jpa when introduce following code, fails error above (i think problem new piece of code): join<t, category> category = from.join("id"); predicates.add(criteriabuilder.like(category....
Read more