python - How to parse labeled values of columns into a Pandas Dataframe (some column values are missing)? -


the follow 2 rows unlabeled dataset, small subset:

random1 147 sub1    95  34  dewdfa3 15000   -1238   sbaasbaqsbarsbatsbausbaxbelaax  aaa:col:uvtwuvwduwduwdww    bbb:col:f   ccc:col:gtatgtca    ddd:col:k20 eee:col:54t ggg:col:-30.5   hhh:col:000.1   iii:col:2   jjj:col:0     random2 123 sub1    996 12  kwnc239 10027    144        lbprlbpslbrdlbsdlbsllbwb    aaa:col:uwttutuvvuwwuuu bbb:col:f   ddd:col:cacgtcgg    eee:col:k19 fff:col:hcc16   ggg:col:873 iii:col:-77 jjj:col:0   kkk:col:0   lll:col:1   mmm:col:212

the first 9 columns consistent throughout dataset, , labeled.

my problem following columns. each value in row labeled column value first, e.g. aaa:col:uvtwuvwduwduwdww column aaa, bbb:col:f column bbb, etc.

however, (1) each row not have same number of columns , (2) columns "missing". first row missing column fff, second row skips column ccc , hhh.

also, notice first row stops @ column jjj, while second column stops @ column mmm.

how 1 allocate 9 + 13 columns of dataframe, , parse these values such if column:value pair didn't exist, column have nan value.

would pandas.read_table() have functionality this?

this "correct" format first row:

random    int     sub    int2    int3    string1    int4    int5    string2                         aaa            bbb    ccc    ddd    eee    fff    ggg .... mmm random1   147    sub1    95      34      dewdfa3    15000   -1238   sbaasbaqsbarsbatsbausbaxbelaax  uvtwuvwduwduwdww    f   dfadfadfa   k20 54t 'nan' -30.5 ....'nan'

related (and unanswered) question here: how import unlabeled , missing columns pandas dataframe?

this it:

text = """random1 147 sub1    95  34  dewdfa3 15000   -1238   sbaasbaqsbarsbatsbausbaxbelaax  aaa:col:uvtwuvwduwduwdww    bbb:col:f   ccc:col:gtatgtca    ddd:col:k20 eee:col:54t ggg:col:-30.5    hhh:col:000.1   iii:col:2  jjj:col:0    random2 123 sub1    996 12  kwnc239 10027    144        lbprlbpslbrdlbsdlbsllbwb    aaa:col:uwttutuvvuwwuuu bbb:col:f   ddd:col:cacgtcgg    eee:col:k19 fff:col:hcc16   ggg:col:873 iii:col:-77 jjj:col:0   kkk:col:0   lll:col:1   mmm:col:212"""  data = [line.split() line in text.split('\n')] data1 = [line[:9] line in data] data2 = [line[9:] line in data]  # list of dictionaries data2, parse columns dict2 = [[dict([d.split(':col:') d in d1]) d1 in data2]  result = pd.concat([pd.dataframe(data1),                     pd.dataframe(dict2)],                    axis=1)  result.iloc[:, 9:]

enter image description here


Comments

Post a Comment

Popular posts from this blog

jOOQ update returning clause with Oracle -

according http://www.jooq.org/doc/3.8/manual/sql-building/sql-statements/update-statement/ jooq doesn't support returning clause in update statements databases except firebird , postgres. does know, still correct oracle jooq? according oracle's documentation, oracle db supports returning clase delete, insert, or update statements. incorrect manual you're right - manual wrong , should fixed: https://github.com/jooq/jooq/issues/5470 single row update returning jooq 3.8 added support single row update .. returning statements. internally, uses callablestatement , pl/sql block of form: begin update "my_table" set "my_table"."column" = 'xyz' "my_table"."id" = 1 returning "my_table"."id", "my_table"."column" ?, ?; ? := sql%rowcount; end; the relevant issue is: https://github.com/jooq/jooq/issues/5190 multi row update returning a...
Read more

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

i have entity inherits other. on other hand, i'm using lombok project reduce boilerplate code, put @data annotation. annotation @data inheritance produces next warning: generating equals/hashcode implementation without call superclass, though class not extend java.lang.object. if intentional, add '@equalsandhashcode(callsuper=false)' type. is advisable add annotation @equalsandhashcode (callsuper = true) or @equalsandhashcode (callsuper = false) ? if not added, 1 callsuper=false or callsuper=true ? the default value false . 1 if don't specify , ignore warning. yes, recommended add @equalsandhashcode annotation on @data annotated classes extend else object. cannot tell if need true or false , depends on class hierarchy, , need examined on case-by-case basis. however, project or package, can configure in lombok.config call super methods if not direct subclass of object. lombok.equalsandhashcode.callsuper = call for configuration system ...
Read more

java - BasicPathUsageException: Cannot join to attribute of basic type -

i using java 8 jpa (hibernate 5.2.1). have clause works perfectly, until introduce clause make use foreign key table values too. i getting following error: basicpathusageexception: cannot join attribute of basic type i think problem related fact have join table, , not sure how create join using join table. employee - employee_category - category model (employee.java): @manytomany(cascade = cascadetype.all, fetch=fetchtype.eager) @jointable ( name="employee_category", joincolumns={ @joincolumn(name="emp_id", referencedcolumnname="id") }, inversejoincolumns={ @joincolumn(name="cat_id", referencedcolumnname="id") } ) private set<category> categories; model (category.java): @id private string id; jpa when introduce following code, fails error above (i think problem new piece of code): join<t, category> category = from.join("id"); predicates.add(criteriabuilder.like(category....
Read more