bash - Airflow pass parameters to dependent task -


what way pass parameter dependent tasks in airflow? have lot of bashes files, , i'm trying migrate approach airflow, don't know how pass properties between tasks.

this real example:

#sqoop bash template sqoop_template = """         sqoop job --exec {{params.job}} -- --target-dir {{params.dir}} --outdir /src/     """  s3_template = """         s3-dist-cp --src= {{params.dir}} "--dest={{params.s3}}     """    #task of extraction in emr t1 = bashoperator(         task_id='extract_account',          bash_command=sqoop_template,          params={'job': 'job', 'dir': 'hdfs:///account/' + time.now().strftime("%y-%m-%d-%h-%m-%s")},         dag=dag) #task upload in s3 backup. t2 = bashoperator(         task_id='s3_upload',         bash_command=s3_template,         params={}, #here need dir name created in t1         depends_on_past=true     )  t2.set_upstream(t1)

in t2 need access dir name created in t1.

solution

#execute valid job sqoop def sqoop_import(table_name, job_name):     s3, hdfs = dirpath(table_name)     sqoop_job = job_default_config(job_name, hdfs)     #call(sqoop_job)     return {'hdfs_dir': hdfs, 's3_dir': s3}  def s3_upload(**context):     hdfs = context['task_instance'].xcom_pull(task_ids='sqoop_import')['hdfs_dir']     s3 = context['task_instance'].xcom_pull(task_ids='sqoop_import')['s3_dir']     s3_cpdist_job = ["s3-dist-cp", "--src=%s" % (hdfs), "--dest=%s" % (s3)]     #call(s3_cpdist_job)     return {'s3_dir': s3} #context['task_instance'].xcom_pull(task_ids='sqoop_import')  def sns_notify(**context):     s3 = context['task_instance'].xcom_pull(task_ids='distcp_s3')['s3_dir']     client = boto3.client('sns')     arn = 'arn:aws:sns:us-east-1:744617668409:pipeline-notification-stg'     response = client.publish(targetarn=arn, message=s3)     return response

that's not definitive solution, improvements welcome. thanks.

check out xcoms - http://airflow.incubator.apache.org/concepts.html#xcoms. these used communicating state between tasks.


Comments

Post a Comment

Popular posts from this blog

jOOQ update returning clause with Oracle -

according http://www.jooq.org/doc/3.8/manual/sql-building/sql-statements/update-statement/ jooq doesn't support returning clause in update statements databases except firebird , postgres. does know, still correct oracle jooq? according oracle's documentation, oracle db supports returning clase delete, insert, or update statements. incorrect manual you're right - manual wrong , should fixed: https://github.com/jooq/jooq/issues/5470 single row update returning jooq 3.8 added support single row update .. returning statements. internally, uses callablestatement , pl/sql block of form: begin update "my_table" set "my_table"."column" = 'xyz' "my_table"."id" = 1 returning "my_table"."id", "my_table"."column" ?, ?; ? := sql%rowcount; end; the relevant issue is: https://github.com/jooq/jooq/issues/5190 multi row update returning a...
Read more

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

i have entity inherits other. on other hand, i'm using lombok project reduce boilerplate code, put @data annotation. annotation @data inheritance produces next warning: generating equals/hashcode implementation without call superclass, though class not extend java.lang.object. if intentional, add '@equalsandhashcode(callsuper=false)' type. is advisable add annotation @equalsandhashcode (callsuper = true) or @equalsandhashcode (callsuper = false) ? if not added, 1 callsuper=false or callsuper=true ? the default value false . 1 if don't specify , ignore warning. yes, recommended add @equalsandhashcode annotation on @data annotated classes extend else object. cannot tell if need true or false , depends on class hierarchy, , need examined on case-by-case basis. however, project or package, can configure in lombok.config call super methods if not direct subclass of object. lombok.equalsandhashcode.callsuper = call for configuration system ...
Read more

java - BasicPathUsageException: Cannot join to attribute of basic type -

i using java 8 jpa (hibernate 5.2.1). have clause works perfectly, until introduce clause make use foreign key table values too. i getting following error: basicpathusageexception: cannot join attribute of basic type i think problem related fact have join table, , not sure how create join using join table. employee - employee_category - category model (employee.java): @manytomany(cascade = cascadetype.all, fetch=fetchtype.eager) @jointable ( name="employee_category", joincolumns={ @joincolumn(name="emp_id", referencedcolumnname="id") }, inversejoincolumns={ @joincolumn(name="cat_id", referencedcolumnname="id") } ) private set<category> categories; model (category.java): @id private string id; jpa when introduce following code, fails error above (i think problem new piece of code): join<t, category> category = from.join("id"); predicates.add(criteriabuilder.like(category....
Read more