Mahendra Yadav

Data Engineer at Azri Solutions

• Twitter: https://www.twitter.com/@userimack/
• Github: http://github.com/userimack/
• Blog: https://userimack.wordpress.com/

Introduction to Airflow

Airflow is a platform to programmatically author, schedule and monitor data pipelines.

Data Pipeline

Existing challenges

Main Advantages

Stats

Logs

Handle task failures

Alert on failures

Monitoring of tasks over time

Pipelines written in Python - Code View 1

Pipelines written in Python - Code View 2

Basic Concepts

Workflow as DAG

• DAG (Directed Acyclic Graph) is a collection of all the tasks
• Reflects the relationships and dependencies between the tasks
• Addition of new DAG is very easy.

Tasks as Operators

• BashOperator
• PythonOperator
• PostgresOperator
• MySqlOperator
• ExternalTaskSensor
• EmailOperator
• SlackOperator
• And many more...

Scheduling the DAGs

DAGs can be scheduled to run at certain frequency by providing the parameter schedule interval.

Executors

Executor is responsible for execution of the jobs.

• Sequential (Default)
• Local
• Celery

DAG Anatomy

Default Arguments

Instantiation

Tasks

Airflow in practice - case study

ETL Pipelines

Download Data

↓

Process

↓

Load

↓

Send Report

☺

Demo

Email

Who uses Airflow

Resources

• Documentation: https://airflow.incubator.apache.org/
• Wiki: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home
• Gitter: https://gitter.im/apache/incubator-airflow
• Mailing List: http://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/

Thank You!

Mahendra Yadav

• Email: mahendra@azrisolutions.com
• Github: http://github.com/userimack/
• Blog: https://userimack.wordpress.com/
• Linkedin: https://www.linkedin.com/in/userimack/
• Twitter: https://www.twitter.com/@userimack/
• Slides: https://userimack.github.io/airflow_slides