Using Dagster with Airflow#

You can find the code for this example on Github

The dagster-airflow package allows users to run Dagster pipelines on Airflow clusters, and to import Airflow DAGs into Dagster instances.

This is not recommended for new projects. This is designed for users who want to use the Dagster API within an existing Airflow cluster, and for those who want tooling support to migrate away from Airflow. Using Dagster within an Airflow limits its capabilities, as you still have to rely on Airflow's scheduler, and not all Dagster features (e.g. manual retries from Dagit) will work.

Airflow Ingest#

This example demonstrates how to use make_dagster_pipeline_from_airflow_dag to compile an Airflow DAG into a Dagster pipeline that can be executed (and explored) the same way as a Dagster-native pipeline.

There are two pipelines in the repo:

airflow_simple_dag demonstrates the use of Airflow templates.
airflow_complex_dag shows the translation of a more complex dependency structure.

from airflow_ingest.airflow_complex_dag import complex_dag
from airflow_ingest.airflow_simple_dag import simple_dag
from dagster import repository
from dagster_airflow.dagster_pipeline_factory import make_dagster_pipeline_from_airflow_dag

airflow_simple_dag = make_dagster_pipeline_from_airflow_dag(simple_dag)
airflow_complex_dag = make_dagster_pipeline_from_airflow_dag(complex_dag)


@repository
def airflow_ingest_example():
    return [airflow_complex_dag, airflow_simple_dag]

Note that the "execution_date" for the Airflow DAG is specified through the pipeline tags. To specify tags, call to:

make_dagster_pipeline_from_airflow_dag(
    dag=dag,
    tags={'airflow_execution_date': utc_execution_date_str}
)