Dynamic DAG’s In Airflow

Dynamic dags are dags which are generated programmatically,In Airflow DAG is defined in a Python file and remains static but there are scenarios where we may want the structure of the DAG to change based on certain conditions, In this cases we use Dynamic DAG’s.

Let’s create a very simple example for this

from datetime import datetime
from airflow.decorators import dag, task

configs = {
"config1": {"message": "first DAG will print this message"},
"config2": {"message": "second DAG will print this message"},
}

for config_name, config in configs.items():
    dag_id = f"dynamic_dag_{config_name}"

    @dag(dag_id=dag_id, start_date=datetime(2023, 1, 1))
    def dynamic_generated_dag():
        @task
        def print_message(message):
            print(message)

        print_message(config["message"])

dynamic_generated_dag()

• In this example, we are creating the dynamically.
• It’s going to create two dags and they will named as dynamic_dag_config1 and dynamic_dag_config1.
• We can verify if this dags has been created in the airflow UI.
• Let’s check our dag on the airflow UI.



• Now let’s run the dags and verify the output of the dags.

We at Helical have more than 10 years of experience in providing solutions and services in the domain of data and have served more than 85+ clients. Please reach out to us for assistance, consulting, services, maintenance as well as POC and to hear about our past experience on Airflow. Please do reach out on nikhilesh@Helicaltech.com

Thank You
Abhishek Mishra
Helical IT Solutions