seon99il

Data Engineering

Airflow TriggerRule & Watcher Pattern

5 min read

In this post, I’ll sort out Trigger_Rule, which I found confusing, and explain how to solve a problem I ran into along the way using the Watcher Pattern. Airflow: 2.9.1

Trigger Rule

Airflow has the following Trigger_rules.

# trigger_rule.py
ALL_SUCCESS = "all_success"
ALL_FAILED = "all_failed"
ALL_DONE = "all_done"
ALL_DONE_SETUP_SUCCESS = "all_done_setup_success"
ONE_SUCCESS = "one_success"
ONE_FAILED = "one_failed"
ONE_DONE = "one_done"
NONE_FAILED = "none_failed"
NONE_FAILED_OR_SKIPPED = "none_failed_or_skipped"
NONE_SKIPPED = "none_skipped"
DUMMY = "dummy"
ALWAYS = "always"
NONE_FAILED_MIN_ONE_SUCCESS = "none_failed_min_one_success"
ALL_SKIPPED = "all_skipped"

Trigger_rule: ONE_FAILED

Runs when at least one of the upstream Tasks has failed.

def fail():
    raise AirflowException("This is a failure task")

def succeed():
    print("This is a success task")

with DAG(
    dag_id='example_dag',
    start_date=datetime.datetime(2021, 1, 1),
    schedule_interval="@once",
    catchup=False,
) as dag:

    task1 = PythonOperator(
        task_id='task1',
        python_callable=fail
    )

    task2 = PythonOperator(
        task_id='task2',
        python_callable=succeed,
        trigger_rule=TriggerRule.ONE_FAILED
    )

    task1 >> task2

With the tasks implemented like this, task1 fails, task2 runs as a result (trigger_rule=TriggerRule.ONE_FAILED), and task2 succeeds.

Result

Trigger_rule: Which Tasks Does a Trigger Rule Apply To?

Now let’s add Task3 as follows.

...
    task3 = PythonOperator(
        task_id='task3',
        python_callable=succeed,
        trigger_rule=TriggerRule.ONE_FAILED
    )

    task1 >> task2 >> task3
  • As in the code above, task1 fails and task2 uses Trigger_rule.ONE_FAILED

Q: Will Task3 run?

Result

The answer is that task3 is skipped. The reason is as follows:

Be aware that trigger rules only rely on the direct upstream (parent) tasks, e.g. TriggerRule.ONE_FAILED will ignore any failed (or upstream_failed) tasks that are not a direct parent of the parameterized task.

In other words, task1 is not a direct upstream (parent) of task3, so the Trigger_Rule does not take effect.

Note that because Task2’s trigger_rule is ONE_FAILED, you get a somewhat odd result like this. With the default ALL_SUCCESS, Task1’s failure would propagate to Task2 (upstream_failed), and task3 would be triggered normally by ONE_FAILED.

Trigger_rule: Propagating Task1’s FAILED State to Task3

So how can we propagate Task1’s FAILED state to task3 while still guaranteeing the order task1, task2, task3?

There are several ways, but you can simply make task1 a direct upstream of task3.

    task1 >> task2
    [task1, task2] >> task3

Trigger_rule: Airflow DAG run status

There is something a little strange here, though. Because task1 failed, you would expect the DAG’s final result to be shown as failed, but when you check the UI, it returns a result slightly different from what you expected.

Airflow UI

You can see why this happens in Airflow’s dagrun.py file.

Airflow: Opening dagrun.py

The reason is that the set of Task Instances used to determine the state of a DAG Run in Airflow is, by default, the Tasks with no downstream, that is, the leaf nodes. This is because the DAG’s dagrun_state is set based on the execution state of the last downstream Task (task2).

# airflow dagrun.py

    @provide_session
    def update_state(
        self, session: Session = NEW_SESSION, execute_callbacks: bool = True
    ) -> tuple[list[TI], DagCallbackRequest | None]:
        """
        Determine the overall state of the DagRun based on the state of its TaskInstances.

        :param session: Sqlalchemy ORM Session
        :param execute_callbacks: Should dag callbacks (success/failure, SLA etc.) be invoked
            directly (default: true) or recorded as a pending request in the ``returned_callback`` property
        :return: Tuple containing tis that can be scheduled in the current loop & `returned_callback` that
            needs to be executed
        """
    ...

    tis_for_dagrun_state = self._tis_for_dagrun_state(dag=dag, tis=tis)

    ...
# airflow dagrun.py

def _tis_for_dagrun_state(self, *, dag, tis):
        """
        Return the collection of tasks that should be considered for evaluation of terminal dag run state.

        Teardown tasks by default are not considered for the purpose of dag run state.  But
        users may enable such consideration with on_failure_fail_dagrun.
        """

        def is_effective_leaf(task):
            for down_task_id in task.downstream_task_ids:
                down_task = dag.get_task(down_task_id)
                if not down_task.is_teardown or down_task.on_failure_fail_dagrun:
                    # we found a down task that is not ignorable; not a leaf
                    return False
            # we found no ignorable downstreams
            # evaluate whether task is itself ignorable
            return not task.is_teardown or task.on_failure_fail_dagrun

        leaf_task_ids = {x.task_id for x in dag.tasks if is_effective_leaf(x)}
        if not leaf_task_ids:
            # can happen if dag is exclusively teardown tasks
            leaf_task_ids = {x.task_id for x in dag.tasks if not x.downstream_list}
        leaf_tis = {ti for ti in tis if ti.task_id in leaf_task_ids if ti.state != TaskInstanceState.REMOVED}
        return leaf_tis

Example of watcher pattern with trigger rules

Let’s see how we can handle the problem above.

The example above is the same as the one in the official Airflow Best Practices documentation.

For example, if there is a task (teardown, TriggerRule.ALL_DONE) that cleans up resources at the very end regardless of the state of the other tasks, you can structure the DAG as follows.

with DAG(
    dag_id="watcher_example",
    schedule="@once",
    start_date=datetime(2021, 1, 1),
    catchup=False,
) as dag:
    failing_task = BashOperator(task_id="failing_task", bash_command="exit 1", retries=0)
    passing_task = BashOperator(task_id="passing_task", bash_command="echo passing_task")
    teardown = BashOperator(
        task_id="teardown",
        bash_command="echo teardown",
        trigger_rule=TriggerRule.ALL_DONE,
    )

    failing_task >> passing_task >> teardown

With this setup, the DAG cannot see failing_task’s failure at the DAG level. (For the same reason as above: teardown always succeeds.)

Here, we can add a watcher task to solve this problem.

@task(trigger_rule=TriggerRule.ONE_FAILED, retries=0)
def watcher():
    raise AirflowException("Failing task because one or more upstream tasks failed.")

with DAG(
    dag_id="watcher_example",
    schedule="@once",
    start_date=datetime(2021, 1, 1),
    catchup=False,
) as dag:
    failing_task = BashOperator(task_id="failing_task", bash_command="exit 1", retries=0)
    passing_task = BashOperator(task_id="passing_task", bash_command="echo passing_task")
    teardown = BashOperator(
        task_id="teardown",
        bash_command="echo teardown",
        trigger_rule=TriggerRule.ALL_DONE,
    )

    failing_task >> passing_task >> teardown
    list(dag.tasks) >> watcher()

By making every task an upstream of the watcher task, the DAG can end up in a failed state if any one task has failed.

This way, if all tasks pass, the watcher is skipped, but if anything fails, the watcher fails and the Dag Run can be put into a failed state too.

If there is no teardown task, the watcher task is not needed.

Previous posts

View all posts →

Comments