seon99il

Data Engineering

Airflow Best Practices

11 min read

Best Practices - Airflow 3.1.2 Documentation — writing Python code to create a Dag object, testing if the code meets your expectations, configuring environment…

This is a translated summary of the Best Practices section of the official Airflow documentation.

Updated At: 2025–11–13: Added Best Practices content for Apache Airflow-based data pipelines

Avoid ‘Top-Level’ Code

The Airflow scheduler parses DAG files very frequently. When it does, all code written at the top level of the file (not inside a Python function) is executed every time.

  • Top-Level Code: code outside the Dag context
import pendulum
from airflow.sdk import DAG
from airflow.sdk import task
import time
import pandas as pd  # 1. Importing a heavy library at the top level

def expensive_api_call():
    """A heavy task that takes 5 seconds (e.g., API call, DB query)"""
    print("Calling API...")
    time.sleep(5)
    return "API result"

my_expensive_response = expensive_api_call()

with DAG(
    dag_id="bad_example_dag",
    schedule=None,
    start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
) as dag:

    @task()
    def print_expensive_api_call():
        # 3. Use the value that has already been computed
        print(my_expensive_response)

Idempotency: Implement Tasks Idempotently

Treat a Task like a “database transaction”. A Task may be retried (re-run) many times, and it must guarantee the same result no matter how many times it runs. This is called idempotency.

“Tasks must be designed to be reproducible” — they must be idempotent and deterministic

Prevent Duplicate Data

Use UPSERT (UPDATE + INSERT) instead of INSERT. With INSERT alone, data can pile up as duplicates on re-runs

Use a Specific Partition

Implement the Task so that it reads and processes a specific partition of data rather than “the latest data”

If the source data changes between retries, the result will differ. Instead, use a logical time such as data_interval_start (Airflow 2.2+) or execution_date (Airflow < 2.2) to explicitly read and write a specific data partition.

Airflow 스케줄링과 Execution Date 동작 원리 — Apache Airflow 스케줄링 시 Execution Date 설정할 때 스케줄되어 실행되는 DAG와 backfill, trigger로 실행되는 DAG에 설정되는 Execution_date 차이를 정리

In other words, if you base logic on the current time (datetime.now()), each run can produce a different result, so you should always use the DAG’s logical execution time.

This doesn’t matter for Tasks that do a Full Refresh (drop-create)

Implement Tasks Deterministically

Deterministic: always returns the same output for the same input (like a pure function in FP)

  • Non-deterministic behavior:
    • Depending on the implicit ordering of data or data structures within a function (e.g., depending on dict order)
    • Using external state, including random values, global variables, external data stored on disk, etc.
    • Processing data in parallel without specifying an explicit order for the results
    • Race conditions within multi-threaded code
    • Improper exception handling

ex.) Applying explicit sorting to the data prevents non-determinism in dataset ordering

Using Variables and Connections

Airflow lets you manage and use DB connections and variables dynamically through Variables and Connections.

Using Airflow Variables Variable.get("my_var") in top-level code also sends a network request to the Airflow DB every time the DAG is parsed.

from airflow.models import Variable
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

# (X) DB access on every DAG parse (Top Level)
SECRET_KEY = Variable.get("my_secret_key") 

k8s_pod_op = KubernetesPodOperator(
    task_id="bad_pod_example",
    cmds=["echo", SECRET_KEY],
    # ...
)

Use Jinja Templates

If you use a Jinja template ({{ ... }}) in an Operator parameter, the variable value is fetched at the time the Task actually runs. (lazy loading)

Call Inside the Task

Call Variable.get() inside the function passed as a PythonOperator’s python_callable

from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

# (O) Using a Jinja template fetches the variable when the Task runs
k8s_pod_op = KubernetesPodOperator(
    task_id="good_pod_example",
    cmds=["echo", "{{ var.value.my_secret_key }}"], 
    # ...
)

Call Outside the Task

If you need to use Connection or Variable outside a Task, it is best to store them in the Airflow DataStore and have the external system look them up from a connection store (SQL), following the DRY principle

DRY: Don’t Repeat Yourself — avoid duplication to minimize changes

Passing Data Between Tasks (XCom vs. Remote Storage)

Airflow can pass data between Tasks using XCom.

However, XCom is written to the DB in JSON format, so when dealing with very large values it is better to pass a file path instead.

Use Remote Storage Instead of the Local Filesystem

In Celery or Kubernetes Executor environments, Tasks run on different servers/containers. A file that one Task saved locally cannot be read by the next Task.

It is better to use remote storage (S3, HDFS, etc.).

  • Recommended pattern:
    1. Task A processes/saves data to S3
    2. Push the saved S3 path (s3://...) via XCom
    3. Task B pulls that path from XCom and reads the data from S3

DAG Testing and Code Quality

Treat Dags as production-level code. Dags should include a variety of relevant tests (unit tests, …) that verify they produce the expected results.

DAG Load Test

Before deploying a DAG file, you should at least verify that there are no syntax or import errors with the following commands

$ python dags/my_dag_file.py

$ time dags/my_dag_file.py # lets you check the execution time

Unit Test

You can use DagBag to write tests that check whether the DAG loads correctly and has the intended number of Tasks and dependencies

import glob
import os
import pytest
from airflow.models import DagBag

DAG_PATH = os.path.join(os.path.dirname(__file__), "../src/dags")
DAG_FILES = glob.glob(DAG_PATH, recursive=True)
@pytest.fixture
def dag_bag():
    return DagBag(dag_folder=DAG_PATH, include_examples=False)

def test_dag_loaded(dag_bag):

    assert dag_bag.import_errors == {}
  • include_examples: ignore the example DAGs

Linting (Code Inspection)

Using a linter such as ruff with the ruff-airflow plugin, you can detect and fix upcoming Airflow changes and inefficient patterns in advance

Ruff — An extremely fast Python linter and code formatter, written in Rust.

$ pip install "ruff>=0.14.3"
$ ruff check dags/ --select AIR3

Consistent Style in Airflow

Using a consistent style when defining DAG and Task dependencies is good for readability

ex.) Pick one consistent style: with DAG() as dag or dag = DAG()

Use Factory Functions for Common Patterns

Common data processing that requires the same steps and transformations and is repeated across many DAGs can be implemented with a factory function that generates the DAG structure

# task_factory.py

def generate_tasks(
    dataset_name, raw_dir, processed_dir, preprocess_script, output_dir, dag
) -> tuple[BaseOperator, BaseOperator]:
    raw_path = os.path.join(raw_dir, dataset_name, "{ds_nodash}.json")
    processed_path = os.path.join(processed_dir, dataset_name, "{ds_nodash}.json")
    output_path = os.path.join(output_dir, dataset_name, "{ds_nodash}.json")

    fetch_task = BashOperator(
        task_id=f"fetch_{dataset_name}",
        bash_command=f"echo 'curl http://example.com/{dataset_name}.json > {raw_path}.json'",
        dag=dag,
    )

    preprocess_task = BashOperator(
        task_id=f"preprocess_{dataset_name}",
        bash_command=f"echo '{preprocess_script} {raw_path} {processed_path}'",
        dag=dag,
    )

    export_task = BashOperator(
        task_id=f"export_{dataset_name}",
        bash_command=f"echo 'cp {processed_path} {output_path}'",
        dag=dag,
    )

    fetch_task >> preprocess_task >> export_task

    # Return first and last task so they can be connected to a larger graph
    return fetch_task, export_task

Data Quality Checks (Self-Checks)

Right after a Task finishes, add a step that uses S3KeySensor, SqlSensor, etc. to check that the previous Task’s output (files, DB data) was created correctly, thereby verifying Data Quality.

Great Expectations: have confidence in your data, no matter what — Explore how our end-to-end SaaS solution for your data quality process and unique Expectation-based approach to testing…

Detecting DAG Failures Accurately with the ‘Watcher Pattern’

After data processing finishes, you sometimes want to always run a teardown Task that cleans up the resources you used (temp files, clusters, etc.). For this you use trigger_rule=TriggerRule.ALL_DONE. This rule means “run no matter whether the preceding Tasks succeeded or failed.”

  • The problem:
    1. main_task fails.
    2. teardown_task (ALL_DONE) runs even though it sees this failure.
    3. teardown_task succeeds.

Result: since teardown_task is the last Task in the DAG, Airflow marks the entire DAG Run as ‘Success’.

To solve this, we add an empty Task that plays the role of a ‘Watcher’.

Watcher: If any other Task fails, it also fails together with it, making the overall DAG state ‘Failed’. Set its trigger_rule to TriggerRule.ONE_FAILED. (Runs if any one of the preceding Tasks fails)

Airflow TriggerRule & Watcher Pattern — 이번 글에서는 헷갈렸던 Trigger_Rule에 대해서 정리하며, 여기서 마주친 문제를 Watcher Pattern을 사용하여 해결하는 방법에 대해서 설명하겠습니다. Airflow: 2.9.1

Handling Conflicting or Complex Python Dependencies

Because Airflow has many Python dependencies, sometimes Airflow’s dependencies conflict with the dependencies your task code expects.

By default, an Airflow environment consists of a single set of Python dependencies and a single Python environment, so it is common for some tasks to need different dependencies than others, and for those dependencies to conflict between tasks.

In general, Airflow is more efficient when used only as an orchestration tool. It is better to separate data processing out to external systems (Spark, …).

Using PythonVirtualenvOperator

With PythonVirtualenvOperator (or @task.virtualenv), you can dynamically create a virtual environment in which a Python callable runs

airflow.providers.standard.operators.python - apache-airflow-providers-standard Documentation — Attributes Bases: airflow.providers.standard.version_compat.BaseOperator Executes a Python callable. When running your…

  1. Creates a virtual environment according to the user’s environment
  2. Serializes the Python callable and passes it to the virtualenv Python interpreter for execution.
  3. Runs it, retrieves the callable’s result, and pushes it via xcom if specified.

Using ExternalPythonOperator

Added in version 2.4, this is a somewhat more complex method, but with far fewer overhead, security, and stability issues. (@task.external_python)

airflow.providers.standard.operators.python - apache-airflow-providers-standard Documentation — Attributes Bases: airflow.providers.standard.version_compat.BaseOperator Executes a Python callable. When running your…

Instead of creating a virtual environment, the Task runs in a pre-existing external virtual environment. This avoids the overhead and problems of recreating a virtual environment, but it must be prepared and deployed together with the Airflow installation.

@task.external_python(task_id="external_python", python=PATH_TO_PYTHON_BINARY)
def callable_external_python():
    """
    Example function that will be performed in a virtual environment.

    Importing at the module level ensures that it will not attempt to import the
    library before it is installed.
    """
    import sys
    from time import sleep

    print(f"Running task via {sys.executable}")
    print("Sleeping")
    for _ in range(4):
        print("Please wait...", flush=True)
        sleep(1)
    print("Finished")

external_python_task = callable_external_python()

DockerOperator / KubernetesPodOperator

You can manage the target script you need to run as a Docker image and run it as a container from Airflow using DockerOperator.

# Dockerfile
FROM python:3.8-slim

RUN python -m pip install click==7.1.1 requests==2.23.0

COPY scripts/fetch_ratings.py /usr/local/bin/fetch-ratings
RUN chmod +x /usr/local/bin/fetch-ratings

# Set PATH so the fetch-ratings command can be used inside the container
ENV PATH="/usr/local/bin:${PATH}"
# dag

...
fetch_ratings = DockerOperator(
        task_id="fetch_ratings",
        image="manning-airflow/movielens-fetch",
        auto_remove="force",
        mount_tmp_dir=False,
        command=[
            "fetch-ratings",
            "--start_date",
            "{{ds}}",
            "--end_date",
            "{{next_ds}}",
            "--output_path",
            "/data/ratings/{{ds}}.json",
            "--user",
            "airflow",
            "--password",
            "airflow",
            "--host",
            "http://host.docker.internal:5001",
        ],
        network_mode="airflow",
        mounts=[
            Mount(
                source="/tmp/airflow/data",  # host path
                target="/data",  # container path
                type="bind",  # specify bind mount type
            )
        ],
    )
...

Caution When Deleting a Task

If you simply delete (or comment out) a Task that is no longer needed (e.g., old_task) from the DAG file of a running DAG, old_task disappears from the UI, and there is no longer any way in the UI to access all of that Task’s past run records (logs, state, history)

If preserving past run history matters, instead of deleting the Task, choose to ‘version up’ the DAG

  1. Copy the existing DAG (my_dag_v1) code and create a new DAG file (with a new dag_id) called my_dag_v2.
  2. In my_dag_v2, delete the Tasks you want to remove or modify the logic.
  3. Change the existing my_dag_v1 to the ‘Pause’ state.
  4. Deploy the new my_dag_v2 in the ‘Active’ state.

This way, past history can always be viewed in my_dag_v1, and the new logic is handled by my_dag_v2, preserving both history and stability.

Efficient Data Processing

It is best to limit processing to the minimum amount of data needed to get the desired result.

Limit the Amount of Data Processed

When implementing a pipeline that stores specific users and their order information

If you operate on the table after a join, unnecessary data beyond the specific users gets loaded.

  1. filter: … from user where user_id in ()
  2. join: user_t join order_t on user_t.id = order_t.user_id

Cache Intermediate Data

When building a pipeline, each Task often passes data downstream. In this case, it is good to cache the intermediate step in a file (S3, …), a database, or XCom so that only the failed downstream can be re-run.

Resource Management

When working with large volumes of data, you can sometimes exceed the limits of the Airflow cluster or other systems used for data processing.

Use Pools When Running Many Tasks in Parallel

Airflow can control concurrency within Airflow using max_active_runs, max_active_tasks, etc.

Using a pool to limit access to Spark or external systems, you can keep from exceeding the limits of those other systems.

PythonOperator(
    task_id="my_task",
    python_callable=my_function,
    pool="spark_pool",  # can be defined through the web UI
)

Detect Long-Running Tasks with SLAs and Alerts

A task or DAG may take longer than usual to run. To detect such situations, you can set an SLA (service level agreement) and receive an alert when the SLA is violated.

default_args = {
    "sla": timedelta(minutes=30),  # A task running longer than 30 minutes is treated as an SLA violation
}

with DAG(
    dag_id="my_dag",
    default_args=default_args,
    "sla_miss_callback": my_sla_miss_callback,  # Callback function to call on SLA violation
schedule_interval = "@daily",
) as dag:
...

Maintaining the Airflow Metadata DB

Airflow stores all DAG run history, Task run states, logs, XCom values, and more in a central metadata DB. After running for a year or two, this data can grow to tens or hundreds of GB, which leads to degraded Airflow performance.

airflow db clean CLI

Airflow provides a built-in CLI command to clean up old run records

This command permanently deletes data, so before running it in production you should take a DB backup (snapshot) and test first with the --dry-run option.

Step 1: Dry Run (Simulation)

First, add the --dry-run flag to check only “how many rows would be deleted from which tables” without actually deleting anything.

# Only check how many dag_run, task_instance, log, and task_fail records
# older than 90 days would be deleted, without actually deleting them
$ airflow db clean \
    --tables dag_run task_instance log task_fail \
    --clean-before-days 90 \
    --dry-run

Step 2: Actual Run

Once you’ve looked at the --dry-run result and decided “it’s fine to delete this much,” back up the DB and run the same command without the --dry-run flag. (If needed, you can add --skip-archiving to skip creating a backup file.)

$ airflow db clean \
    --tables dag_run task_instance log task_fail \
    --clean-before-days 90

Previous posts

View all posts →

Comments