Airflow Best Practices
11 min read

This is a translated summary of the Best Practices section of the official Airflow documentation.
Updated At: 2025–11–13: Added Best Practices content for Apache Airflow-based data pipelines
Avoid ‘Top-Level’ Code
The Airflow scheduler parses DAG files very frequently. When it does, all code written at the top level of the file (not inside a Python function) is executed every time.
- Top-Level Code: code outside the Dag context
import pendulum
from airflow.sdk import DAG
from airflow.sdk import task
import time
import pandas as pd # 1. Importing a heavy library at the top level
def expensive_api_call():
"""A heavy task that takes 5 seconds (e.g., API call, DB query)"""
print("Calling API...")
time.sleep(5)
return "API result"
my_expensive_response = expensive_api_call()
with DAG(
dag_id="bad_example_dag",
schedule=None,
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
) as dag:
@task()
def print_expensive_api_call():
# 3. Use the value that has already been computed
print(my_expensive_response)
Idempotency: Implement Tasks Idempotently
Treat a Task like a “database transaction”. A Task may be retried (re-run) many times, and it must guarantee the same result no matter how many times it runs. This is called idempotency.
“Tasks must be designed to be reproducible” — they must be idempotent and deterministic
Prevent Duplicate Data
Use
UPSERT(UPDATE + INSERT) instead ofINSERT. WithINSERTalone, data can pile up as duplicates on re-runs
Use a Specific Partition
Implement the Task so that it reads and processes a specific partition of data rather than “the latest data”
If the source data changes between retries, the result will differ. Instead, use a logical time such as data_interval_start (Airflow 2.2+) or execution_date (Airflow < 2.2) to explicitly read and write a specific data partition.
In other words, if you base logic on the current time (datetime.now()), each run can produce a different result, so you should always use the DAG’s logical execution time.
This doesn’t matter for Tasks that do a Full Refresh (drop-create)
Implement Tasks Deterministically
Deterministic: always returns the same output for the same input (like a pure function in FP)
- Non-deterministic behavior:
- Depending on the implicit ordering of data or data structures within a function (e.g., depending on dict order)
- Using external state, including random values, global variables, external data stored on disk, etc.
- Processing data in parallel without specifying an explicit order for the results
- Race conditions within multi-threaded code
- Improper exception handling
ex.) Applying explicit sorting to the data prevents non-determinism in dataset ordering
Using Variables and Connections
Airflow lets you manage and use DB connections and variables dynamically through Variables and Connections.
Using Airflow Variables Variable.get("my_var") in top-level code also sends a network request to the Airflow DB every time the DAG is parsed.
from airflow.models import Variable
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
# (X) DB access on every DAG parse (Top Level)
SECRET_KEY = Variable.get("my_secret_key")
k8s_pod_op = KubernetesPodOperator(
task_id="bad_pod_example",
cmds=["echo", SECRET_KEY],
# ...
)
Use Jinja Templates
If you use a Jinja template (
{{ ... }}) in an Operator parameter, the variable value is fetched at the time the Task actually runs. (lazy loading)
Call Inside the Task
Call
Variable.get()inside the function passed as a PythonOperator’spython_callable
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
# (O) Using a Jinja template fetches the variable when the Task runs
k8s_pod_op = KubernetesPodOperator(
task_id="good_pod_example",
cmds=["echo", "{{ var.value.my_secret_key }}"],
# ...
)
Call Outside the Task
If you need to use
ConnectionorVariableoutside a Task, it is best to store them in the Airflow DataStore and have the external system look them up from a connection store (SQL), following the DRY principle
DRY: Don’t Repeat Yourself — avoid duplication to minimize changes
Passing Data Between Tasks (XCom vs. Remote Storage)
Airflow can pass data between Tasks using XCom.
However, XCom is written to the DB in JSON format, so when dealing with very large values it is better to pass a file path instead.
Use Remote Storage Instead of the Local Filesystem
In Celery or Kubernetes Executor environments, Tasks run on different servers/containers. A file that one Task saved locally cannot be read by the next Task.
It is better to use remote storage (S3, HDFS, etc.).
- Recommended pattern:
- Task A processes/saves data to S3
- Push the saved S3 path (
s3://...) via XCom - Task B pulls that path from XCom and reads the data from S3
DAG Testing and Code Quality
Treat Dags as production-level code. Dags should include a variety of relevant tests (unit tests, …) that verify they produce the expected results.
DAG Load Test
Before deploying a DAG file, you should at least verify that there are no syntax or
importerrors with the following commands
$ python dags/my_dag_file.py
$ time dags/my_dag_file.py # lets you check the execution time
Unit Test
You can use
DagBagto write tests that check whether the DAG loads correctly and has the intended number of Tasks and dependencies
import glob
import os
import pytest
from airflow.models import DagBag
DAG_PATH = os.path.join(os.path.dirname(__file__), "../src/dags")
DAG_FILES = glob.glob(DAG_PATH, recursive=True)
@pytest.fixture
def dag_bag():
return DagBag(dag_folder=DAG_PATH, include_examples=False)
def test_dag_loaded(dag_bag):
assert dag_bag.import_errors == {}
- include_examples: ignore the example DAGs
Linting (Code Inspection)
Using a linter such as
ruffwith theruff-airflowplugin, you can detect and fix upcoming Airflow changes and inefficient patterns in advance
Ruff — An extremely fast Python linter and code formatter, written in Rust.
$ pip install "ruff>=0.14.3"
$ ruff check dags/ --select AIR3
Consistent Style in Airflow
Using a consistent style when defining DAG and Task dependencies is good for readability
ex.) Pick one consistent style: with DAG() as dag or dag = DAG()
Use Factory Functions for Common Patterns
Common data processing that requires the same steps and transformations and is repeated across many DAGs can be implemented with a factory function that generates the DAG structure
# task_factory.py
def generate_tasks(
dataset_name, raw_dir, processed_dir, preprocess_script, output_dir, dag
) -> tuple[BaseOperator, BaseOperator]:
raw_path = os.path.join(raw_dir, dataset_name, "{ds_nodash}.json")
processed_path = os.path.join(processed_dir, dataset_name, "{ds_nodash}.json")
output_path = os.path.join(output_dir, dataset_name, "{ds_nodash}.json")
fetch_task = BashOperator(
task_id=f"fetch_{dataset_name}",
bash_command=f"echo 'curl http://example.com/{dataset_name}.json > {raw_path}.json'",
dag=dag,
)
preprocess_task = BashOperator(
task_id=f"preprocess_{dataset_name}",
bash_command=f"echo '{preprocess_script} {raw_path} {processed_path}'",
dag=dag,
)
export_task = BashOperator(
task_id=f"export_{dataset_name}",
bash_command=f"echo 'cp {processed_path} {output_path}'",
dag=dag,
)
fetch_task >> preprocess_task >> export_task
# Return first and last task so they can be connected to a larger graph
return fetch_task, export_task
Data Quality Checks (Self-Checks)
Right after a Task finishes, add a step that uses
S3KeySensor,SqlSensor, etc. to check that the previous Task’s output (files, DB data) was created correctly, thereby verifying Data Quality.
Detecting DAG Failures Accurately with the ‘Watcher Pattern’
After data processing finishes, you sometimes want to always run a
teardownTask that cleans up the resources you used (temp files, clusters, etc.). For this you usetrigger_rule=TriggerRule.ALL_DONE. This rule means “run no matter whether the preceding Tasks succeeded or failed.”

- The problem:
main_taskfails.teardown_task(ALL_DONE) runs even though it sees this failure.teardown_tasksucceeds.
Result: since teardown_task is the last Task in the DAG, Airflow marks the entire DAG Run as ‘Success’.
To solve this, we add an empty Task that plays the role of a ‘Watcher’.
Watcher: If any other Task fails, it also fails together with it, making the overall DAG state ‘Failed’. Set its trigger_rule to TriggerRule.ONE_FAILED. (Runs if any one of the preceding Tasks fails)
Handling Conflicting or Complex Python Dependencies
Because Airflow has many Python dependencies, sometimes Airflow’s dependencies conflict with the dependencies your task code expects.
By default, an Airflow environment consists of a single set of Python dependencies and a single Python environment, so it is common for some tasks to need different dependencies than others, and for those dependencies to conflict between tasks.
In general, Airflow is more efficient when used only as an orchestration tool. It is better to separate data processing out to external systems (Spark, …).
Using PythonVirtualenvOperator
With PythonVirtualenvOperator (or
@task.virtualenv), you can dynamically create a virtual environment in which a Python callable runs
- Creates a virtual environment according to the user’s environment
- Serializes the Python callable and passes it to the virtualenv Python interpreter for execution.
- Runs it, retrieves the callable’s result, and pushes it via xcom if specified.
Using ExternalPythonOperator
Added in version 2.4, this is a somewhat more complex method, but with far fewer overhead, security, and stability issues. (
@task.external_python)
Instead of creating a virtual environment, the Task runs in a pre-existing external virtual environment. This avoids the overhead and problems of recreating a virtual environment, but it must be prepared and deployed together with the Airflow installation.
@task.external_python(task_id="external_python", python=PATH_TO_PYTHON_BINARY)
def callable_external_python():
"""
Example function that will be performed in a virtual environment.
Importing at the module level ensures that it will not attempt to import the
library before it is installed.
"""
import sys
from time import sleep
print(f"Running task via {sys.executable}")
print("Sleeping")
for _ in range(4):
print("Please wait...", flush=True)
sleep(1)
print("Finished")
external_python_task = callable_external_python()
DockerOperator / KubernetesPodOperator
You can manage the target script you need to run as a Docker image and run it as a container from Airflow using DockerOperator.
# Dockerfile
FROM python:3.8-slim
RUN python -m pip install click==7.1.1 requests==2.23.0
COPY scripts/fetch_ratings.py /usr/local/bin/fetch-ratings
RUN chmod +x /usr/local/bin/fetch-ratings
# Set PATH so the fetch-ratings command can be used inside the container
ENV PATH="/usr/local/bin:${PATH}"
# dag
...
fetch_ratings = DockerOperator(
task_id="fetch_ratings",
image="manning-airflow/movielens-fetch",
auto_remove="force",
mount_tmp_dir=False,
command=[
"fetch-ratings",
"--start_date",
"{{ds}}",
"--end_date",
"{{next_ds}}",
"--output_path",
"/data/ratings/{{ds}}.json",
"--user",
"airflow",
"--password",
"airflow",
"--host",
"http://host.docker.internal:5001",
],
network_mode="airflow",
mounts=[
Mount(
source="/tmp/airflow/data", # host path
target="/data", # container path
type="bind", # specify bind mount type
)
],
)
...
Caution When Deleting a Task
If you simply delete (or comment out) a Task that is no longer needed (e.g.,
old_task) from the DAG file of a running DAG,old_taskdisappears from the UI, and there is no longer any way in the UI to access all of that Task’s past run records (logs, state, history)
If preserving past run history matters, instead of deleting the Task, choose to ‘version up’ the DAG
- Copy the existing DAG (
my_dag_v1) code and create a new DAG file (with a newdag_id) calledmy_dag_v2. - In
my_dag_v2, delete the Tasks you want to remove or modify the logic. - Change the existing
my_dag_v1to the ‘Pause’ state. - Deploy the new
my_dag_v2in the ‘Active’ state.
This way, past history can always be viewed in my_dag_v1, and the new logic is handled by my_dag_v2, preserving both history and stability.
Efficient Data Processing
It is best to limit processing to the minimum amount of data needed to get the desired result.
Limit the Amount of Data Processed
When implementing a pipeline that stores specific users and their order information
If you operate on the table after a join, unnecessary data beyond the specific users gets loaded.
- filter:
… from user where user_id in () - join:
user_t join order_t on user_t.id = order_t.user_id
Cache Intermediate Data
When building a pipeline, each Task often passes data downstream. In this case, it is good to cache the intermediate step in a file (S3, …), a database, or XCom so that only the failed downstream can be re-run.
Resource Management
When working with large volumes of data, you can sometimes exceed the limits of the Airflow cluster or other systems used for data processing.
Use Pools When Running Many Tasks in Parallel
Airflow can control concurrency within Airflow using
max_active_runs,max_active_tasks, etc.
Using a pool to limit access to Spark or external systems, you can keep from exceeding the limits of those other systems.
PythonOperator(
task_id="my_task",
python_callable=my_function,
pool="spark_pool", # can be defined through the web UI
)
Detect Long-Running Tasks with SLAs and Alerts
A task or DAG may take longer than usual to run. To detect such situations, you can set an SLA (service level agreement) and receive an alert when the SLA is violated.
default_args = {
"sla": timedelta(minutes=30), # A task running longer than 30 minutes is treated as an SLA violation
}
with DAG(
dag_id="my_dag",
default_args=default_args,
"sla_miss_callback": my_sla_miss_callback, # Callback function to call on SLA violation
schedule_interval = "@daily",
) as dag:
...
Maintaining the Airflow Metadata DB
Airflow stores all DAG run history, Task run states, logs, XCom values, and more in a central metadata DB. After running for a year or two, this data can grow to tens or hundreds of GB, which leads to degraded Airflow performance.
airflow db clean CLI
Airflow provides a built-in CLI command to clean up old run records
This command permanently deletes data, so before running it in production you should take a DB backup (snapshot) and test first with the --dry-run option.
Step 1: Dry Run (Simulation)
First, add the
--dry-runflag to check only “how many rows would be deleted from which tables” without actually deleting anything.
# Only check how many dag_run, task_instance, log, and task_fail records
# older than 90 days would be deleted, without actually deleting them
$ airflow db clean \
--tables dag_run task_instance log task_fail \
--clean-before-days 90 \
--dry-run
Step 2: Actual Run
Once you’ve looked at the
--dry-runresult and decided “it’s fine to delete this much,” back up the DB and run the same command without the--dry-runflag. (If needed, you can add--skip-archivingto skip creating a backup file.)
$ airflow db clean \
--tables dag_run task_instance log task_fail \
--clean-before-days 90