Airflow UnitTest with DAG BAG
7 min read

This post explains DAG_BAG-based Airflow unit testing and CI setup using GitHub Actions. Airflow: 2.9.1
Running unit tests may require a local (Airflow) environment.
$ airflow db init
Overview
For testing in Airflow, you can test each Task through the CLI, e.g.
airflow tasks test <task_id>.
Today, we’ll write DAG_BAG-based unit tests and even build a GitHub Actions-based CI environment.
GitHub - seon99il/airflow-boilerplate — BoilerPlate seon99il/airflow-boilerplate
That repository has sample code for CI and DagBag-based unit tests.
What is DAG BAG?
As the name suggests, a DagBag is a bag that holds all of Airflow’s DAG (Directed Acyclic Graph) objects.
The Airflow scheduler and webserver periodically scan (parse) every Python file in dags_folder. They collect all valid DAG objects found in the process and keep them in an in-memory collection called a DagBag.
- If a DAG file has a Python syntax error or fails to import, the
DagBagcannot load that file and records it as animport_error
A DAG unit test using DagBag is about checking: “Can my DAG file at least be loaded properly by Airflow?”
Unit Test
apache-airflowandpytestmust be installed in the environment where the tests run.
$ pip install apache-airflow
$ pip install pytest
# Structure
.
├── README.md
├── pytest.ini # Test environment settings
├── src # Source Root Dir
│ ├── Dockerfile
│ ├── __init__.py
│ ├── config
│ ├── dags
│ ├── data
│ ├── docker-compose.yaml
│ ├── logs
│ ├── plugins
│ ├── requirements-base.txt
│ └── requirements.txt
├── tests # Test Root Dir
│ ├── test_dags.py
│ └── test_hello_world_dag.py
└── uv.lock
DAG Loading (Smoke Test)
This is the most important and basic test. Import
DagBagdirectly and check that the target DAG file loads without errors.
import glob
import os
import pytest
from airflow.models import DagBag
DAG_PATH = os.path.join(os.path.dirname(__file__), "../src/dags")
DAG_FILE_PATH = os.path.join(DAG_PATH, "hello_world.py")
DAG_FILES = glob.glob(DAG_PATH, recursive=True)
@pytest.fixture
def dag_bag():
return DagBag(dag_folder=DAG_PATH, include_examples=False)
def test_dag_loaded(dag_bag):
assert not dag_bag.import_errors, f"DAG Import Errors: {dag_bag.import_errors}"
Checks that there is no import_error
include_examples=False: Ignores the example DAGs.pytest.fixture: Lets you pass objects as parameters
DAG Attributes and Default Settings Test
Once the DAG loads correctly, check that the detailed settings are right
Purpose of the test: Ensures that the DAG’s key metadata, such as schedule_interval, retries, and owner, is set as intended.
import glob
import os
import pytest
from airflow.models import DagBag
DAG_PATH = os.path.join(os.path.dirname(__file__), "../src/dags")
DAG_FILE_PATH = os.path.join(DAG_PATH, "hello_world.py")
DAG_FILES = glob.glob(DAG_PATH, recursive=True)
@pytest.fixture
def dag_bag():
return DagBag(dag_folder=DAG_PATH, include_examples=False)
def test_dag_loaded(dag_bag):
assert not dag_bag.import_errors, f"DAG Import Errors: {dag_bag.import_errors}"
def test_dag_id_exists_in_dagbag(dag_bag):
"""
Check that the expected DAG ID exists in the DagBag
"""
expected_dag_id = "hello_world" # ID defined in the actual DAG file
dag = dag_bag.get_dag(expected_dag_id)
assert dag is not None, f"DAG '{expected_dag_id}' not found in DagBag."
assert dag.dag_id == expected_dag_id
print(dag.tasks)
def test_dag_default_args(dag_bag):
"""
Check that the DAG's default_args are set correctly
"""
my_dag = dag_bag.get_dag("hello_world")
assert my_dag.default_args['owner'] == 'data_team'
assert my_dag.default_args.get('retries') == 3
def test_dag_schedule_and_tags(dag_bag):
"""
Check that the schedule interval, tags, etc. are set correctly
"""
my_dag = dag_bag.get_dag("hello_world")
assert my_dag.schedule_interval == "0 12 * * *"
assert "production" in my_dag.tags
assert my_dag.catchup == False
- DAG: You can fetch a specific DAG with
dagbag.get_dag("hello_world")
Task Count and Existence Test
Check that the Tasks in the DAG have not been dropped or added.
Purpose of the test: Prevents Tasks from being accidentally deleted, or unintended Tasks from being added, during refactoring.
import os
import pytest
from airflow.models import DagBag
DAG_PATH = os.path.join(os.path.dirname(__file__), "../src/dags")
@pytest.fixture
def dag_bag():
return DagBag(dag_folder=DAG_PATH, include_examples=False)
@pytest.fixture
def hello_world_dag(dag_bag):
return dag_bag.get_dag("hello_world")
def test_task_count(hello_world_dag):
"""
Check that the total number of Tasks defined in the DAG is correct
"""
assert len(hello_world_dag.tasks) == 2
def test_tasks_exist(hello_world_dag):
"""
Check that the required Task IDs exist in the DAG
"""
expected_task_ids = ["print_hello", "print_hello2"]
# task_ids is a list
for task_id in expected_task_ids:
assert task_id in hello_world_dag.task_ids
Task Dependency (Structure) Test
Check that the order (dependencies) between Tasks, the core of a DAG, is set correctly
Purpose of the test: Verifies that the flow (graph) of the data pipeline matches the business logic.
def test_task_dependencies(hello_world_dag):
"""
Check that the upstream/downstream relationships between Tasks are correct
"""
process_task = hello_world_dag.get_task("print_hello")
assert len(process_task.upstream_task_ids) == 0
assert "print_hello2" in process_task.downstream_task_ids
Operator Detail Settings Test
Check that the arguments used by an individual Task (Operator) (e.g.,
bash_command,sql) are correct.
Purpose of the test: Checks that the Operator’s core logic (the command to run, the query, etc.) is set accurately.
def test_bash_operator_command(hello_world_dag):
"""
Check that the Operator is configured correctly
"""
print_hello_task = hello_world_dag.get_task("print_hello")
assert print_hello_task.operator_name == "PythonOperator"
Task Internal Logic Test (Pure Unit Test)
If you run a complex Python function using the
@taskdecorator (TaskFlow API) orPythonOperator, you can test that function on its own.
Purpose of the test: Tests only the function’s business logic, in isolation from the Airflow framework.
def test_dag_loaded():
from dags.hello_world import return_name
expected_name = "John Doe"
assert return_name(expected_name) == expected_name
- To resolve import errors, you need to set
pythonpathin pytest.ini, and it is safer to run pytest from the project root.
# ./pytest.ini
[pytest]
pythonpath = . src
CI Workflow with GitHub Actions
Now let’s write the CI logic based on the tests we wrote.
name: Airflow DAG CI
on:
push:
branches: [ "main", "dev" ]
pull_request:
branches: [ "main", "dev" ]
jobs:
ci-test:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10.12"
- name: Set commit range for diff
id: commit-range
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
echo "range=${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}" >> $GITHUB_OUTPUT
echo "Checking PR changes: ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}"
elif [ "${{ github.event_name }}" = "push" ]; then
echo "range=${{ github.event.before }}...${{ github.event.after }}" >> $GITHUB_OUTPUT
echo "Checking push changes: ${{ github.event.before }}...${{ github.event.after }}"
else
echo "range=HEAD~1...HEAD" >> $GITHUB_OUTPUT
echo "Defaulting to changes in last commit"
fi
- name: 🔎 Check for changed requirements
id: changed-reqs
run: |
CHANGED_FILES=$(git diff --name-only ${{ steps.commit-range.outputs.range }} -- src/requirements-base.txt src/requirements.txt || true)
if [ -z "$CHANGED_FILES" ]; then
echo "✅ No requirement files changed in this commit range."
echo "Cache *should* be a HIT (if a cache exists)."
else
echo "⚠️ Found changed requirement files (cache *may* be a MISS):"
echo "$CHANGED_FILES"
fi
- name: Get pip cache dir
id: pip-cache
run: |
echo "dir=$(pip cache dir)" >> $GITHUB_OUTPUT
- name: Cache base dependencies
uses: actions/cache@v4
id: cache-base
with:
path: ${{ steps.pip-cache.outputs.dir }}
key: ${{ runner.os }}-pip-base-${{ hashFiles('src/requirements-base.txt') }}
restore-keys: |
${{ runner.os }}-pip-base-
- name: Cache all dependencies
uses: actions/cache@v4
id: cache-full
with:
path: ${{ steps.pip-cache.outputs.dir }}
key: ${{ runner.os }}-pip-full-${{ hashFiles('src/requirements-base.txt', 'src/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-base-${{ hashFiles('src/requirements-base.txt') }}
${{ runner.os }}-pip-base-
- name: Install dependencies
working-directory: src
run: |
python -m pip install --upgrade pip
pip install -r requirements-base.txt
pip install -r requirements.txt
- name: Mock Airflow Env
run: |
airflow db init
- name: Run pytest
run: |
pytest -v --disable-warnings
- name: 📋 Report cache status
if: always()
run: |
echo "--- Cache Status Report ---"
echo "Base Cache (requirements-base.txt): ${{ steps.cache-base.outputs.cache-hit == 'true' && '✅ HIT' || '❌ MISS' }}"
echo "Full Cache (requirements.txt): ${{ steps.cache-full.outputs.cache-hit == 'true' && '✅ HIT' || '❌ MISS' }}"
echo "---------------------------"
- Airflow’s Python dependencies rarely change, so they are split into
requirements-base.txtand cached - You need to set up the metadata database with
airflow db init