seon99il

Python

Airflow UnitTest with DAG BAG

7 min read

This post explains DAG_BAG-based Airflow unit testing and CI setup using GitHub Actions. Airflow: 2.9.1

Running unit tests may require a local (Airflow) environment.

$ airflow db init

Overview

For testing in Airflow, you can test each Task through the CLI, e.g. airflow tasks test <task_id>.

Today, we’ll write DAG_BAG-based unit tests and even build a GitHub Actions-based CI environment.

GitHub - seon99il/airflow-boilerplate — BoilerPlate seon99il/airflow-boilerplate

That repository has sample code for CI and DagBag-based unit tests.

What is DAG BAG?

As the name suggests, a DagBag is a bag that holds all of Airflow’s DAG (Directed Acyclic Graph) objects.

The Airflow scheduler and webserver periodically scan (parse) every Python file in dags_folder. They collect all valid DAG objects found in the process and keep them in an in-memory collection called a DagBag.

  • If a DAG file has a Python syntax error or fails to import, the DagBag cannot load that file and records it as an import_error

A DAG unit test using DagBag is about checking: “Can my DAG file at least be loaded properly by Airflow?”

Unit Test

apache-airflow and pytest must be installed in the environment where the tests run.

$ pip install apache-airflow
$ pip install pytest
# Structure
.
├── README.md
├── pytest.ini # Test environment settings
├── src # Source Root Dir
│   ├── Dockerfile
│   ├── __init__.py
│   ├── config
│   ├── dags
│   ├── data
│   ├── docker-compose.yaml
│   ├── logs
│   ├── plugins
│   ├── requirements-base.txt
│   └── requirements.txt
├── tests # Test Root Dir
│   ├── test_dags.py
│   └── test_hello_world_dag.py
└── uv.lock

DAG Loading (Smoke Test)

This is the most important and basic test. Import DagBag directly and check that the target DAG file loads without errors.

import glob
import os
import pytest
from airflow.models import DagBag

DAG_PATH = os.path.join(os.path.dirname(__file__), "../src/dags")
DAG_FILE_PATH = os.path.join(DAG_PATH, "hello_world.py")
DAG_FILES = glob.glob(DAG_PATH, recursive=True)

@pytest.fixture
def dag_bag():
    return DagBag(dag_folder=DAG_PATH, include_examples=False)

def test_dag_loaded(dag_bag):

    assert not dag_bag.import_errors, f"DAG Import Errors: {dag_bag.import_errors}"

Checks that there is no import_error

  • include_examples=False: Ignores the example DAGs.
  • pytest.fixture: Lets you pass objects as parameters

DAG Attributes and Default Settings Test

Once the DAG loads correctly, check that the detailed settings are right

Purpose of the test: Ensures that the DAG’s key metadata, such as schedule_interval, retries, and owner, is set as intended.

import glob
import os
import pytest
from airflow.models import DagBag

DAG_PATH = os.path.join(os.path.dirname(__file__), "../src/dags")
DAG_FILE_PATH = os.path.join(DAG_PATH, "hello_world.py")
DAG_FILES = glob.glob(DAG_PATH, recursive=True)

@pytest.fixture
def dag_bag():
    return DagBag(dag_folder=DAG_PATH, include_examples=False)

def test_dag_loaded(dag_bag):

    assert not dag_bag.import_errors, f"DAG Import Errors: {dag_bag.import_errors}"

def test_dag_id_exists_in_dagbag(dag_bag):
    """
    Check that the expected DAG ID exists in the DagBag
    """
    expected_dag_id = "hello_world" # ID defined in the actual DAG file

    dag = dag_bag.get_dag(expected_dag_id)

    assert dag is not None, f"DAG '{expected_dag_id}' not found in DagBag."
    assert dag.dag_id == expected_dag_id
    print(dag.tasks)

def test_dag_default_args(dag_bag):

    """
    Check that the DAG's default_args are set correctly
    """
    my_dag = dag_bag.get_dag("hello_world")
    assert my_dag.default_args['owner'] == 'data_team'
    assert my_dag.default_args.get('retries') == 3

def test_dag_schedule_and_tags(dag_bag):
    """
    Check that the schedule interval, tags, etc. are set correctly
    """
    my_dag = dag_bag.get_dag("hello_world")
    assert my_dag.schedule_interval == "0 12 * * *"
    assert "production" in my_dag.tags
    assert my_dag.catchup == False
  • DAG: You can fetch a specific DAG with dagbag.get_dag("hello_world")

Task Count and Existence Test

Check that the Tasks in the DAG have not been dropped or added.

Purpose of the test: Prevents Tasks from being accidentally deleted, or unintended Tasks from being added, during refactoring.

import os

import pytest
from airflow.models import DagBag

DAG_PATH = os.path.join(os.path.dirname(__file__), "../src/dags")

@pytest.fixture
def dag_bag():
    return DagBag(dag_folder=DAG_PATH, include_examples=False)

@pytest.fixture
def hello_world_dag(dag_bag):
    return dag_bag.get_dag("hello_world")

def test_task_count(hello_world_dag):
    """
    Check that the total number of Tasks defined in the DAG is correct
    """
    assert len(hello_world_dag.tasks) == 2

def test_tasks_exist(hello_world_dag):
    """
    Check that the required Task IDs exist in the DAG
    """
    expected_task_ids = ["print_hello", "print_hello2"]

    # task_ids is a list
    for task_id in expected_task_ids:
        assert task_id in hello_world_dag.task_ids

Task Dependency (Structure) Test

Check that the order (dependencies) between Tasks, the core of a DAG, is set correctly

Purpose of the test: Verifies that the flow (graph) of the data pipeline matches the business logic.

def test_task_dependencies(hello_world_dag):
    """
    Check that the upstream/downstream relationships between Tasks are correct
    """
    process_task = hello_world_dag.get_task("print_hello")

    assert len(process_task.upstream_task_ids) == 0

    assert "print_hello2" in process_task.downstream_task_ids

Operator Detail Settings Test

Check that the arguments used by an individual Task (Operator) (e.g., bash_command, sql) are correct.

Purpose of the test: Checks that the Operator’s core logic (the command to run, the query, etc.) is set accurately.

def test_bash_operator_command(hello_world_dag):
    """
    Check that the Operator is configured correctly
    """

    print_hello_task = hello_world_dag.get_task("print_hello")

    assert print_hello_task.operator_name == "PythonOperator"

Task Internal Logic Test (Pure Unit Test)

If you run a complex Python function using the @task decorator (TaskFlow API) or PythonOperator, you can test that function on its own.

Purpose of the test: Tests only the function’s business logic, in isolation from the Airflow framework.

def test_dag_loaded():
    from dags.hello_world import return_name
    expected_name = "John Doe"

    assert  return_name(expected_name) == expected_name
  • To resolve import errors, you need to set pythonpath in pytest.ini, and it is safer to run pytest from the project root.
# ./pytest.ini
[pytest]
pythonpath = . src

CI Workflow with GitHub Actions

Now let’s write the CI logic based on the tests we wrote.

name: Airflow DAG CI

on:
  push:
    branches: [ "main", "dev" ]
  pull_request:
    branches: [ "main", "dev" ]

jobs:
  ci-test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.10.12"

      - name: Set commit range for diff
        id: commit-range
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            echo "range=${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}" >> $GITHUB_OUTPUT
            echo "Checking PR changes: ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}"
          elif [ "${{ github.event_name }}" = "push" ]; then
            echo "range=${{ github.event.before }}...${{ github.event.after }}" >> $GITHUB_OUTPUT
            echo "Checking push changes: ${{ github.event.before }}...${{ github.event.after }}"
          else
            echo "range=HEAD~1...HEAD" >> $GITHUB_OUTPUT
            echo "Defaulting to changes in last commit"
          fi

      - name: 🔎 Check for changed requirements
        id: changed-reqs
        run: |
          CHANGED_FILES=$(git diff --name-only ${{ steps.commit-range.outputs.range }} -- src/requirements-base.txt src/requirements.txt || true)
          
          if [ -z "$CHANGED_FILES" ]; then
            echo "✅ No requirement files changed in this commit range."
            echo "Cache *should* be a HIT (if a cache exists)."
          else
            echo "⚠️ Found changed requirement files (cache *may* be a MISS):"
            echo "$CHANGED_FILES"
          fi

      - name: Get pip cache dir
        id: pip-cache
        run: |
          echo "dir=$(pip cache dir)" >> $GITHUB_OUTPUT

      - name: Cache base dependencies
        uses: actions/cache@v4
        id: cache-base
        with:
          path: ${{ steps.pip-cache.outputs.dir }}
          key: ${{ runner.os }}-pip-base-${{ hashFiles('src/requirements-base.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-base-

      - name: Cache all dependencies
        uses: actions/cache@v4
        id: cache-full
        with:
          path: ${{ steps.pip-cache.outputs.dir }}
          key: ${{ runner.os }}-pip-full-${{ hashFiles('src/requirements-base.txt', 'src/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-base-${{ hashFiles('src/requirements-base.txt') }}
            ${{ runner.os }}-pip-base-

      - name: Install dependencies
        working-directory: src
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements-base.txt
          pip install -r requirements.txt

      - name: Mock Airflow Env
        run: |
          airflow db init

      - name: Run pytest
        run: |
          pytest -v --disable-warnings

      - name: 📋 Report cache status
        if: always()
        run: |
          echo "--- Cache Status Report ---"
          echo "Base Cache (requirements-base.txt): ${{ steps.cache-base.outputs.cache-hit == 'true' && '✅ HIT' || '❌ MISS' }}"
          echo "Full Cache (requirements.txt): ${{ steps.cache-full.outputs.cache-hit == 'true' && '✅ HIT' || '❌ MISS' }}"
          echo "---------------------------"
  • Airflow’s Python dependencies rarely change, so they are split into requirements-base.txt and cached
  • You need to set up the metadata database with airflow db init

Previous posts

View all posts →

Comments