seon99il

Data Engineering

Airflow Scheduling and How Execution Date Works

7 min read

This post sorts out the confusing parts of Airflow scheduling’s Execution Date (Logical Date and Data Interval since 2.2). Airflow: 2.9.1

Terminology

Airflow has several date concepts for scheduling, most notably start_date, end_date, and execution_date.

In Airflow, these carry slightly different meanings from the execution date concept we usually understand.

start_date

The time at which the first job is scheduled

If start_date is in the future, the Airflow DAG will not run.

end_date

The time at which the last job is scheduled

execution_date

Despite its name, execution_date does not mean the time a Task actually runs; it means the start of the Time Window that the Task intends to process.

How Execution Date Works

from datetime import datetime

from airflow import DAG

with DAG(
    dag_id="execution_date_example",
    start_date=datetime(2025, 11, 1),
    end_date=datetime(2025, 11, 10),
    catchup=False,
    schedule="0 2 * * *",
) as dag:

That is, if a job has a start_date of 2025-11-01 and runs every day at 2 AM, the DAG whose execution_date is 2025-11-01T02:00:00 runs at 2025-11-02T02:00:00.

Time the DAG was scheduled (schedule)Time the DAG actually ranExecution Date
2025-11-01 02:00:002025-11-02 02:00:002025-11-01 02:00:00
2025-11-02 02:00:002025-11-03 02:00:002025-11-02 02:00:00
2025-11-03 02:00:002025-11-04 02:00:002025-11-03 02:00:00
2025-11-04 02:00:002025-11-05 02:00:002025-11-04 02:00:00
2025-11-05 02:00:002025-11-06 02:00:002025-11-05 02:00:00
2025-11-06 02:00:002025-11-07 02:00:002025-11-06 02:00:00
2025-11-07 02:00:002025-11-08 02:00:002025-11-07 02:00:00
2025-11-08 02:00:002025-11-09 02:00:002025-11-08 02:00:00
2025-11-09 02:00:002025-11-10 02:00:002025-11-09 02:00:00
2025-11-10 02:00:002025-11-11 02:00:002025-11-10 02:00:00

start_date + schedule_interval(1 Day)

In traditional batch data processing, a job has a target period of data, and the data generated in that period is processed after the period has passed.

That is why, in the figure above, the data generated from 25-04-09T02:00:00 to 25-04-10T02:00:00 is processed with execution_date(25-04-09T02:00:00) as the reference.

How Execution Date Works in DAGs Run by Backfill or Manually

For manually run DAGs (airflow dags trigger, airflow dags backfill), execution_date behaves slightly differently from the approach described above.

I expected the Time Window concept described above to be applied as is.

Q: How do you run a DAG whose execution_date is 2025-11-01T02:00:00?

airflow dags trigger <dag_id> --exec-date <?>

  1. 2025-11-01T02:00:00
  2. 2025-11-02T02:00:00

The answer is 1. When you specify the date directly (--exec-date), the value you specify is used as is, rather than the datetime one interval ahead as in the scheduled approach.

$ airflow dags trigger execution_date_example --exec-date 2025-11-01T02:00:00

# [2025-11-04T09:55:41.391+0000] {__init__.py:43} INFO - Loaded API auth backend: airflow.api.auth.backend.basic_auth
# [2025-11-04T09:55:41.392+0000] {__init__.py:43} INFO - Loaded API auth backend: airflow.api.auth.backend.session
#      |                |                | data_interval_ | data_interval_ |          | external_trigg | last_schedulin |               |          |            |
# conf | dag_id         | dag_run_id     | start          | end            | end_date | er             | g_decision     | logical_date  | run_type | start_date | state
# =====+================+================+================+================+==========+================+================+===============+==========+============+=======
# {}   | execution_date | manual__2025-1 | 2025-10-31     | 2025-11-01     | None     | True           | None           | 2025-11-01    | manual   | None       | queued
#      | _example       | 1-01T02:00:00+ | 02:00:00+00:00 | 02:00:00+00:00 |          |                |                | 02:00:00+00:0 |          |            |
#      |                | 00:00          |                |                |          |                |                | 0             |          |            |

logical_date(execution_date): 2025-11-01T02:00:00

How execution_date Is Set in Backfill

airflow dags backfill <dag_id> -s <date> -e <date>

  • -s <date>: Start of the execution_date range to backfill
  • -e <date>: End of the execution_date range to backfill

As with the trigger command, the date you specify is used as the execution_date here too.

Caveats When Specifying a Backfill Range

If you backfill a DAG that runs every day at 2 AM using the command airflow dags backfill <dag_id> -s 2025-11-01 -e 2025-11-03, the following backfill runs are executed.

Note that if you don’t specify a time, it is filled with 00.

  1. It finds the execution_dates between 2025-11-01T00:00:00 ~ 2025-11-03T00:00:00 → [2025-11-01T02:00:00, 2025-11-02T02:00:00]
  2. The DAG runs with those dates as their execution_date are backfilled.
from datetime import datetime

from airflow import DAG
from airflow.operators.python import PythonOperator

with DAG(
    dag_id="execution_date_example",
    start_date=datetime(2025, 11, 1),
    end_date=datetime(2025, 11, 10),
    catchup=False,
    schedule="0 2 * * *",
) as dag:

    execution_task = PythonOperator(
        task_id='print_execution_date',
        python_callable=lambda **kwargs: print(f"""
        Now DateTime is: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
        Execution date is: {kwargs['ti'].execution_date}"""),
    )
$ airflow dags backfill execution_date_example -s 2025-11-01 -e 2025-11-03
[2025-11-04T09:57:00.237+0000] {dagbag.py:545} INFO - Filling up the DagBag from /opt/airflow/dags
[2025-11-04T09:57:00.637+0000] {executor_loader.py:235} INFO - Loaded executor: CeleryExecutor
... INFO - [backfill progress] | finished run 0 of 2 | ...
...
... INFO - execution_date=2025-11-02 02:00:00+00:00, run_id=backfill__2025-11-02T02:00:00+00:00, run_start_date=2025-11-04 09:57:00.679021+00:00, run_end_date=2025-11-04 09:57:10.658985+00:00, run_duration=9.979964, state=success, external_trigger=False, run_type=backfill, data_interval_start=2025-11-02 02:00:00+00:00, data_interval_end=2025-11-03 02:00:00+00:00, dag_hash=None
...
... INFO - execution_date=2025-11-01 02:00:00+00:00, run_id=manual__2025-11-01T02:00:00+00:00, run_start_date=2025-11-04 09:57:00.671024+00:00, run_end_date=2025-11-04 09:57:10.660169+00:00, run_duration=9.989145, state=success, external_trigger=True, run_type=backfill, data_interval_start=2025-10-31 02:00:00+00:00, data_interval_end=2025-11-01 02:00:00+00:00, dag_hash=709734cea349e98cdc2be5c5df0d25bd
... INFO - [backfill progress] | finished run 2 of 2 | ...

Tasks Run via Clear Task

When a DAG is run directly from the Web UI, the time of the run becomes its execution_date, and even if you re-run it with clear_task, it is re-run with the previously assigned execution_date.

Logical Date, data_interval_start, data_interval_end

Since Airflow 2.2, execution_date has been replaced by logical_date, date_interval_start, date_interval_end. This section explains how the above behaves under these names.

  • logical_date: The same value as execution_date, and it behaves the same way as described above.
  • date_interval_start: The start of the interval.
  • date_interval_end: The end of the interval.

How it works

As above, logical_date behaves the same way in manual and backfill runs. Now let’s look at how date_interval_start and end are set.

with DAG(
    dag_id="best_practices_example_dag",
    start_date=datetime(2024, 1, 1),
    schedule="0 11 * * *", # a DAG that runs every day at 11:00
    catchup=False,
) as dag:

This is easy to understand once you grasp the concepts described above.

  • manual_trigger: 2025-11-09 10:56:50: a trigger before 11:00 on the 9th Since the 11:00 data for 11-09 is not yet complete, start and end are set as follows.
    • start: 2025-11-07 11:00:00+00:00
    • end: 2025-11-08 11:00:00+00:00
  • manual_trigger: 2025-11-09 11:01:50: a trigger after 11:00 on the 9th In this case, the 11:00 data for November 9th is complete, so start and end are set as follows.
    • start: 2025-11-08 11:00:00+00:00
    • end: 2025-11-09 11:00:00+00:00

Airflow 3.x

From Airflow 3.x onward, you must set create_cron_data_intervals to True for it to behave as above.

# airflow.cfg

...

create_cron_data_intervals = True

...

Conclusion

A scheduled job runs the DAG whose execution_date is the start of its Time Window at the end of that Time Window, whereas a DAG run via trigger, the Web UI, or Backfill runs the DAG using the time you specify as its execution_date.

Previous posts

View all posts →

Comments