Add on_skipped_intervals_callback for catchup=False DAGs

[Tegh25]

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

---

closes: #66791

When a Dag is configured with catchup=False and the scheduler advances past missed intervals (for example after a restart or re-enable), the skip happens silently: no DagRun rows are created and no hook fires. Operators who need observability for skipped partitions (SLA accounting, compliance audit trails, dashboard counters) had to compare last_automated_data_interval against wall clock externally.

This PR adds two symmetric extension points:

1. on_skipped_intervals_callback — a Dag-level callback on the SDK
   DAG constructor, using the same context-based signature as
   on_success_callback / on_failure_callback. The context includes
   dag, reason ("skipped_intervals"), and skipped_range (a
   DataInterval from the previous automated run's data_interval_end to
   the new run's data_interval_start).

2. on_intervals_skipped — an AIP-61 listener hookspec for plugins
   that need in-process observability without reloading the Dag file.
   The listener receives a SkippedIntervalsSummary with the same
   skipped_range.

How it works

Skipped intervals are detected at scheduled DagRun creation time (timetable-agnostic):

• When the scheduler creates a new SCHEDULED run for a catchup=False
  Dag, it compares the new run's data_interval_start against the
  data_interval_end of the most recent prior scheduled run.
• If there is a gap, it builds a SkippedIntervalsSummary with
  skipped_range spanning those two boundaries. The scheduler does not
  enumerate every skipped interval on the hot path.
• Detection is gated: this runs only when not dag.catchup AND
  (has_on_skipped_intervals_callback is set OR an on_intervals_skipped
  listener implementation is registered).

Listener fires synchronously in the scheduler (same pattern as on_dag_run_running for Dag start events).

Callback follows the existing Dag callback pipeline:
DagSkippedIntervalsCallbackRequest → DatabaseCallbackSink → Dag File Processor reloads the Dag file and invokes the callable. The skipped gap is serialized as a (start, end) datetime pair (skipped_range) in the request because no DagRun row exists for skipped intervals and the processor cannot reliably reconstruct the gap later.

To enumerate individual skipped intervals (for example to count them or log each one), call context["dag"].iter_dagrun_infos_between(skipped_range.start, skipped_range.end) inside the callback, or the equivalent on the Dag in a listener implementation.

Files changed

Area	Files
SDK Dag definition	task-sdk/src/airflow/sdk/definitions/dag.py, task-sdk/src/airflow/sdk/definitions/context.py
Timetable types	airflow-core/src/airflow/timetables/base.py
Serialization	serialization/definitions/dag.py, serialized_objects.py
Callback transport	callbacks/callback_requests.py, dag_processing/processor.py
Supervisor schema	task-sdk/src/airflow/sdk/execution_time/schema/versions/v2026_06_23.py, schema.json
Listener	listeners/spec/dag.py (new), listeners/listener.py
Scheduler	jobs/scheduler_job_runner.py

Usage (preview)

from airflow.sdk import DAG


def notify_skipped(context):
    skipped_range = context["skipped_range"]
    print(f"Gap: {skipped_range.start} -> {skipped_range.end}")
    for info in context["dag"].iter_dagrun_infos_between(
        skipped_range.start, skipped_range.end
    ):
        if info.data_interval is None:
            continue
        interval = info.data_interval
        print(f"Skipped: {interval.start} -> {interval.end}")


with DAG(
    "my_dag",
    schedule="@daily",
    catchup=False,
    on_skipped_intervals_callback=notify_skipped,
) as dag:
    ...

Follow-up (Will be added to this PR)

• Unit and integration tests (scheduler skip detection, callback execution, listener hook, serialization roundtrip)

• User documentation (on_skipped_intervals_callback in Dag reference, on_intervals_skipped in listeners docs)

• Newsfragment in airflow-core/newsfragments/ (user-facing feature)

Test plan

To be completed before marking Ready for review:

• Scheduler creates skip callback request when catchup=False Dag jumps past intervals and callback is configured

• Listener on_intervals_skipped fires with correct intervals

• Dag File Processor executes callback with skipped_intervals in context

• No overhead when callback and listener are both unset

• Serialization roundtrip preserves has_on_skipped_intervals_callback

• prek run --from-ref main --stage pre-commit passes

---

Was generative AI tooling used to co-author this PR?

• Yes - Cursor (Auto)
  Generated-by: Cursor (Auto) following the guidelines

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Tegh25]

Hi @ferruzzi, please review when you get the chance. Thanks

[ferruzzi]

Sorry for the delay, I'll look tomorrow. The failing CI is going to have to be addressed. Run your static-checks locally for a start, then rebase main and do a force-push. We'll see what's left failing and can look at them together if you need a hand.

[ferruzzi]

Good start, left some comments and I can make another pass when you get those addresseed

[ferruzzi]

Left a question for discussion but otherwise LGTM.

[ferruzzi]

Please leave this comment open as a reminder to verify this file before we merge.

[uranusjr]

I feel we lack a discussion on this feature. Can someone raise a thread on the dev list so maintainers are aware of this?

[ferruzzi]

Looks like all of my comments have been addressed. I am leaving the Cadwyn review to @uranusjr, but once he is happy with that portion and the CI is green I'm happy to approve.

[Tegh25]

Hey @uranusjr, can you please submit a 2nd review for this PR when you have time? Thank you

[potiuk]

Thanks for the thorough work here — the callback/listener symmetry, the serialization round-trip coverage, docs, and newsfragment are all in good shape, and the earlier review points (the O(1) redesign, moving the listener call out of the retry transaction, the ignored-keys handling) look well addressed.

I think there's one correctness issue that should be resolved before this lands.

Blocking — skip detection false-fires on every run for the default timetable family (airflow-core/src/airflow/serialization/definitions/dag.py / jobs/scheduler_job_runner.py)

summarize_skipped_intervals_between reports a skip whenever prev_interval_end < new_interval_start:

if prev_interval_end >= new_interval_start:
    return None
return SkippedIntervalsSummary(skipped_range=DataInterval(prev_interval_end, new_interval_start))

That's correct only for contiguous data intervals (CronDataIntervalTimetable / DeltaDataIntervalTimetable). But the default config is scheduler.create_cron_data_intervals = False and scheduler.create_delta_data_intervals = False, so a plain schedule="@daily" or schedule=timedelta(...) resolves to CronTriggerTimetable / DeltaTriggerTimetable, whose data interval defaults to zero-width (interval=timedelta() → DataInterval(next_start, next_start)).

For a zero-width-interval timetable, consecutive normal runs are never contiguous:

• run day 1 → data_interval = [day1 00:00, day1 00:00]
• run day 2 (no downtime, nothing skipped) → data_interval = [day2 00:00, day2 00:00]
• prev.data_interval_end (day1) < new.data_interval_start (day2) → a "skip" is reported, and the callback + listener + a Dag-processor callback dispatch fire on every run.

So under the default configuration the feature fires continuously even when nothing was skipped — the opposite of the documented contract ("notified when the scheduler advances past missed intervals").

The current tests don't catch this because they hand-construct full-width contiguous intervals (data_interval=(prev_start, prev_start + 1 day) and a full-width next_dagrun_data_interval) rather than the zero-width intervals a Trigger timetable actually produces at runtime.

Suggested direction: instead of the bare prev_end < new_start comparison, ask the timetable whether at least one scheduled logical time lies strictly between prev_interval_end and new_interval_start. That's a boundary check (e.g. timetable.next_dagrun_info(...) from prev_interval_end, test whether it falls before new_interval_start) — no full enumeration, so it stays cheap. Please add a regression test using the default (Trigger) timetable with no skips that asserts the callback does not fire.

Smaller observations

• jobs/scheduler_job_runner.py — get_listener_manager().hook.on_intervals_skipped.get_hookimpls() is evaluated twice for the same dag in one scheduling loop (once in _collect_skipped_intervals, again in _create_dag_runs). Worth computing once.
• serialization/definitions/dag.py — summarize_skipped_intervals_between currently uses nothing from the DAG/timetable; addressing the point above will give it a real reason to live on the DAG.
• dag_processing/processor.py — stray blank line right after the def _execute_dag_skipped_intervals_callback(...) signature.

I'll leave the Cadwyn / supervisor-schema-versioning portion to @uranusjr, as @ferruzzi noted.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Airflow maintainer. After you've addressed the points above and pushed an update, an Apache Airflow maintainer — a real person — will take the next look at the PR. The findings cite the project's review criteria; if you think one of them is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Airflow handles maintainer review: contributing-docs/05_pull_requests.rst.

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[ferruzzi]

a plain schedule="@daily" or schedule=timedelta(...) resolves to CronTriggerTimetable / DeltaTriggerTimetable, whose data interval defaults to zero-width (interval=timedelta() → DataInterval(next_start, next_start)).

Oh, nice catch, I didn't realize that. Sneaky bug could have been a pain to troubleshoot, too.

So the right solution is to use timetable.next_dagrun_info() to calculate when the next run would have been by giving it the previous interval's end, and seeing if that next run would have been before the new_interval_start, right?

[Tegh25]

Thank you @potiuk and @ferruzzi for the comments. I implemented the recommended fix using next_dagrun_info() and addressed the smaller observations in the latest commits.

I noticed that dag-run.rst didn't include any info on trigger timetables and was quite outdated. I opened PR #70167 to update the documentation on that page.

Please take a look at this updated PR and #70167 when you have the chance. Would appreciate any further feedback.