Skip to content

Fix grid view topological sort for cross-group dependencies - #67964

Closed
alex-stiff wants to merge 6 commits into
apache:mainfrom
alex-stiff:patch-1
Closed

Fix grid view topological sort for cross-group dependencies#67964
alex-stiff wants to merge 6 commits into
apache:mainfrom
alex-stiff:patch-1

Conversation

@alex-stiff

@alex-stiff alex-stiff commented Jun 3, 2026

Copy link
Copy Markdown

When specifying group dependencies like first_stage.end >> second_stage.start, the ordering within the Airflow UI does not respect the default topological sorting.

Minimal DAG to reproduce:

from datetime import datetime
from airflow.sdk import DAG, task, TaskGroup


@task
def noop_task():
    pass


with DAG("grid_sort_bug_repro", start_date=datetime(2024, 1, 1), schedule=None, catchup=False):

    with TaskGroup("stage_b"):
        b_start = noop_task()
        b_end = noop_task()
        b_start >> b_end

    with TaskGroup("stage_a"):
        a_start = noop_task()
        a_end = noop_task()
        a_start >> a_end

    b_end >> a_start  # task-level dep, bypasses group-level relationship

This shows in the grid view UI as:

stage_a
stage_b

Which is alphabetical, rather than the desired stage_b coming before stage_a.

This is happening because the _project_child_deps function only checks child.upstream_task_ids populated explicitly. We need to also scan the upstream_task_ids of the TaskGroup's root tasks so that the tasks within the group are also visible to this function.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Important

🛠️ Maintainer triage note for @alex-stiff · by @potiuk · 2026-06-17 14:51 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed:

  • Static / docs checks failing (CI image checks / Static checks). Run them locally with prek run --all-files (or pre-commit run --all-files) and push the fixes.
  • Failing test jobs: Low dep tests:core / All-core:LowestDeps:14:3.10:Always...Core, MySQL tests: core / DB-core:MySQL:8.0:3.10:Always...Core, Postgres tests: core / DB-core:Postgres:14:3.10:Always...Core, Sqlite tests: core / DB-core:Sqlite:3.10:Always...Core. Reproduce and fix locally, then push.
  • See the Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

@boring-cyborg

boring-cyborg Bot commented Jun 3, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@ashb ashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add unit tests covering it.

@alex-stiff

Copy link
Copy Markdown
Author

Please add unit tests covering it.

Thank you, added.

@potiuk
potiuk marked this pull request as draft June 18, 2026 21:47
@potiuk

potiuk commented Jul 8, 2026

Copy link
Copy Markdown
Member

Note

🛠️ Maintainer triage note for @alex-stiff · by @potiuk · 2026-07-08 16:05 UTC

This draft has been inactive for ~20 days, so I'm closing it to keep the review queue tidy — no judgment on the work itself.

No rush — reopen it (or open a fresh PR) whenever you're ready to continue; nothing is lost.

Automated triage — may be imperfect; a maintainer takes the next look.

@potiuk potiuk closed this Jul 8, 2026
@alex-stiff

Copy link
Copy Markdown
Author

@potiuk would you mind reopening this for me please, I don't seem to have perms! Thank you.

jason810496 pushed a commit to hkc-8010/my-airflow-repository that referenced this pull request Jul 28, 2026
…deps

TaskGroup._project_child_deps only looked at a group's own upstream_task_ids,
which stays empty for a direct group-to-group dependency (list or individual
`>>`) and for a task-level dependency that crosses into another group's entry
task. Both cases sorted the group as if it had no upstream at all.

Now also pulls in the group's upstream_group_ids and its root tasks' upstream
task ids before projecting sibling dependencies. Applied to both the
serialization-layer sort and the mirrored design-time sort in task-sdk.

closes: apache#65291

Related: apache#67964 (closed for inactivity, written against the
topological_sort implementation before PR apache#67288/apache#67688 rewrote it) and
apache#65639 (draft, same issue, also predates the rewrite).
jason810496 added a commit that referenced this pull request Jul 28, 2026
…dependencies (#69933)

* Fix grid/graph view topological sort for group-level and cross-group deps

TaskGroup._project_child_deps only looked at a group's own upstream_task_ids,
which stays empty for a direct group-to-group dependency (list or individual
`>>`) and for a task-level dependency that crosses into another group's entry
task. Both cases sorted the group as if it had no upstream at all.

Now also pulls in the group's upstream_group_ids and its root tasks' upstream
task ids before projecting sibling dependencies. Applied to both the
serialization-layer sort and the mirrored design-time sort in task-sdk.

closes: #65291

Related: #67964 (closed for inactivity, written against the
topological_sort implementation before PR #67288/#67688 rewrote it) and
#65639 (draft, same issue, also predates the rewrite).

* Address review feedback: cache get_task_group_dict, describe test intent not issue numbers

viiccwen pointed out that fetching the group map inside topological_sort() rebuilds
the whole DAG's group tree on every nested group's own call, turning a render with
G groups into an O(G^2) cost. get_task_group_dict() is now memoized per DAG instance
(kept behind a small private helper since methodtools.lru_cache has no type stubs and
would otherwise widen the public method's return type to Any for every caller).

Also reworded test comments/docstrings that cited issue numbers to describe what's
actually being verified instead.

* Hoist common logic into shared lib

* Remove caching on get_task_group_dict

* Remove stale get_task_group_dict cache tests

The cache these tests asserted was removed in the previous commit, so the
identity check and the _get_task_group_dict_cached.cache_info() assertions
no longer apply.

* Add call-level task group memo to reduce calc

* Tidy Typy

* Add test for task group memoing

---------

Co-authored-by: TP <uranusjr@apache.org>
Co-authored-by: LIU ZHE YOU <zhu424.dev@gmail.com>
pierrejeambrun pushed a commit that referenced this pull request Jul 28, 2026
…dependencies (#69933) (#70591)

* Fix grid/graph view topological sort for group-level and cross-group deps

TaskGroup._project_child_deps only looked at a group's own upstream_task_ids,
which stays empty for a direct group-to-group dependency (list or individual
`>>`) and for a task-level dependency that crosses into another group's entry
task. Both cases sorted the group as if it had no upstream at all.

Now also pulls in the group's upstream_group_ids and its root tasks' upstream
task ids before projecting sibling dependencies. Applied to both the
serialization-layer sort and the mirrored design-time sort in task-sdk.

closes: #65291

Related: #67964 (closed for inactivity, written against the
topological_sort implementation before PR #67288/#67688 rewrote it) and
#65639 (draft, same issue, also predates the rewrite).

* Address review feedback: cache get_task_group_dict, describe test intent not issue numbers

viiccwen pointed out that fetching the group map inside topological_sort() rebuilds
the whole DAG's group tree on every nested group's own call, turning a render with
G groups into an O(G^2) cost. get_task_group_dict() is now memoized per DAG instance
(kept behind a small private helper since methodtools.lru_cache has no type stubs and
would otherwise widen the public method's return type to Any for every caller).

Also reworded test comments/docstrings that cited issue numbers to describe what's
actually being verified instead.

* Hoist common logic into shared lib

* Remove caching on get_task_group_dict

* Remove stale get_task_group_dict cache tests

The cache these tests asserted was removed in the previous commit, so the
identity check and the _get_task_group_dict_cached.cache_info() assertions
no longer apply.

* Add call-level task group memo to reduce calc

* Tidy Typy

* Add test for task group memoing

---------

Co-authored-by: TP <uranusjr@apache.org>
Co-authored-by: LIU ZHE YOU <zhu424.dev@gmail.com>
(cherry picked from commit d7aa929)

# Conflicts:
#	airflow-core/src/airflow/api_fastapi/core_api/services/ui/task_group.py

Co-authored-by: Hemkumar Chheda <95332229+hkc-8010@users.noreply.github.com>
vatsrahul1001 pushed a commit that referenced this pull request Aug 5, 2026
…dependencies (#69933) (#70591)

* Fix grid/graph view topological sort for group-level and cross-group deps

TaskGroup._project_child_deps only looked at a group's own upstream_task_ids,
which stays empty for a direct group-to-group dependency (list or individual
`>>`) and for a task-level dependency that crosses into another group's entry
task. Both cases sorted the group as if it had no upstream at all.

Now also pulls in the group's upstream_group_ids and its root tasks' upstream
task ids before projecting sibling dependencies. Applied to both the
serialization-layer sort and the mirrored design-time sort in task-sdk.

closes: #65291

Related: #67964 (closed for inactivity, written against the
topological_sort implementation before PR #67288/#67688 rewrote it) and
#65639 (draft, same issue, also predates the rewrite).

* Address review feedback: cache get_task_group_dict, describe test intent not issue numbers

viiccwen pointed out that fetching the group map inside topological_sort() rebuilds
the whole DAG's group tree on every nested group's own call, turning a render with
G groups into an O(G^2) cost. get_task_group_dict() is now memoized per DAG instance
(kept behind a small private helper since methodtools.lru_cache has no type stubs and
would otherwise widen the public method's return type to Any for every caller).

Also reworded test comments/docstrings that cited issue numbers to describe what's
actually being verified instead.

* Hoist common logic into shared lib

* Remove caching on get_task_group_dict

* Remove stale get_task_group_dict cache tests

The cache these tests asserted was removed in the previous commit, so the
identity check and the _get_task_group_dict_cached.cache_info() assertions
no longer apply.

* Add call-level task group memo to reduce calc

* Tidy Typy

* Add test for task group memoing

---------

Co-authored-by: TP <uranusjr@apache.org>
Co-authored-by: LIU ZHE YOU <zhu424.dev@gmail.com>
(cherry picked from commit d7aa929)

# Conflicts:
#	airflow-core/src/airflow/api_fastapi/core_api/services/ui/task_group.py

Co-authored-by: Hemkumar Chheda <95332229+hkc-8010@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants