Fix grid/graph view topological sort for group-level and cross-group dependencies by hkc-8010 · Pull Request #69933 · apache/airflow

hkc-8010 · 2026-07-15T16:54:09Z

When a TaskGroup depends on another TaskGroup through a list ([group_a, group_b] >> group_c)
or through a task-level dependency that crosses into another group's entry task, the grid/graph
view sorts the groups as if there were no dependency at all. _project_child_deps only looks at
a group's own upstream_task_ids, which stays empty in both cases: group-to-group edges only
populate upstream_group_ids, and a task-level edge into another group's entry task never
touches the group object itself.

This also pulls in a group's upstream_group_ids and its root tasks' upstream ids before
projecting sibling dependencies, in both the serialization-layer sort (airflow-core) and the
design-time sort (task-sdk) - they share the same algorithm since the #67288/#67688
performance rewrite.

closes: #65291

related: #67964 and #65639 both attempted this fix earlier but were written against the
pre-rewrite topological_sort implementation (Kahn's-algorithm style), which no longer exists
after #67288 and #67688 replaced it with the sweep/pass-numbering projection used today. Both
were closed for inactivity/never reviewed rather than merged.

Changes

airflow-core/src/airflow/serialization/definitions/taskgroup.py: _project_child_deps also
considers group-to-group edges (upstream_group_ids) and cross-group task-level edges (root
tasks' upstream_task_ids)
task-sdk/src/airflow/sdk/definitions/taskgroup.py: same fix, mirrored design-time sort
Tests for both new scenarios, plus two existing tests whose hardcoded expected orderings were
corrected (they were encoding the old, buggy order)

Was generative AI tooling used to co-author this PR?

Yes (Claude Code)

Generated-by: Claude Code following the guidelines

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

viiccwen

left comments!

uranusjr

I added a couple of commits to hoist common logic into the shared lib. This looks good enough to me now.

Also, I intentionally removed methodtools.lru_cache from get_task_group_dict since an SDK task group can change shape between calls and cause the cache to become stale. The serialized object is generally short-lived (the canonical source is in the database), so the cache is not generally too useful anyway.

…deps TaskGroup._project_child_deps only looked at a group's own upstream_task_ids, which stays empty for a direct group-to-group dependency (list or individual `>>`) and for a task-level dependency that crosses into another group's entry task. Both cases sorted the group as if it had no upstream at all. Now also pulls in the group's upstream_group_ids and its root tasks' upstream task ids before projecting sibling dependencies. Applied to both the serialization-layer sort and the mirrored design-time sort in task-sdk. closes: apache#65291 Related: apache#67964 (closed for inactivity, written against the topological_sort implementation before PR apache#67288/apache#67688 rewrote it) and apache#65639 (draft, same issue, also predates the rewrite).

…ent not issue numbers viiccwen pointed out that fetching the group map inside topological_sort() rebuilds the whole DAG's group tree on every nested group's own call, turning a render with G groups into an O(G^2) cost. get_task_group_dict() is now memoized per DAG instance (kept behind a small private helper since methodtools.lru_cache has no type stubs and would otherwise widen the public method's return type to Any for every caller). Also reworded test comments/docstrings that cited issue numbers to describe what's actually being verified instead.

The cache these tests asserted was removed in the previous commit, so the identity check and the _get_task_group_dict_cached.cache_info() assertions no longer apply.

uranusjr · 2026-07-28T07:26:00Z

I added a call-level memo dict to achieve the same end result as the previous LRU cache (that is problematic for other reasons).

jason810496

#70573 was just merged -- which should resolve all the provider test failures in last run.

jason810496

The patch LGTM, my comments are non-blocking and they won't effect the prod code.

github-actions · 2026-07-28T10:11:25Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker d7aa929 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

jason810496 · 2026-07-28T11:12:04Z

Manually backport in #70591.

…dependencies (#69933) (#70591) * Fix grid/graph view topological sort for group-level and cross-group deps TaskGroup._project_child_deps only looked at a group's own upstream_task_ids, which stays empty for a direct group-to-group dependency (list or individual `>>`) and for a task-level dependency that crosses into another group's entry task. Both cases sorted the group as if it had no upstream at all. Now also pulls in the group's upstream_group_ids and its root tasks' upstream task ids before projecting sibling dependencies. Applied to both the serialization-layer sort and the mirrored design-time sort in task-sdk. closes: #65291 Related: #67964 (closed for inactivity, written against the topological_sort implementation before PR #67288/#67688 rewrote it) and #65639 (draft, same issue, also predates the rewrite). * Address review feedback: cache get_task_group_dict, describe test intent not issue numbers viiccwen pointed out that fetching the group map inside topological_sort() rebuilds the whole DAG's group tree on every nested group's own call, turning a render with G groups into an O(G^2) cost. get_task_group_dict() is now memoized per DAG instance (kept behind a small private helper since methodtools.lru_cache has no type stubs and would otherwise widen the public method's return type to Any for every caller). Also reworded test comments/docstrings that cited issue numbers to describe what's actually being verified instead. * Hoist common logic into shared lib * Remove caching on get_task_group_dict * Remove stale get_task_group_dict cache tests The cache these tests asserted was removed in the previous commit, so the identity check and the _get_task_group_dict_cached.cache_info() assertions no longer apply. * Add call-level task group memo to reduce calc * Tidy Typy * Add test for task group memoing --------- Co-authored-by: TP <uranusjr@apache.org> Co-authored-by: LIU ZHE YOU <zhu424.dev@gmail.com> (cherry picked from commit d7aa929) # Conflicts: # airflow-core/src/airflow/api_fastapi/core_api/services/ui/task_group.py Co-authored-by: Hemkumar Chheda <95332229+hkc-8010@users.noreply.github.com>

hkc-8010 requested review from amoghrajesh, ashb, bolkedebruin and kaxil as code owners July 15, 2026 16:54

boring-cyborg Bot added area:DAG-processing area:task-sdk labels Jul 15, 2026

eladkal added this to the Airflow 3.3.1 milestone Jul 15, 2026

eladkal added type:bug-fix Changelog: Bug Fixes backport-to-v3-3-test Backport to v3-3-test labels Jul 15, 2026

viiccwen reviewed Jul 16, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/utils/test_task_group.py Outdated

Comment thread airflow-core/tests/unit/utils/test_task_group.py Outdated

Comment thread airflow-core/src/airflow/serialization/definitions/taskgroup.py

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 18, 2026

hkc-8010 requested a review from viiccwen July 19, 2026 04:45

kaxil requested review from uranusjr and removed request for viiccwen July 21, 2026 18:05

kaxil reviewed Jul 21, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/utils/test_task_group.py

uranusjr reviewed Jul 25, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/serialization/definitions/taskgroup.py Outdated

uranusjr requested a review from potiuk as a code owner July 28, 2026 00:59

uranusjr approved these changes Jul 28, 2026

View reviewed changes

uranusjr force-pushed the fix/taskgroup-grid-topological-order branch from 4521039 to 2266042 Compare July 28, 2026 01:20

hkc-8010 and others added 5 commits July 28, 2026 04:03

Hoist common logic into shared lib

0ebe633

Remove caching on get_task_group_dict

b6158d4

Remove stale get_task_group_dict cache tests

dedf545

The cache these tests asserted was removed in the previous commit, so the identity check and the _get_task_group_dict_cached.cache_info() assertions no longer apply.

jason810496 force-pushed the fix/taskgroup-grid-topological-order branch from 2b20511 to dedf545 Compare July 28, 2026 04:04

jason810496 mentioned this pull request Jul 28, 2026

Keep SSH remote job running when its PTY session hangs up #70573

Merged

1 task

Add call-level task group memo to reduce calc

e764e75

uranusjr requested review from ephraimbuddy, pierrejeambrun and rawwar as code owners July 28, 2026 07:23

uranusjr requested review from bugraoz93, choo121600, henry3260, jason810496 and shubhamraj-git as code owners July 28, 2026 07:23

uranusjr added 2 commits July 28, 2026 15:35

Tidy Typy

d0e6e2c

Add test for task group memoing

acf074a

jason810496 reviewed Jul 28, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/utils/test_task_group.py

Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py

jason810496 approved these changes Jul 28, 2026

View reviewed changes

jason810496 merged commit d7aa929 into apache:main Jul 28, 2026
154 of 155 checks passed

jason810496 mentioned this pull request Jul 28, 2026

Address review nits from grid/graph topological sort fix #70590

Merged

1 task

Navigation Menu

Uh oh!

Fix grid/graph view topological sort for group-level and cross-group dependencies - #69933

Fix grid/graph view topological sort for group-level and cross-group dependencies#69933
jason810496 merged 8 commits into
apache:mainfrom
hkc-8010:fix/taskgroup-grid-topological-order

hkc-8010 commented Jul 15, 2026

Uh oh!

viiccwen left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uranusjr left a comment

Uh oh!

uranusjr commented Jul 28, 2026

Uh oh!

jason810496 left a comment •

edited

Loading

Uh oh!

jason810496 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

jason810496 commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

hkc-8010 commented Jul 15, 2026

Changes

Was generative AI tooling used to co-author this PR?

Uh oh!

viiccwen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uranusjr left a comment

Choose a reason for hiding this comment

Uh oh!

uranusjr commented Jul 28, 2026

Uh oh!

jason810496 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026

Backport failed to create: v3-3-test. View the failure log Run details

Uh oh!

jason810496 commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jason810496 left a comment •

edited

Loading