Fix grid view topological sort for cross-group dependencies - #67964
Fix grid view topological sort for cross-group dependencies#67964alex-stiff wants to merge 6 commits into
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
ashb
left a comment
There was a problem hiding this comment.
Please add unit tests covering it.
Thank you, added. |
|
Note 🛠️ Maintainer triage note for @alex-stiff · by This draft has been inactive for ~20 days, so I'm closing it to keep the review queue tidy — no judgment on the work itself. No rush — reopen it (or open a fresh PR) whenever you're ready to continue; nothing is lost. Automated triage — may be imperfect; a maintainer takes the next look. |
|
@potiuk would you mind reopening this for me please, I don't seem to have perms! Thank you. |
…deps TaskGroup._project_child_deps only looked at a group's own upstream_task_ids, which stays empty for a direct group-to-group dependency (list or individual `>>`) and for a task-level dependency that crosses into another group's entry task. Both cases sorted the group as if it had no upstream at all. Now also pulls in the group's upstream_group_ids and its root tasks' upstream task ids before projecting sibling dependencies. Applied to both the serialization-layer sort and the mirrored design-time sort in task-sdk. closes: apache#65291 Related: apache#67964 (closed for inactivity, written against the topological_sort implementation before PR apache#67288/apache#67688 rewrote it) and apache#65639 (draft, same issue, also predates the rewrite).
…dependencies (#69933) * Fix grid/graph view topological sort for group-level and cross-group deps TaskGroup._project_child_deps only looked at a group's own upstream_task_ids, which stays empty for a direct group-to-group dependency (list or individual `>>`) and for a task-level dependency that crosses into another group's entry task. Both cases sorted the group as if it had no upstream at all. Now also pulls in the group's upstream_group_ids and its root tasks' upstream task ids before projecting sibling dependencies. Applied to both the serialization-layer sort and the mirrored design-time sort in task-sdk. closes: #65291 Related: #67964 (closed for inactivity, written against the topological_sort implementation before PR #67288/#67688 rewrote it) and #65639 (draft, same issue, also predates the rewrite). * Address review feedback: cache get_task_group_dict, describe test intent not issue numbers viiccwen pointed out that fetching the group map inside topological_sort() rebuilds the whole DAG's group tree on every nested group's own call, turning a render with G groups into an O(G^2) cost. get_task_group_dict() is now memoized per DAG instance (kept behind a small private helper since methodtools.lru_cache has no type stubs and would otherwise widen the public method's return type to Any for every caller). Also reworded test comments/docstrings that cited issue numbers to describe what's actually being verified instead. * Hoist common logic into shared lib * Remove caching on get_task_group_dict * Remove stale get_task_group_dict cache tests The cache these tests asserted was removed in the previous commit, so the identity check and the _get_task_group_dict_cached.cache_info() assertions no longer apply. * Add call-level task group memo to reduce calc * Tidy Typy * Add test for task group memoing --------- Co-authored-by: TP <uranusjr@apache.org> Co-authored-by: LIU ZHE YOU <zhu424.dev@gmail.com>
…dependencies (#69933) (#70591) * Fix grid/graph view topological sort for group-level and cross-group deps TaskGroup._project_child_deps only looked at a group's own upstream_task_ids, which stays empty for a direct group-to-group dependency (list or individual `>>`) and for a task-level dependency that crosses into another group's entry task. Both cases sorted the group as if it had no upstream at all. Now also pulls in the group's upstream_group_ids and its root tasks' upstream task ids before projecting sibling dependencies. Applied to both the serialization-layer sort and the mirrored design-time sort in task-sdk. closes: #65291 Related: #67964 (closed for inactivity, written against the topological_sort implementation before PR #67288/#67688 rewrote it) and #65639 (draft, same issue, also predates the rewrite). * Address review feedback: cache get_task_group_dict, describe test intent not issue numbers viiccwen pointed out that fetching the group map inside topological_sort() rebuilds the whole DAG's group tree on every nested group's own call, turning a render with G groups into an O(G^2) cost. get_task_group_dict() is now memoized per DAG instance (kept behind a small private helper since methodtools.lru_cache has no type stubs and would otherwise widen the public method's return type to Any for every caller). Also reworded test comments/docstrings that cited issue numbers to describe what's actually being verified instead. * Hoist common logic into shared lib * Remove caching on get_task_group_dict * Remove stale get_task_group_dict cache tests The cache these tests asserted was removed in the previous commit, so the identity check and the _get_task_group_dict_cached.cache_info() assertions no longer apply. * Add call-level task group memo to reduce calc * Tidy Typy * Add test for task group memoing --------- Co-authored-by: TP <uranusjr@apache.org> Co-authored-by: LIU ZHE YOU <zhu424.dev@gmail.com> (cherry picked from commit d7aa929) # Conflicts: # airflow-core/src/airflow/api_fastapi/core_api/services/ui/task_group.py Co-authored-by: Hemkumar Chheda <95332229+hkc-8010@users.noreply.github.com>
…dependencies (#69933) (#70591) * Fix grid/graph view topological sort for group-level and cross-group deps TaskGroup._project_child_deps only looked at a group's own upstream_task_ids, which stays empty for a direct group-to-group dependency (list or individual `>>`) and for a task-level dependency that crosses into another group's entry task. Both cases sorted the group as if it had no upstream at all. Now also pulls in the group's upstream_group_ids and its root tasks' upstream task ids before projecting sibling dependencies. Applied to both the serialization-layer sort and the mirrored design-time sort in task-sdk. closes: #65291 Related: #67964 (closed for inactivity, written against the topological_sort implementation before PR #67288/#67688 rewrote it) and #65639 (draft, same issue, also predates the rewrite). * Address review feedback: cache get_task_group_dict, describe test intent not issue numbers viiccwen pointed out that fetching the group map inside topological_sort() rebuilds the whole DAG's group tree on every nested group's own call, turning a render with G groups into an O(G^2) cost. get_task_group_dict() is now memoized per DAG instance (kept behind a small private helper since methodtools.lru_cache has no type stubs and would otherwise widen the public method's return type to Any for every caller). Also reworded test comments/docstrings that cited issue numbers to describe what's actually being verified instead. * Hoist common logic into shared lib * Remove caching on get_task_group_dict * Remove stale get_task_group_dict cache tests The cache these tests asserted was removed in the previous commit, so the identity check and the _get_task_group_dict_cached.cache_info() assertions no longer apply. * Add call-level task group memo to reduce calc * Tidy Typy * Add test for task group memoing --------- Co-authored-by: TP <uranusjr@apache.org> Co-authored-by: LIU ZHE YOU <zhu424.dev@gmail.com> (cherry picked from commit d7aa929) # Conflicts: # airflow-core/src/airflow/api_fastapi/core_api/services/ui/task_group.py Co-authored-by: Hemkumar Chheda <95332229+hkc-8010@users.noreply.github.com>
When specifying group dependencies like
first_stage.end >> second_stage.start, the ordering within the Airflow UI does not respect the defaulttopologicalsorting.Minimal DAG to reproduce:
This shows in the grid view UI as:
Which is alphabetical, rather than the desired
stage_bcoming beforestage_a.This is happening because the
_project_child_depsfunction only checkschild.upstream_task_idspopulated explicitly. We need to also scan theupstream_task_idsof the TaskGroup's root tasks so that the tasks within the group are also visible to this function.Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.Important
🛠️ Maintainer triage note for @alex-stiff · by
@potiuk· 2026-06-17 14:51 UTCHelpful heads-up from the maintainers — please address before this PR can be reviewed:
CI image checks / Static checks). Run them locally withprek run --all-files(orpre-commit run --all-files) and push the fixes.Low dep tests:core / All-core:LowestDeps:14:3.10:Always...Core,MySQL tests: core / DB-core:MySQL:8.0:3.10:Always...Core,Postgres tests: core / DB-core:Postgres:14:3.10:Always...Core,Sqlite tests: core / DB-core:Sqlite:3.10:Always...Core. Reproduce and fix locally, then push.The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.
Automated triage — may be imperfect; a maintainer takes the next look.