Added documentation explaining difference deferred vs async operators by dabla · Pull Request #63500 · apache/airflow

dabla · 2026-03-13T09:46:22Z

Added documentation explaining difference deferred vs async operators, related to the PR #60268 which added native async support for PythonOperators.

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

ChatGPT was used to help rephrasing documentation.

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

…async operators with examples and explaining what the difference is

jscheffl

Looks okay for me, some proposals as feedback.

dabla · 2026-03-13T20:37:48Z

Thanks for the thorough review @jscheffl will fix the remarks.

kaxil

Doc review of the new "Deferred vs Async Operators" page. The content fills a real gap — users need this guidance. Most comments are about broken cross-references (build failures) and making the examples more complete for users.

uranusjr · 2026-03-14T07:29:07Z

I feel we should also discuss when you should not use one of the other. For example

You probaly shouldn’t prefer async over defer if the wait can be long (since an async operator is kept running in a process, which can waste resource).
You probably shouldn’t use defer if deferring happens many times and are generally short (even if you write a custom deferrable operator) since it stops the process every time it defers and restarts everything it resumes, which can introduce a lot of overhead.

Also, the mention about existing deferrable operator can be a bit misleading IMO. If your use case is better for defer, and there’s no existing deferrable operator available, your best course would be to write one yourself, not to use async. It should depend more about the use case, not what Airflow has available.

…e as suggested by TP

jscheffl

Looks good for me now!

Feedback by others welcome though.

@kaxil

…apache#63500) * refactor: Added page explaining when to use deferred and when to use async operators with examples and explaining what the difference is * refactor: Already updated some parts regarding to remarks * refactor: Improved documentation with recommendations from @kaxil * refactor: Added section to explain when not to use async or deferrable as suggested by TP * refactor: Added reference from deferring as Jens suggested * refactor: Fixed doc reference * refactor: Reformatted deferred_http_operator_dag example * refactor: Added MS Graph async example * refactor: Changed order of MS Graph Async Example * refactor: Updated MS Graph example * refactor: Fixed indentation of bullet lists * refactor: Reformatted Async Multiplexing Example * refactor: Fixed reference to deferred vs async in deferring document * refactor: Put full url instead of doc reference to tsk-sdk --------- Co-authored-by: David Blain <david.blain@b-holding.be>

@kaxil

…apache#63500) * refactor: Added page explaining when to use deferred and when to use async operators with examples and explaining what the difference is * refactor: Already updated some parts regarding to remarks * refactor: Improved documentation with recommendations from @kaxil * refactor: Added section to explain when not to use async or deferrable as suggested by TP * refactor: Added reference from deferring as Jens suggested * refactor: Fixed doc reference * refactor: Reformatted deferred_http_operator_dag example * refactor: Added MS Graph async example * refactor: Changed order of MS Graph Async Example * refactor: Updated MS Graph example * refactor: Fixed indentation of bullet lists * refactor: Reformatted Async Multiplexing Example * refactor: Fixed reference to deferred vs async in deferring document * refactor: Put full url instead of doc reference to tsk-sdk --------- Co-authored-by: David Blain <david.blain@b-holding.be>

@kaxil

…apache#63500) * refactor: Added page explaining when to use deferred and when to use async operators with examples and explaining what the difference is * refactor: Already updated some parts regarding to remarks * refactor: Improved documentation with recommendations from @kaxil * refactor: Added section to explain when not to use async or deferrable as suggested by TP * refactor: Added reference from deferring as Jens suggested * refactor: Fixed doc reference * refactor: Reformatted deferred_http_operator_dag example * refactor: Added MS Graph async example * refactor: Changed order of MS Graph Async Example * refactor: Updated MS Graph example * refactor: Fixed indentation of bullet lists * refactor: Reformatted Async Multiplexing Example * refactor: Fixed reference to deferred vs async in deferring document * refactor: Put full url instead of doc reference to tsk-sdk --------- Co-authored-by: David Blain <david.blain@b-holding.be>

@kaxil

…apache#63500) * refactor: Added page explaining when to use deferred and when to use async operators with examples and explaining what the difference is * refactor: Already updated some parts regarding to remarks * refactor: Improved documentation with recommendations from @kaxil * refactor: Added section to explain when not to use async or deferrable as suggested by TP * refactor: Added reference from deferring as Jens suggested * refactor: Fixed doc reference * refactor: Reformatted deferred_http_operator_dag example * refactor: Added MS Graph async example * refactor: Changed order of MS Graph Async Example * refactor: Updated MS Graph example * refactor: Fixed indentation of bullet lists * refactor: Reformatted Async Multiplexing Example * refactor: Fixed reference to deferred vs async in deferring document * refactor: Put full url instead of doc reference to tsk-sdk --------- Co-authored-by: David Blain <david.blain@b-holding.be>

@kaxil

…apache#63500) * refactor: Added page explaining when to use deferred and when to use async operators with examples and explaining what the difference is * refactor: Already updated some parts regarding to remarks * refactor: Improved documentation with recommendations from @kaxil * refactor: Added section to explain when not to use async or deferrable as suggested by TP * refactor: Added reference from deferring as Jens suggested * refactor: Fixed doc reference * refactor: Reformatted deferred_http_operator_dag example * refactor: Added MS Graph async example * refactor: Changed order of MS Graph Async Example * refactor: Updated MS Graph example * refactor: Fixed indentation of bullet lists * refactor: Reformatted Async Multiplexing Example * refactor: Fixed reference to deferred vs async in deferring document * refactor: Put full url instead of doc reference to tsk-sdk --------- Co-authored-by: David Blain <david.blain@b-holding.be>

refactor: Added page explaining when to use deferred and when to use …

57fe34a

…async operators with examples and explaining what the difference is

dabla requested review from amoghrajesh, ashb and kaxil as code owners March 13, 2026 09:46

boring-cyborg Bot added the area:task-sdk label Mar 13, 2026

jscheffl reviewed Mar 13, 2026

View reviewed changes

refactor: Already updated some parts regarding to remarks

e7b4c01

kaxil reviewed Mar 14, 2026

View reviewed changes

dabla added 2 commits March 14, 2026 12:11

Merge branch 'main' into feature/add-docs-async-python

35faa5d

refactor: Improved documentation with recommendations from @kaxil

8638713

dabla force-pushed the feature/add-docs-async-python branch from ecbd95d to 8638713 Compare March 14, 2026 11:42

dabla added 2 commits March 14, 2026 12:47

refactor: Added section to explain when not to use async or deferrabl…

38c4008

…e as suggested by TP

refactor: Added reference from deferring as Jens suggested

65f6205

jscheffl approved these changes Mar 14, 2026

View reviewed changes

Comment thread task-sdk/docs/deferred-vs-async-operators.rst

dabla added 4 commits March 15, 2026 08:48

refactor: Fixed doc reference

512fd11

refactor: Reformatted deferred_http_operator_dag example

e2463a4

refactor: Added MS Graph async example

7f128a6

Merge branch 'main' into feature/add-docs-async-python

a6a7bfc

uranusjr reviewed Mar 16, 2026

View reviewed changes

Comment thread task-sdk/docs/deferred-vs-async-operators.rst Outdated

David Blain and others added 6 commits March 16, 2026 10:30

refactor: Changed order of MS Graph Async Example

616b258

refactor: Updated MS Graph example

e72798b

refactor: Fixed indentation of bullet lists

175b43b

Merge branch 'main' into feature/add-docs-async-python

76367cb

refactor: Reformatted Async Multiplexing Example

1ced166

refactor: Fixed reference to deferred vs async in deferring document

77690ac

dabla marked this pull request as draft March 16, 2026 18:00

dabla marked this pull request as ready for review March 16, 2026 18:00

Merge branch 'main' into feature/add-docs-async-python

09acd59

dabla added 3 commits March 16, 2026 20:38

Merge branch 'main' into feature/add-docs-async-python

9a1e77e

Merge branch 'main' into feature/add-docs-async-python

db68e43

refactor: Put full url instead of doc reference to tsk-sdk

3caba72

dabla requested review from kaxil and uranusjr March 17, 2026 17:21

dabla merged commit 2af6077 into apache:main Mar 17, 2026
71 checks passed

Conversation

dabla commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was generative AI tooling used to co-author this PR?

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dabla commented Mar 13, 2026

Uh oh!

kaxil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uranusjr commented Mar 14, 2026

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dabla commented Mar 13, 2026 •

edited

Loading