Update task lifecycle diagram for missing states

[pykenny]

Purpose

Updates task state the Adds 2 missing states ("skipped", "deferred") to the diagram, with more details on state transition conditions.

• Task Instance State Documentation (v2.10.4)

Details

• Adds missing states to the diagram ("skipped" and "deferred")
• Adds missing condition nodes to state transition
• Adds "trigger" component (which handles deferred tasks) to the diagram
• Coverts multiple conditional branch in the graph into sequence (multi-staged) of binary conditional branches
• Split states into three categories: "shared states", "states only for sensors", and "states only for deferrable tasks"
• Uses different icons to differentiate task states and Airflow components
• Uses diagrams library for automated plotting and pre-commit integration; rename "task_lifecycle_diagram" to "diagram_task_lifecycle" to comply with pre-commit condition

Related Issues

• Missing states in task state lifecycle diagram #40185

closes: #40185

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[eladkal]

Nice.
note that task can start execution directly from triggerer without going to the worker first
#38674

I do have some problem with just updating the image like that. It doesn't offer the ability to make changes easily.
Ideally we should have a script that generate the image (diagram as code) similar to how we generate the ERD diagram for the database.
we can use https://github.com/mingrammer/diagrams

[pykenny]

The original diagram:

Diagram after adding triggerer tasks:

Question: Does a triggerer task enter "queued" state during its lifecycle, or it's directly passed to the triggerer once it gets into "scheduled" state?

[potiuk]

Is there a way we can easily get to update that picture when it changes in the future ?

I am a big fan of generating such images from text / code rather than drawing them by hand. We even already have a number of such images generated from the code via pre-commit - using the cool diagrams library:

Python file: https://github.com/apache/airflow/blob/main/docs/apache-airflow/img/diagram_basic_airflow_architecture.py
Resulting diagram: https://github.com/apache/airflow/blob/main/docs/apache-airflow/img/diagram_basic_airflow_architecture.png

And it's enough to re-run the python file to get the new diagram generated based on it.

I somehow don't see a viable path how we can - in the future - easily modify such state diagram if all we have is the .png file or if we need some external tool (what?) to modify the diagram.

Yes it might maybe look nicer when generated by hand, but for me long-term maintainabilty is far more important. Otherwise we will have exactly the same situation that we already experienced - where outdated pictures were basically "lying" to our users about someething - where they were supposed to reflect the reality. This is pretty much inevitable, if we have no easy way to regenerate the images with updated changes - and where drawing those by-hand is not really an option.

WDYT @pykenny ? Maybe you should try to replicate the diagram using similar approach as our architecture diagrams?

[pykenny]

Got your point, I see how the project change its way to manage graphs since last year. That will just take longer to complete in my out-of-date dev environment, however (just realized PyCharm Community haven't had uv integration yet, at least not in their stable versions).

(Update) PyCharm starts supporting uv environment from 2024.3.2 release.

[potiuk]

Got your point, I see how the project change its way to manage graphs since last year. That will just take longer to complete in my out-of-date dev environment, however (just realized PyCharm Community haven't had uv integration yet, at least not in their stable versions).

uv sync -> creates .venv that you can add as existing virtual environment. That's all you need.

[pykenny]

I know it better goes into discussion... but diagrams is not included in the installed dependencies (if uv sync --all-extras --all-groups installs all dependencies possible, and of course I can't do uv pip install diagrams to get a makeshift patch now), and pre-commit install extra dependencies in their cache, instead of virtual environment uv's using. Is that designed so on purpose?

(Update) Oh. So is that the cause?

(pyproject.toml)
The doc extras are not available in the released packages. They are only available when you install Airflow from sources in editable installation - i.e. one that you are usually using to contribute to Airflow.

(Update) From uv lockfile, seems that it blocks installing diagrams on MacOS, perhaps because of requirement on Graphviz installation:

    { name = "diagrams", marker = "sys_platform != 'darwin'" },

(Update) As the result, turned to using pip install approach as fallback, then pip install diagrams to add the package into environment (and of course, Graphviz is already installed on the machine).

[potiuk]

(Update) From uv lockfile, seems that it blocks installing diagrams on MacOS, perhaps because of requirement on Graphviz installation:

Yep. that's likely.

[pykenny]

Okay, here's the new one with diagrams library:

Also let me know if there's any wrong with the workflow described in the graph. For instance:

• Is a triggerer-only task passed to the trigger from scheduler or executor? (assumed to be executor in current version)
• Why is "upstream_failed" viewed as an intermediate state? #46038 (assumed to be terminal state in current version)

[eladkal]

LGTM

[eladkal]

@potiuk any further comments?

[eladkal]

Rebased to make sure CI is green. I'll merge after

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.