Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Dynamic Task Mapping

Dynamic Task Mapping allows a way for a workflow to create a number of tasks at runtime based upon current data, rather than the Dag author having to know in advance how many tasks would be needed.

This is similar to defining your tasks in a for loop, but instead of having the DAG file fetch the data and do that itself, the scheduler can do this based on the output of a previous task.
This is similar to defining your tasks in a for loop, but instead of having the DAG file fetch the data and do that itself, the scheduler can do this based on the output of an upstream task.
Unlike a Python for-loop executed at DAG parse time, dynamic task mapping defers task creation until runtime, allowing the scheduler to determine the exact number of task instances based on upstream task outputs.
Right before a mapped task is executed the scheduler will create *n* copies of the task, one for each input.

Expand Down Expand Up @@ -120,7 +120,7 @@ The ``make_list`` task runs as a normal task and must return a list or dict (see
Repeated mapping
----------------

The result of one mapped task can also be used as input to the next mapped task.
The result of one mapped task can also be used as input to the downstream mapped task.

.. code-block:: python

Expand Down
4 changes: 2 additions & 2 deletions airflow-core/docs/best-practices.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ Communication
--------------

Airflow executes tasks of a Dag on different servers in case you are using :doc:`Kubernetes executor <apache-airflow-providers-cncf-kubernetes:kubernetes_executor>` or :doc:`Celery executor <apache-airflow-providers-celery:celery_executor>`.
Therefore, you should not store any file or config in the local filesystem as the next task is likely to run on a different server without access to it — for example, a task that downloads the data file that the next task processes.
Therefore, you should not store any file or config in the local filesystem as the downstream task is likely to run on a different server without access to it — for example, a task that downloads the data file that the downstream task processes.
In the case of :class:`Local executor <airflow.executors.local_executor.LocalExecutor>`,
storing a file on disk can make retries harder e.g., your task requires a config file that is deleted by another task in Dag.

Expand Down Expand Up @@ -815,7 +815,7 @@ Self-Checks
------------

You can also implement checks in a Dag to make sure the tasks are producing the results as expected.
As an example, if you have a task that pushes data to S3, you can implement a check in the next task. For example, the check could
As an example, if you have a task that pushes data to S3, you can implement a check in the downstream task. For example, the check could
make sure that the partition is created in S3 and perform some simple checks to determine if the data is correct.


Expand Down
2 changes: 1 addition & 1 deletion airflow-core/docs/howto/dynamic-dag-generation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Dynamic Dag Generation

This document describes creation of Dags that have a structure generated dynamically, but where the number of
tasks in the Dag does not change between Dag Runs. If you want to implement a Dag where number of Tasks (or
Task Groups as of Airflow 2.6) can change based on the output/result of previous tasks, see
Task Groups as of Airflow 2.6) can change based on the output/result of upstream tasks, see
:doc:`/authoring-and-scheduling/dynamic-task-mapping`.

.. note:: Consistent sequence of generating tasks and task groups
Expand Down
4 changes: 2 additions & 2 deletions airflow-core/docs/tutorial/objectstorage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Here's what's happening:
- Using ``ObjectStoragePath``, we write the data directly to cloud storage as Parquet

This is a classic TaskFlow pattern. The object key changes each day, allowing us to run this daily and build a dataset
over time. We return the final object path to be used in the next task.
over time. We return the final object path to be used in the downstream task.

Why this is cool: No boto3, no GCS client setup, no credentials juggling. Just simple file semantics that work across
storage backends.
Expand All @@ -133,7 +133,7 @@ A few key things to note:
- We use ``path.fs`` to grab the right filesystem object and register it with DuckDB
- Finally, we query the Parquet file using SQL and return a pandas DataFrame

Notice that the function doesn't recreate the path manually -- it gets the full path from the previous task using Xcom.
Notice that the function doesn't recreate the path manually -- it gets the full path from the upstream task using Xcom.
This makes the task portable and decoupled from earlier logic.

Bringing It All Together
Expand Down
2 changes: 1 addition & 1 deletion airflow-core/docs/tutorial/taskflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ that Airflow can schedule and run. Here's the ``extract`` task:

|

The function's return value is passed to the next task — no manual use of ``XComs`` required. Under the hood, TaskFlow
The function's return value is passed to the downstream task — no manual use of ``XComs`` required. Under the hood, TaskFlow
uses ``XComs`` to manage data passing automatically, abstracting away the complexity of manual XCom management from the
previous methods. You'll define ``transform`` and ``load`` tasks using the same pattern.

Expand Down
4 changes: 2 additions & 2 deletions providers/amazon/docs/operators/ssm.rst
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ To retrieve the output and execution details from an SSM command that has been e

This operator is useful for:

* Retrieving output from commands executed by :class:`~airflow.providers.amazon.aws.operators.ssm.SsmRunCommandOperator` in previous tasks
* Retrieving output from commands executed by :class:`~airflow.providers.amazon.aws.operators.ssm.SsmRunCommandOperator` in upstream tasks
* Getting output from SSM commands executed outside of Airflow
* Inspecting command results for debugging or data processing purposes

Expand All @@ -108,7 +108,7 @@ To retrieve output from all instances that executed a command:

get_all_output = SsmGetCommandInvocationOperator(
task_id="get_command_output",
command_id='{{ ti.xcom_pull(task_ids="run_command") }}', # From previous task
command_id='{{ ti.xcom_pull(task_ids="run_command") }}', # From upstream task
)

To retrieve output from a specific instance:
Expand Down