From ba2fc551cff4b56358a4d2056dfba038bbb22eb5 Mon Sep 17 00:00:00 2001 From: hankehly Date: Sat, 20 Aug 2022 12:09:55 +0900 Subject: [PATCH 1/5] Update python.rst docs --- docs/apache-airflow/howto/operator/python.rst | 22 +++++++++++++------ 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/apache-airflow/howto/operator/python.rst b/docs/apache-airflow/howto/operator/python.rst index ac9070208f717..dc75742247451 100644 --- a/docs/apache-airflow/howto/operator/python.rst +++ b/docs/apache-airflow/howto/operator/python.rst @@ -22,8 +22,11 @@ PythonOperator ============== -Use the :class:`~airflow.operators.python.PythonOperator` to execute -Python callables. +Use the ``@task`` decorator to execute Python callables. + +.. warning:: + Previous versions of Airflow required using :class:`~airflow.operators.python.PythonOperator` + to execute Python callables. The ``@task`` decorater is now the recommended method. .. exampleinclude:: /../../airflow/example_dags/example_python_operator.py :language: python @@ -34,8 +37,8 @@ Python callables. Passing in arguments ^^^^^^^^^^^^^^^^^^^^ -Use the ``op_args`` and ``op_kwargs`` arguments to pass additional arguments -to the Python callable. +Pass extra arguments to the ``@task`` decorated function as you would with +a normal Python function. .. exampleinclude:: /../../airflow/example_dags/example_python_operator.py :language: python @@ -60,7 +63,12 @@ is evaluated as a :ref:`Jinja template `. PythonVirtualenvOperator ======================== -Use the :class:`~airflow.operators.python.PythonVirtualenvOperator` to execute Python callables inside a new Python virtual environment. The ``virtualenv`` package needs to be installed in the environment that runs Airflow (as optional dependency ``pip install airflow[virtualenv] --constraint ...``). +Use the ``@task.virtualenv`` decorator to execute Python callables inside a new Python virtual environment. +The ``virtualenv`` package needs to be installed in the environment that runs Airflow (as optional dependency ``pip install airflow[virtualenv] --constraint ...``). + +.. warning:: + Previous versions of Airflow required using the :class:`~airflow.operators.python.PythonVirtualenvOperator`. + The ``@task.virtualenv`` decorater is now the recommended method. .. exampleinclude:: /../../airflow/example_dags/example_python_operator.py :language: python @@ -71,8 +79,8 @@ Use the :class:`~airflow.operators.python.PythonVirtualenvOperator` to execute P Passing in arguments ^^^^^^^^^^^^^^^^^^^^ -You can use the ``op_args`` and ``op_kwargs`` arguments the same way you use it in the PythonOperator. -Unfortunately we currently do not support to serialize ``var`` and ``ti`` / ``task_instance`` due to incompatibilities +Pass extra arguments to the ``@task.virtualenv`` decorated function as you would with a normal Python function. +Unfortunately, Airflow does not support serializing ``var``, ``ti`` and ``task_instance`` due to incompatibilities with the underlying library. For Airflow context variables make sure that you either have access to Airflow through setting ``system_site_packages`` to ``True`` or add ``apache-airflow`` to the ``requirements`` argument. Otherwise you won't have access to the most context variables of Airflow in ``op_kwargs``. From 2f4fc16bea1c24a88e1c18601e5b44dd78feaf3f Mon Sep 17 00:00:00 2001 From: hankehly Date: Sat, 20 Aug 2022 12:57:29 +0900 Subject: [PATCH 2/5] Add TaskFlow API section to migration guide --- .../upgrading-from-1-10/index.rst | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/apache-airflow/upgrading-from-1-10/index.rst b/docs/apache-airflow/upgrading-from-1-10/index.rst index dd58b2f4f4b73..5e2dd25444078 100644 --- a/docs/apache-airflow/upgrading-from-1-10/index.rst +++ b/docs/apache-airflow/upgrading-from-1-10/index.rst @@ -36,7 +36,7 @@ Python 3.9 support was added from Airflow 2.1.2. Airflow 2.3.0 dropped support for Python 3.6. It's tested with Python 3.7, 3.8, 3.9 and 3.10. -If you have a specific task that still requires Python 2 then you can use the :class:`~airflow.operators.python.PythonVirtualenvOperator` or the ``KubernetesPodOperator`` for this. +If you have a specific task that still requires Python 2 then you can use the ``@task.virtualenv`` decorator, ``@task.docker`` decorator or the ``KubernetesPodOperator`` for this. For a list of breaking changes between Python 2 and Python 3, please refer to this `handy blog `_ @@ -476,9 +476,8 @@ In the new model a user can accomplish the same thing using the following code u from kubernetes.client import models as k8s - second_task = PythonOperator( + @task( task_id="four_task", - python_callable=test_volume_mount, executor_config={ "pod_override": k8s.V1Pod( spec=k8s.V1PodSpec( @@ -501,8 +500,12 @@ In the new model a user can accomplish the same thing using the following code u ], ) ) - }, + } ) + def test_volume_mount(): + pass + + second_task = test_volume_mount() For Airflow 2.0, the traditional ``executor_config`` will continue operation with a deprecation warning, but will be removed in a future version. @@ -924,6 +927,23 @@ instead. On top of that, a new dag.callback_exceptions counter metric has been added to help better monitor callback exceptions. +Migrating to TaskFlow API +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Airflow 2.0 introduced the TaskFlow API to simplify the declaration of Python callable tasks. +Users are encouraged to replace classic operators with their TaskFlow decorator alternatives. +For details, see :doc:`/tutorial_taskflow_api`. + +============================= ============================================ +Classic Operator TaskFlow Decorator +============================= ============================================ +``PythonOperator`` ``@task`` (short for ``@task.python``) +``PythonVirtualenvOperator`` ``@task.virtualenv`` +``BranchPythonOperator`` ``@task.branch`` +``DockerOperator`` ``@task.docker`` +============================= ============================================ + + Airflow CLI changes in 2.0 ^^^^^^^^^^^^^^^^^^^^^^^^^^ From fc66a4efb5d418e0aaee583406aa1fe9b3421621 Mon Sep 17 00:00:00 2001 From: hankehly Date: Sun, 21 Aug 2022 15:38:41 +0900 Subject: [PATCH 3/5] Make feedback updates --- docs/apache-airflow/howto/operator/python.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/apache-airflow/howto/operator/python.rst b/docs/apache-airflow/howto/operator/python.rst index dc75742247451..7e225a9f0462a 100644 --- a/docs/apache-airflow/howto/operator/python.rst +++ b/docs/apache-airflow/howto/operator/python.rst @@ -25,8 +25,8 @@ PythonOperator Use the ``@task`` decorator to execute Python callables. .. warning:: - Previous versions of Airflow required using :class:`~airflow.operators.python.PythonOperator` - to execute Python callables. The ``@task`` decorater is now the recommended method. + The ``@task`` decorator is recommended over the classic :class:`~airflow.operators.python.PythonOperator` + to execute Python callables. .. exampleinclude:: /../../airflow/example_dags/example_python_operator.py :language: python @@ -67,8 +67,8 @@ Use the ``@task.virtualenv`` decorator to execute Python callables inside a new The ``virtualenv`` package needs to be installed in the environment that runs Airflow (as optional dependency ``pip install airflow[virtualenv] --constraint ...``). .. warning:: - Previous versions of Airflow required using the :class:`~airflow.operators.python.PythonVirtualenvOperator`. - The ``@task.virtualenv`` decorater is now the recommended method. + The ``@task.virtualenv`` decorator is recommended over the classic :class:`~airflow.operators.python.PythonVirtualenvOperator` + to execute Python callables inside new Python virtual environments. .. exampleinclude:: /../../airflow/example_dags/example_python_operator.py :language: python From fad7585dcbfcd3eaf136a7db70f25b157e90cae6 Mon Sep 17 00:00:00 2001 From: Hank Ehly Date: Tue, 23 Aug 2022 12:58:35 +0900 Subject: [PATCH 4/5] Fix static check errors --- docs/apache-airflow/upgrading-from-1-10/index.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/apache-airflow/upgrading-from-1-10/index.rst b/docs/apache-airflow/upgrading-from-1-10/index.rst index 5e2dd25444078..b8c2c90615b46 100644 --- a/docs/apache-airflow/upgrading-from-1-10/index.rst +++ b/docs/apache-airflow/upgrading-from-1-10/index.rst @@ -476,6 +476,7 @@ In the new model a user can accomplish the same thing using the following code u from kubernetes.client import models as k8s + @task( task_id="four_task", executor_config={ @@ -500,11 +501,12 @@ In the new model a user can accomplish the same thing using the following code u ], ) ) - } + }, ) def test_volume_mount(): pass + second_task = test_volume_mount() For Airflow 2.0, the traditional ``executor_config`` will continue operation with a deprecation warning, From dfd43274474c85bd76bf301254e1c1e2d7c7db49 Mon Sep 17 00:00:00 2001 From: Hank Ehly Date: Wed, 24 Aug 2022 11:20:59 +0900 Subject: [PATCH 5/5] Update index.rst --- docs/apache-airflow/upgrading-from-1-10/index.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/apache-airflow/upgrading-from-1-10/index.rst b/docs/apache-airflow/upgrading-from-1-10/index.rst index b8c2c90615b46..6ad1725440df6 100644 --- a/docs/apache-airflow/upgrading-from-1-10/index.rst +++ b/docs/apache-airflow/upgrading-from-1-10/index.rst @@ -36,7 +36,7 @@ Python 3.9 support was added from Airflow 2.1.2. Airflow 2.3.0 dropped support for Python 3.6. It's tested with Python 3.7, 3.8, 3.9 and 3.10. -If you have a specific task that still requires Python 2 then you can use the ``@task.virtualenv`` decorator, ``@task.docker`` decorator or the ``KubernetesPodOperator`` for this. +If you have a specific task that still requires Python 2 then you can use the ``@task.virtualenv``, ``@task.docker`` or ``@task.kubernetes`` decorators for this. For a list of breaking changes between Python 2 and Python 3, please refer to this `handy blog `_ @@ -943,6 +943,7 @@ Classic Operator TaskFlow Decorator ``PythonVirtualenvOperator`` ``@task.virtualenv`` ``BranchPythonOperator`` ``@task.branch`` ``DockerOperator`` ``@task.docker`` +``KubernetesPodOperator`` ``@task.kubernetes`` ============================= ============================================