From 89dd998e832fb0e52638e5a7403e3f791b28a306 Mon Sep 17 00:00:00 2001 From: bugraoz93 Date: Thu, 7 May 2026 21:19:18 +0200 Subject: [PATCH 1/4] Create Airflow CLI implementation guide for 3.3.0 and later --- .../27_cli_implementation_guide.rst | 128 ++++++++++++++++++ contributing-docs/README.rst | 4 + 2 files changed, 132 insertions(+) create mode 100644 contributing-docs/27_cli_implementation_guide.rst diff --git a/contributing-docs/27_cli_implementation_guide.rst b/contributing-docs/27_cli_implementation_guide.rst new file mode 100644 index 0000000000000..6ffa0890638c9 --- /dev/null +++ b/contributing-docs/27_cli_implementation_guide.rst @@ -0,0 +1,128 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +CLI Implementation Guide (AIP-94) +================================== + +This document describes the direction for implementing new CLI functionality in Apache Airflow +following `AIP-94: Decoupling Remote Commands from Airflow CLI to airflowctl +`_. + +.. contents:: Table of Contents + :depth: 2 + :local: + + +Overview +-------- + +As of Airflow 3.3 (tracked via `GitHub Projects #570 and #571 +`_), the ``airflow`` CLI commands are being +rearchitected so that **remote commands delegate their implementation to the** ``airflowctl`` +**HTTP client** rather than accessing the metadata database directly. + +The user-facing ``airflow`` CLI commands remain unchanged — users keep running +``airflow dags list``, ``airflow pools get``, and so on. What changes is the internal +implementation: remote commands call the Public (Core) API through the ``airflowctl`` client +instead of querying the database directly. This enforces RBAC, removes direct database exposure, +and eliminates duplicate code paths. + +This builds on AIP-81, which introduced the distinction between *local* and *remote* commands. + +Decision Rules +-------------- + +**Adding a brand-new command** + +Prefer adding new commands directly to ``airflowctl`` rather than to the ``airflow`` CLI. +If the operation is achievable via the Public API, implement it in ``airflowctl`` only unless +there is a strong reason it must live in core (e.g., it is tightly coupled to a local process +or a deployment concern that has no API representation). Adding it to the ``airflow`` CLI as +well is discouraged — it creates duplicate maintenance surface without user benefit. + +**Modifying an existing command** + +For existing ``airflow`` CLI commands, apply one rule: + +- **Achievable via the Public API?** → rewire the implementation to use the ``airflowctl`` + HTTP client. The command stays in ``airflow-core/src/airflow/cli/`` but delegates all data + access to ``airflowctl``. It must not import SQLAlchemy models or call ``session``\-based + helpers. +- **Not achievable via the Public API** (database shell, migrations, process management, + deployment configuration) → pure core implementation inside ``airflow-core``. No API call, + no ``airflowctl`` dependency. + +.. list-table:: + :header-rows: 1 + :widths: 35 35 30 + + * - Scenario + - Where it goes + - Notes + * - New command, achievable via Public API + - ``airflowctl`` only + - Do not add to ``airflow`` CLI unless strongly needed in core + * - New command, not achievable via Public API + - ``airflow`` CLI, pure core + - Admin/local commands only + * - Existing command, achievable via Public API + - ``airflow`` CLI → delegates to ``airflowctl`` client + - Rewire; no direct DB access + * - Existing command, not achievable via Public API + - ``airflow`` CLI, pure core + - No change in approach + +Implementing a Command Backed by the Public API +------------------------------------------------ + +Source location: ``airflow-core/src/airflow/cli/`` + +1. Add or update the command in the appropriate CLI group. +2. Use the ``airflowctl`` HTTP client (``airflow-ctl/src/airflowctl/``) for all data access. +3. If the required API endpoint does not exist yet, add it first + (see `Adding API Endpoints <16_adding_api_endpoints.rst>`__). +4. Add tests under ``airflow-core/tests/cli/`` that mock or exercise the HTTP client. +5. Run integration tests with: + + .. code-block:: bash + + breeze testing airflow-ctl-integration-test + +Implementing a Pure Core Command +---------------------------------- + +Source location: ``airflow-core/src/airflow/cli/`` + +1. Add or update the command in the appropriate admin group (e.g., ``db``, ``config``). +2. Add ``(admin only)`` to the ``help`` string. +3. Add tests under ``airflow-core/tests/cli/``. + +Bug Fixes +---------- + +- Bug fixes for existing ``airflow`` CLI commands should target the ``v3-2-test`` branch + (backported to ``main`` as needed), as the CLI rearchitecture work is scheduled for 3.3. +- Bug fixes for the ``airflowctl`` client itself target ``main``. + +References +----------- + +- `AIP-94 Confluence page `_ +- `Adding API Endpoints <16_adding_api_endpoints.rst>`__ +- `Airflow Ctl Tests `__ +- `GitHub Project #570 `_ +- `GitHub Project #571 `_ diff --git a/contributing-docs/README.rst b/contributing-docs/README.rst index 6313acf1a96b6..505f006442937 100644 --- a/contributing-docs/README.rst +++ b/contributing-docs/README.rst @@ -136,3 +136,7 @@ Maintainer Tools ``pr-triage`` and ``pr-stats`` skills that maintainers invoke from Claude Code to sweep the open Pull Request queue (run deterministic quality checks, propose actions under explicit per-batch confirmation) and to surface backlog statistics by ``area:*`` label. + +* `CLI Implementation Guide <27_cli_implementation_guide.rst>`__ describes where to implement new + CLI features following AIP-94: remote commands go to ``airflowctl``, admin/deployment commands + stay in the ``airflow`` CLI. From 481de1dd813ff852d8815ba2ad097d58b4d26011 Mon Sep 17 00:00:00 2001 From: bugraoz93 Date: Thu, 7 May 2026 21:34:09 +0200 Subject: [PATCH 2/4] Remove airflowctl tests from contributors docs as it is coupled with ctl not core --- contributing-docs/27_cli_implementation_guide.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/contributing-docs/27_cli_implementation_guide.rst b/contributing-docs/27_cli_implementation_guide.rst index 6ffa0890638c9..d3b86f946db3f 100644 --- a/contributing-docs/27_cli_implementation_guide.rst +++ b/contributing-docs/27_cli_implementation_guide.rst @@ -96,11 +96,7 @@ Source location: ``airflow-core/src/airflow/cli/`` 3. If the required API endpoint does not exist yet, add it first (see `Adding API Endpoints <16_adding_api_endpoints.rst>`__). 4. Add tests under ``airflow-core/tests/cli/`` that mock or exercise the HTTP client. -5. Run integration tests with: - .. code-block:: bash - - breeze testing airflow-ctl-integration-test Implementing a Pure Core Command ---------------------------------- From a7a6ab132b363ad2dea98f9dc40295702aca0be1 Mon Sep 17 00:00:00 2001 From: bugraoz93 Date: Thu, 7 May 2026 21:38:41 +0200 Subject: [PATCH 3/4] Clarify document --- .../27_cli_implementation_guide.rst | 137 +++++++++++------- 1 file changed, 84 insertions(+), 53 deletions(-) diff --git a/contributing-docs/27_cli_implementation_guide.rst b/contributing-docs/27_cli_implementation_guide.rst index d3b86f946db3f..ca4ef259e344a 100644 --- a/contributing-docs/27_cli_implementation_guide.rst +++ b/contributing-docs/27_cli_implementation_guide.rst @@ -30,41 +30,31 @@ following `AIP-94: Decoupling Remote Commands from Airflow CLI to airflowctl Overview -------- -As of Airflow 3.3 (tracked via `GitHub Projects #570 and #571 -`_), the ``airflow`` CLI commands are being -rearchitected so that **remote commands delegate their implementation to the** ``airflowctl`` -**HTTP client** rather than accessing the metadata database directly. - -The user-facing ``airflow`` CLI commands remain unchanged — users keep running -``airflow dags list``, ``airflow pools get``, and so on. What changes is the internal -implementation: remote commands call the Public (Core) API through the ``airflowctl`` client -instead of querying the database directly. This enforces RBAC, removes direct database exposure, -and eliminates duplicate code paths. - -This builds on AIP-81, which introduced the distinction between *local* and *remote* commands. +Airflow ships two CLIs: -Decision Rules --------------- +- **airflow** (``airflow-core``) — bundled with the core distribution. Hosts both legacy + remote commands (being rewired internally) and admin/local commands that have no Public + API equivalent. +- **airflowctl** (``airflow-ctl``) — a standalone CLI distributed separately that talks to a + running Airflow instance exclusively through the Public (Core) API. -**Adding a brand-new command** - -Prefer adding new commands directly to ``airflowctl`` rather than to the ``airflow`` CLI. -If the operation is achievable via the Public API, implement it in ``airflowctl`` only unless -there is a strong reason it must live in core (e.g., it is tightly coupled to a local process -or a deployment concern that has no API representation). Adding it to the ``airflow`` CLI as -well is discouraged — it creates duplicate maintenance surface without user benefit. +As of Airflow 3.3 (tracked via `GitHub Projects #570 and #571 +`_), CLI work follows two rules: -**Modifying an existing command** +1. **New commands** that are achievable via the Public API are added to ``airflowctl`` + **only**. Adding the same command to the ``airflow`` CLI as well is discouraged — it + duplicates maintenance surface without user benefit. +2. **Existing** ``airflow`` **CLI remote commands** stay in place (so users keep running + ``airflow dags list``, ``airflow pools get``, …) but are rewired internally to call the + Public API via the ``airflowctl`` HTTP client instead of accessing the metadata + database directly. -For existing ``airflow`` CLI commands, apply one rule: +Both rules enforce RBAC, remove direct database exposure for remote operations, and eliminate +duplicate code paths. This builds on AIP-81, which introduced the distinction between *local* +and *remote* commands. -- **Achievable via the Public API?** → rewire the implementation to use the ``airflowctl`` - HTTP client. The command stays in ``airflow-core/src/airflow/cli/`` but delegates all data - access to ``airflowctl``. It must not import SQLAlchemy models or call ``session``\-based - helpers. -- **Not achievable via the Public API** (database shell, migrations, process management, - deployment configuration) → pure core implementation inside ``airflow-core``. No API call, - no ``airflowctl`` dependency. +Decision Table +--------------- .. list-table:: :header-rows: 1 @@ -75,44 +65,85 @@ For existing ``airflow`` CLI commands, apply one rule: - Notes * - New command, achievable via Public API - ``airflowctl`` only - - Do not add to ``airflow`` CLI unless strongly needed in core + - Do not add to the ``airflow`` CLI unless strongly needed in core * - New command, not achievable via Public API - - ``airflow`` CLI, pure core - - Admin/local commands only - * - Existing command, achievable via Public API - - ``airflow`` CLI → delegates to ``airflowctl`` client - - Rewire; no direct DB access - * - Existing command, not achievable via Public API - - ``airflow`` CLI, pure core - - No change in approach - -Implementing a Command Backed by the Public API ------------------------------------------------- + - ``airflow`` CLI (admin/local) + - Admin/local commands only — see below + * - Existing ``airflow`` CLI command, achievable via Public API + - ``airflow`` CLI → delegates to the ``airflowctl`` HTTP client + - Rewire; no direct DB access, no SQLAlchemy/``session`` usage + * - Existing ``airflow`` CLI command, not achievable via Public API + - ``airflow`` CLI, unchanged + - Stays as a pure ``airflow-core`` implementation + +"Not achievable via the Public API" means the operation has no API representation and is +inherently admin/local in nature — database shell, schema migrations, process management, +or deployment configuration that requires direct infrastructure access. + +Adding a New Command (Public API achievable) +--------------------------------------------- + +Add the command to ``airflowctl`` **only**. Do not also add it to the ``airflow`` CLI unless +there is a strong reason it must live in core (e.g., it is tightly coupled to a local process +or a deployment concern with no API representation). -Source location: ``airflow-core/src/airflow/cli/`` +Source location: ``airflow-ctl/src/airflowctl/ctl/commands/`` + +HTTP client and operations: ``airflow-ctl/src/airflowctl/api/`` (``client.py``, ``operations.py``). -1. Add or update the command in the appropriate CLI group. -2. Use the ``airflowctl`` HTTP client (``airflow-ctl/src/airflowctl/``) for all data access. +1. Add the command under the appropriate group module in + ``airflow-ctl/src/airflowctl/ctl/commands/``. +2. Call the Public API through the ``airflowctl`` HTTP client (``airflowctl.api.client``) and + the operations layer in ``airflowctl.api.operations``. Do not import ``airflow-core`` + models or touch the metadata database. 3. If the required API endpoint does not exist yet, add it first (see `Adding API Endpoints <16_adding_api_endpoints.rst>`__). -4. Add tests under ``airflow-core/tests/cli/`` that mock or exercise the HTTP client. +4. Add tests under ``airflow-ctl/tests/``. +5. Run integration tests with: + + .. code-block:: bash + + breeze testing airflow-ctl-integration-test + +Rewiring an Existing ``airflow`` CLI Command +--------------------------------------------- + +Use this when an existing ``airflow`` CLI remote command still talks to the database +directly and needs to go through the Public API instead. The user-facing command name and +arguments stay the same. + +Source location: ``airflow-core/src/airflow/cli/`` + +1. Replace direct database access with calls through the ``airflowctl`` HTTP client + (``airflowctl.api.client`` / ``airflowctl.api.operations``). Remove SQLAlchemy model + imports and ``session``\-based helpers from the command. +2. If the required API endpoint does not exist yet, add it first + (see `Adding API Endpoints <16_adding_api_endpoints.rst>`__). +3. Update tests under ``airflow-core/tests/cli/`` to mock or exercise the HTTP client + instead of the database. +Adding an Admin/Local Command (no Public API equivalent) +--------------------------------------------------------- -Implementing a Pure Core Command ----------------------------------- +Use this only when the operation cannot reasonably be exposed through the Public API — +typically database shell, schema migrations, process management, or deployment-time +configuration. Source location: ``airflow-core/src/airflow/cli/`` -1. Add or update the command in the appropriate admin group (e.g., ``db``, ``config``). -2. Add ``(admin only)`` to the ``help`` string. +1. Add the command to an appropriate admin group (e.g., ``db``, ``config``). +2. Add ``(admin only)`` to the ``help`` string so users know the command requires direct + infrastructure access. 3. Add tests under ``airflow-core/tests/cli/``. Bug Fixes ---------- -- Bug fixes for existing ``airflow`` CLI commands should target the ``v3-2-test`` branch - (backported to ``main`` as needed), as the CLI rearchitecture work is scheduled for 3.3. -- Bug fixes for the ``airflowctl`` client itself target ``main``. +- Bug fixes for existing ``airflow`` CLI commands should target the ``v3-2-test`` branch. + The rearchitecture in 3.3 may rewrite or replace the affected code paths on ``main``, so + fix on ``v3-2-test`` first and forward-port to ``main`` only if the same code path still + exists there. +- Bug fixes for ``airflowctl`` itself target ``main``. References ----------- From 4eab802615b37e75e46166d0e020784f7e9f39be Mon Sep 17 00:00:00 2001 From: bugraoz93 Date: Fri, 8 May 2026 21:06:16 +0200 Subject: [PATCH 4/4] Update version and simplify --- .../27_cli_implementation_guide.rst | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/contributing-docs/27_cli_implementation_guide.rst b/contributing-docs/27_cli_implementation_guide.rst index ca4ef259e344a..50b96ec59b273 100644 --- a/contributing-docs/27_cli_implementation_guide.rst +++ b/contributing-docs/27_cli_implementation_guide.rst @@ -15,12 +15,10 @@ specific language governing permissions and limitations under the License. -CLI Implementation Guide (AIP-94) +CLI Implementation Guide ================================== -This document describes the direction for implementing new CLI functionality in Apache Airflow -following `AIP-94: Decoupling Remote Commands from Airflow CLI to airflowctl -`_. +This document describes the direction for implementing new CLI functionality in Apache Airflow. .. contents:: Table of Contents :depth: 2 @@ -38,7 +36,7 @@ Airflow ships two CLIs: - **airflowctl** (``airflow-ctl``) — a standalone CLI distributed separately that talks to a running Airflow instance exclusively through the Public (Core) API. -As of Airflow 3.3 (tracked via `GitHub Projects #570 and #571 +Following AIP-94 (tracked via `GitHub Projects #570 and #571 `_), CLI work follows two rules: 1. **New commands** that are achievable via the Public API are added to ``airflowctl`` @@ -136,15 +134,6 @@ Source location: ``airflow-core/src/airflow/cli/`` infrastructure access. 3. Add tests under ``airflow-core/tests/cli/``. -Bug Fixes ----------- - -- Bug fixes for existing ``airflow`` CLI commands should target the ``v3-2-test`` branch. - The rearchitecture in 3.3 may rewrite or replace the affected code paths on ``main``, so - fix on ``v3-2-test`` first and forward-port to ``main`` only if the same code path still - exists there. -- Bug fixes for ``airflowctl`` itself target ``main``. - References -----------