From d07db20ff8fded778789004d4849ead101a0d585 Mon Sep 17 00:00:00 2001 From: Wei Lee Date: Tue, 7 Jul 2026 22:41:14 +0800 Subject: [PATCH] Document when to use common.ai vs vendor-specific AI providers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The common.ai, openai, and anthropic docs landing pages now explain how to choose between the vendor-neutral operators and the native API wrappers — including where the agent loop runs (Airflow worker vs vendor-managed service). --- providers/anthropic/docs/index.rst | 35 ++++++++++++++++++++++++++ providers/common/ai/docs/index.rst | 40 ++++++++++++++++++++++++++++++ providers/openai/docs/index.rst | 35 ++++++++++++++++++++++++++ 3 files changed, 110 insertions(+) diff --git a/providers/anthropic/docs/index.rst b/providers/anthropic/docs/index.rst index 582dbff0402e2..f80f3a129c4af 100644 --- a/providers/anthropic/docs/index.rst +++ b/providers/anthropic/docs/index.rst @@ -19,6 +19,41 @@ ``apache-airflow-providers-anthropic`` ====================================== +When to use this provider +-------------------------- + +Use ``anthropic`` when a Dag needs Anthropic's native API surface — services Anthropic runs +for you, which no vendor-neutral operator wraps: + +* ``AnthropicBatchOperator`` and ``AnthropicBatchSensor`` — submit a Claude + `Message Batches `__ + job for asynchronous bulk processing and wait for it to complete. +* ``AnthropicAgentSessionOperator`` — start a Managed Agents session in which the agent loop + runs server-side on Anthropic's infrastructure; the Airflow task only kicks off the session + and waits for its outcome. + +Use :doc:`apache-airflow-providers-common-ai:index` instead when the AI step should be run by +Airflow itself and stay vendor-neutral: + +* Generation, classification, or structured extraction with ``LLMOperator`` — it works with + Claude via a connection, and switching to another model provider later is a connection + change, not a Dag rewrite. +* Agents whose loop runs **in the Airflow worker** with ``AgentOperator`` — Airflow-defined + toolsets (SQL, hooks, MCP servers), human-in-the-loop review, and durable step replay. + +Both ``AgentOperator`` and ``AnthropicAgentSessionOperator`` run agents with Claude models; +the difference is where the loop executes and who controls the tools. ``AgentOperator`` +orchestrates the loop in the worker and calls tools you define in the Dag. +``AnthropicAgentSessionOperator`` hands the whole session to Anthropic's managed service — +the agent runs autonomously against a pre-created agent and environment, without +Airflow-side tool execution. + +For example, running the Message Batches API directly: + +.. exampleinclude:: /../tests/system/anthropic/example_anthropic_batch.py + :language: python + :start-after: [START howto_operator_anthropic_batch] + :end-before: [END howto_operator_anthropic_batch] .. toctree:: :hidden: diff --git a/providers/common/ai/docs/index.rst b/providers/common/ai/docs/index.rst index db89a9a87e698..eb1f768de6050 100644 --- a/providers/common/ai/docs/index.rst +++ b/providers/common/ai/docs/index.rst @@ -19,6 +19,46 @@ ``apache-airflow-providers-common-ai`` ################################################## +When to use this provider +-------------------------- + +``common.ai`` is the vendor-neutral way to put LLM and agent steps in a Dag. It is built on +`pydantic-ai `__, so the model vendor (OpenAI, Anthropic, Google, +Bedrock, …) is picked by the connection ``llm_conn_id`` points at — switching providers later +is a connection change, not a Dag rewrite. The AI step is orchestrated by Airflow: the model +calls, the agent loop, and any tools all run in the Airflow worker, where they get retries, +logging, and observability like any other task. + +Use it when a Dag needs: + +* **Generation, classification, summarization, or structured extraction** — + :doc:`LLMOperator and @task.llm `, with Pydantic-typed output pushed to XCom. +* **Branching on a model's decision** — :doc:`LLMBranchOperator `. +* **Agents with tools** — :doc:`AgentOperator ` runs a multi-turn agent loop + in the worker, calling Airflow-defined toolsets (SQL, hooks, MCP servers), with optional + human-in-the-loop review and durable step replay. +* **Document pipelines** — loading, file analysis, embeddings, and retrieval for RAG + (see :doc:`operators/index`). + +Use a vendor's own provider instead when the Dag needs that vendor's **native API surface** — +a service the vendor runs for you, which no vendor-neutral operator wraps: + +* :doc:`apache-airflow-providers-openai:index` — the Embeddings, Responses, and Batch APIs. +* :doc:`apache-airflow-providers-anthropic:index` — the Claude Message Batches API, and + Managed Agents sessions where the agent loop runs on Anthropic's infrastructure rather + than in the Airflow worker. + +As a rule of thumb: if Airflow should *run* the AI step (and the model should stay +swappable), use ``common.ai``; if the Dag *submits work to* a vendor-managed service and +waits for the result, use that vendor's provider. + +For example, this ``LLMOperator`` call is unchanged whether ``llm_conn_id`` points at an +OpenAI, Anthropic, or other pydantic-ai-supported connection: + +.. exampleinclude:: /../../ai/src/airflow/providers/common/ai/example_dags/example_llm.py + :language: python + :start-after: [START howto_operator_llm_basic] + :end-before: [END howto_operator_llm_basic] .. toctree:: :hidden: diff --git a/providers/openai/docs/index.rst b/providers/openai/docs/index.rst index 7b2d212ea53ce..3f8dbfe12dc93 100644 --- a/providers/openai/docs/index.rst +++ b/providers/openai/docs/index.rst @@ -19,6 +19,41 @@ ``apache-airflow-providers-openai`` ====================================== +When to use this provider +-------------------------- + +Use ``openai`` when a Dag needs OpenAI's native API surface — thin wrappers over +OpenAI-specific endpoints and options: + +* ``OpenAIEmbeddingOperator`` — call the Embeddings API directly, e.g. to feed a vector + store. +* ``OpenAIResponseOperator`` — call the + `Responses API `__ with + OpenAI-specific parameters. +* ``OpenAITriggerBatchOperator`` and ``OpenAIHook`` — submit a + `Batch API `__ job for asynchronous bulk + processing and wait for it to complete. + +Use :doc:`apache-airflow-providers-common-ai:index` instead when the AI step should be run by +Airflow itself and stay vendor-neutral: + +* Generation, classification, or structured extraction with ``LLMOperator`` — it works with + OpenAI models via a connection, and switching to another model provider later is a + connection change, not a Dag rewrite. +* Agents whose loop runs in the Airflow worker with ``AgentOperator`` — Airflow-defined + toolsets (SQL, hooks, MCP servers), human-in-the-loop review, and durable step replay. +* Document-to-vector-store pipelines with its document loader, embedding, and retrieval + operators, which are not tied to OpenAI's embedding models. + +In short: pick ``openai`` to reach an OpenAI-only endpoint; pick ``common.ai`` to keep the +Dag portable across model providers. + +For example, calling the Responses API directly: + +.. exampleinclude:: /../../openai/tests/system/openai/example_openai.py + :language: python + :start-after: [START howto_operator_openai_response] + :end-before: [END howto_operator_openai_response] .. toctree:: :hidden: