Airflow Data Quality Provider Plugin #69413
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -294,6 +294,7 @@ conda | |
| conf | ||
| Config | ||
| config | ||
| ConfigDict | ||
| configfile | ||
| configMap | ||
| configmap | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,2 @@ | ||
| www-hash.txt | ||
| *.iml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| --- | ||
| default_stages: [pre-commit, pre-push] | ||
| minimum_prek_version: '0.3.4' | ||
| default_language_version: | ||
| python: python3 | ||
| node: 22.19.0 | ||
| golang: 1.24.0 | ||
| repos: | ||
| - repo: local | ||
| hooks: | ||
| - id: ts-compile-lint-common-dataquality-ui | ||
| name: Compile Data Quality UI | ||
| description: Type-check changed TypeScript files in the Data Quality plugin UI | ||
| language: node | ||
| files: | | ||
| (?x) | ||
| ^src/airflow/providers/common/dataquality/plugins/www/.*\.(js|ts|tsx|yaml|css|json)$ | ||
| exclude: | | ||
| (?x) | ||
| ^src/airflow/providers/common/dataquality/plugins/www/node-modules/.*| | ||
| ^src/airflow/providers/common/dataquality/plugins/www/.pnpm-store | ||
| entry: ../../../scripts/ci/prek/ts_compile_lint_common_dataquality.py | ||
| additional_dependencies: ['pnpm@10.25.0'] | ||
| pass_filenames: true | ||
| require_serial: true | ||
| - id: compile-common-dataquality-provider-assets | ||
| name: Compile Common Data Quality provider assets | ||
| language: node | ||
| stages: ['pre-commit', 'manual'] | ||
| files: ^src/airflow/providers/common/dataquality/plugins/www/ | ||
| entry: ../../../scripts/ci/prek/compile_provider_assets.py dataquality | ||
| pass_filenames: false | ||
| additional_dependencies: ['pnpm@10.25.0'] | ||
| - id: generate-common-dataquality-ruleset-schema | ||
| name: Generate Data Quality RuleSet schema | ||
| language: python | ||
| entry: ../../../scripts/ci/prek/generate_common_dataquality_ruleset_schema.py | ||
| pass_filenames: false | ||
| always_run: true | ||
| files: > | ||
| (?x) | ||
| ^src/airflow/providers/common/dataquality/rules/.*\.py$| | ||
| ^src/airflow/providers/common/dataquality/skills/dataquality-rule-authoring/references/ruleset\.schema\.json$ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| .. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| .. Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| .. _dq:agents: | ||
|
|
||
| Generating rules with an LLM | ||
| ============================== | ||
|
|
||
| Writing a :class:`~airflow.providers.common.dataquality.rules.RuleSet` by hand for every table doesn't scale. | ||
| An LLM can propose one from a table's column definitions instead, given the check catalog as | ||
| context. | ||
|
|
||
| The ``dataquality-rule-authoring`` skill | ||
| ---------------------------------------- | ||
|
|
||
| This provider ships an `Agent Skill <https://agentskills.io>`__ at | ||
| ``airflow/providers/common/dataquality/skills/dataquality-rule-authoring/``: a ``SKILL.md`` documenting the | ||
| ``RuleSet``/``DQRule`` fields and check catalog, plus a generated JSON Schema | ||
| (``references/ruleset.schema.json``) for validation. | ||
|
|
||
| Point ``common.ai``'s :doc:`AgentSkillsToolset <apache-airflow-providers-common-ai:toolsets>` at | ||
| it, and give the model ``output_type=RuleSet`` so pydantic-ai validates -- and self-corrects -- | ||
| its output before the task completes: | ||
|
|
||
| .. exampleinclude:: /../src/airflow/providers/common/dataquality/example_dags/example_dq_llm_generated_ruleset.py | ||
| :language: python | ||
| :start-after: [START howto_task_dq_generate_ruleset_with_llm] | ||
| :end-before: [END howto_task_dq_generate_ruleset_with_llm] | ||
|
|
||
| Requires ``apache-airflow-providers-common-ai[skills]`` and a configured ``llm_conn_id``. | ||
|
|
||
| Wiring the result into a check | ||
| --------------------------------- | ||
|
|
||
| ``@task.dq_check`` can leave ``ruleset=`` unset and return the LLM task's result at execution | ||
| time instead: | ||
|
|
||
| .. exampleinclude:: /../src/airflow/providers/common/dataquality/example_dags/example_dq_llm_generated_ruleset.py | ||
| :language: python | ||
| :start-after: [START howto_decorator_dq_check_llm_runtime_ruleset] | ||
| :end-before: [END howto_decorator_dq_check_llm_runtime_ruleset] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| .. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| .. Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| .. _dq:assets: | ||
|
|
||
| Assets and quality gating | ||
| ============================ | ||
|
|
||
| Quality rules can travel with the :class:`~airflow.sdk.Asset` they describe instead of being | ||
| scattered across every Dag that checks it, and a downstream consumer Dag can refuse to run when | ||
|
Collaborator
There was a problem hiding this comment. I'm not a big fan of describing random bad practices that the code replaces. I suggest
|
||
| the data it was triggered by did not meet a minimum quality bar. | ||
|
|
||
| Attaching a ruleset to an asset | ||
| ---------------------------------- | ||
|
|
||
| :func:`~airflow.providers.common.dataquality.assets.asset_quality` stores ruleset, connection, and table | ||
| configuration inside ``Asset.extra`` under the ``airflow.dataquality`` key, so it is serialized with the | ||
| Dag and needs no Airflow core changes: | ||
|
|
||
| .. exampleinclude:: /../src/airflow/providers/common/dataquality/example_dags/example_dq_require_quality.py | ||
| :language: python | ||
| :start-after: [START howto_asset_quality] | ||
| :end-before: [END howto_asset_quality] | ||
|
|
||
| Pass the resulting asset to :class:`~airflow.providers.common.dataquality.operators.dq_check.DQCheckOperator` | ||
| via ``asset=`` (see :doc:`operators`) instead of ``table``/``ruleset``/``conn_id``: the operator | ||
| resolves all three from the asset's config, adds the asset to its own outlets, and attaches its | ||
| summary -- including the quality ``score`` used below -- to the asset event. | ||
|
|
||
| Only one Dag should call ``asset_quality()`` for a given asset ``name``/``uri``. Airflow keeps one | ||
| shared record per asset across all Dags, so if more than one Dag attaches (or omits) config for the | ||
| same asset, whichever Dag parsed most recently determines what's stored. | ||
|
Collaborator
There was a problem hiding this comment. Why is this preferred over a loud dag parse error? |
||
|
|
||
| Gating a consumer Dag on quality | ||
| ------------------------------------ | ||
|
|
||
| :func:`~airflow.providers.common.dataquality.assets.require_quality` builds a ``@task.short_circuit`` task that | ||
| reads the ``score`` off the asset event that triggered the current run, and skips every | ||
| downstream task when that event has no quality summary at all, or its score is below | ||
| ``min_score``: | ||
|
|
||
| .. exampleinclude:: /../src/airflow/providers/common/dataquality/example_dags/example_dq_require_quality.py | ||
| :language: python | ||
| :start-after: [START howto_require_quality] | ||
| :end-before: [END howto_require_quality] | ||
|
|
||
| Put the gate first in a Dag scheduled by the asset, and chain everything else after it: | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| with DAG("orders_consumer", schedule=orders_asset) as consumer: | ||
| gate = require_quality(orders_asset, min_score=0.95) | ||
| gate >> process_orders() | ||
|
|
||
| ``min_score`` must be between ``0`` and ``1``; the check considers only the *most recent* | ||
| triggering event for the asset when a run was triggered by several. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| .. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| .. Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| .. include:: /../../../../devel-common/src/sphinx_exts/includes/providers-configurations-ref.rst | ||
| .. include:: /../../../../devel-common/src/sphinx_exts/includes/sections-and-options.rst |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a middle ground between "by hand" and "via LLM" - deterministic scripts. This also allows users w/o LLM access to use the tools. These can be bundled into the SKILL or placed somewhere in the plugin.