Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci-amd-arm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,7 @@ jobs:
secrets:
DOCS_AWS_ACCESS_KEY_ID: ${{ secrets.DOCS_AWS_ACCESS_KEY_ID }}
DOCS_AWS_SECRET_ACCESS_KEY: ${{ secrets.DOCS_AWS_SECRET_ACCESS_KEY }}
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

providers:
name: "provider distributions tests"
Expand Down
40 changes: 37 additions & 3 deletions .github/workflows/ci-image-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,8 @@ on: # yamllint disable-line rule:truthy
required: true
DOCS_AWS_SECRET_ACCESS_KEY:
required: true
SLACK_BOT_TOKEN:
required: false


permissions:
Expand Down Expand Up @@ -252,23 +254,55 @@ jobs:
uses: apache/infrastructure-actions/stash/restore@1c35b5ccf8fba5d4c3fdf25a045ca91aa0cbc468
with:
path: ./generated/_inventory_cache/
key: cache-docs-inventory-v1-${{ hashFiles('**/pyproject.toml') }}
key: cache-docs-inventory-v1
id: restore-docs-inventory-cache
- name: "Building docs with ${{ matrix.flag }} flag"
env:
DOCS_LIST_AS_STRING: ${{ inputs.docs-list-as-string }}
run: >
breeze build-docs ${DOCS_LIST_AS_STRING} ${{ matrix.flag }} --refresh-airflow-inventories
- name: "Check for missing third-party inventories"
id: check-missing-inventories
if: always()
shell: bash
run: |
MARKER_FILE="./generated/_inventory_cache/.missing_third_party_inventories"
if [[ -f "${MARKER_FILE}" ]]; then
echo "missing=true" >> "${GITHUB_OUTPUT}"
echo "::warning::Missing third-party inventories:"
cat "${MARKER_FILE}"
echo "packages<<EOF" >> "${GITHUB_OUTPUT}"
cat "${MARKER_FILE}" >> "${GITHUB_OUTPUT}"
echo "EOF" >> "${GITHUB_OUTPUT}"
else
echo "missing=false" >> "${GITHUB_OUTPUT}"
fi
- name: "Notify Slack about missing inventories (canary only)"
if: >-
inputs.canary-run == 'true' &&
steps.check-missing-inventories.outputs.missing == 'true' &&
matrix.flag == '--docs-only'
uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
with:
method: chat.postMessage
token: ${{ env.SLACK_BOT_TOKEN }}
# yamllint disable rule:line-length
payload: |
channel: "internal-airflow-ci-cd"
text: "⚠️ Missing 3rd-party doc inventories in canary build on *${{ github.ref_name }}*\n\nPackages:\n${{ steps.check-missing-inventories.outputs.packages }}\n\n<https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|View build log>"
# yamllint enable rule:line-length
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
- name: "Save docs inventory cache"
uses: apache/infrastructure-actions/stash/save@1c35b5ccf8fba5d4c3fdf25a045ca91aa0cbc468
with:
path: ./generated/_inventory_cache/
key: cache-docs-inventory-v1-${{ hashFiles('**/pyproject.toml') }}
key: cache-docs-inventory-v1
if-no-files-found: 'error'
retention-days: '2'
# If we upload from multiple matrix jobs we could end up with a race condition. so just pick one job
# to be responsible for updating it. https://github.com/actions/upload-artifact/issues/506
if: steps.restore-docs-inventory-cache != 'true' && matrix.flag == '--docs-only'
if: steps.restore-docs-inventory-cache.outputs.stash-hit != 'true' && matrix.flag == '--docs-only'
- name: "Upload build docs"
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
Expand Down
25 changes: 24 additions & 1 deletion contributing-docs/11_documentation_building.rst
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,30 @@ For example:

breeze build-docs --doc-only --clean fab

Will build ``fab`` provider documentation and clean inventories and other build artifacts before.
Will build ``fab`` provider documentation and clean build artifacts before.

Inventory cache handling
........................

When building documentation, Sphinx downloads intersphinx inventories from external sources (both Airflow
packages hosted on S3 and third-party packages like Pandas, SQLAlchemy, etc.). These inventories enable
cross-references between documentation sets.

By default, missing third-party inventories produce warnings but do **not** fail the build. This is
because third-party inventory servers can be temporarily unavailable and should not block documentation
builds. If a cached version of the inventory exists, it will be used instead.

The following flags control inventory behavior:

- ``--clean-inventory-cache`` — deletes the inventory cache before fetching. Use this when you want
to force a completely fresh download of all inventories.
- ``--clean-build`` — cleans build artifacts (``_build``, ``_doctrees``, ``apis``) but does **not**
delete the inventory cache. This allows rebuilding docs from scratch while preserving cached
inventories.
- ``--refresh-airflow-inventories`` — forces a refresh of only Airflow package inventories, without
cleaning build artifacts or external inventories.
- ``--fail-on-missing-third-party-inventories`` — fails the build if any third-party inventory cannot
be downloaded (useful for publishing workflows where complete cross-references are important).

You can also use ``breeze build-docs --help`` to see available options and head to
`breeze documentation <../dev/breeze/doc/03_developer_tasks.rst>`__ to learn more about the ``breeze``
Expand Down
13 changes: 13 additions & 0 deletions dev/breeze/doc/03_developer_tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,19 @@ package names and can be used to select more than one package with single filter

breeze build-docs --package-filter apache-airflow-providers-*

Inventory cache handling
^^^^^^^^^^^^^^^^^^^^^^^^

When building documentation, Sphinx downloads intersphinx inventories to enable cross-references
between documentation sets. By default, missing third-party inventories (e.g., Pandas, SQLAlchemy)
produce warnings but do **not** fail the build — third-party servers can be temporarily unavailable.
If a cached version exists, it will be used with a warning.

Use ``--clean-inventory-cache`` to force a fresh download of all inventories, or
``--fail-on-missing-third-party-inventories`` to fail the build when any third-party inventory
is missing (useful for publishing). Note that ``--clean-build`` cleans build artifacts but
preserves the inventory cache.

Often errors during documentation generation come from the docstrings of auto-api generated classes.
During the docs building auto-api generated files are stored in the ``generated`` folder. This helps you
easily identify the location the problems with documentation originated from.
Expand Down
4 changes: 4 additions & 0 deletions dev/breeze/doc/09_release_management_tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -933,6 +933,10 @@ These are all available flags of ``workflow-run`` command:
``--site-env`` specifies the environment to use for the site (e.g., auto, live, staging). the default is auto, based on the ref it decides live or staging.
``--refresh-site`` specifies whether to refresh the site after publishing the documentation. This triggers workflow on apache/airflow-site repository to refresh the site.
``--skip-write-to-stable-folder`` specifies the documentation packages to skip writing to the stable folder.
``--ignore-missing-inventories`` when set, the publish workflow will not fail if third-party intersphinx
inventories cannot be downloaded. By default, the publish workflow fails on missing inventories to ensure
complete cross-references in published documentation. Use this flag only when you need to publish despite
temporary third-party inventory outages.


These are all available flags of ``workflow-run publish-docs`` command:
Expand Down
Loading
Loading