[DO NOT MERGE] Add dev/prune_github_releases_assets.py by potiuk · Pull Request #66314 · apache/airflow

potiuk · 2026-05-03T17:05:15Z

Summary

Add a maintenance script for the apache/airflow GitHub Releases page so
it complies with the ASF release-distribution
policy:
only ASF-hosted sources are authoritative; everything else is convenience.

The script lives at dev/prune_github_releases_assets.py and runs via
uv run (PEP-723 inline metadata), three modes, all dry-run by default:

inventory — dump every release with its assets and whether the
disclaimer block is already present, write a JSON inventory file.
update-bodies — prepend a disclaimer block to every in-scope release
body that doesn't yet carry one. The block names
https://downloads.apache.org/airflow/... (current) and
https://archive.apache.org/dist/airflow/... (archived) as the only
authoritative sources, points at https://downloads.apache.org/airflow/KEYS
for signature verification, and mentions PyPI / ArtifactHub as
convenience mirrors that are byte-for-byte identical to the ASF
artifacts.
delete-assets — delete every asset from every in-scope release
(irreversible).

Default scope is every release whose tag is not a 1.x core Airflow
release: core 2.x / 3.x, helm-chart/X.Y.Z, airflow-ctl/X.Y.Z,
the legacy upgrade-check/X.Y.Z releases, and release candidates /
betas of any of the above. Pass --apply to actually mutate.

Uses the local gh CLI's auth, so the operator must already be
authenticated to an account with permission to edit releases on
apache/airflow before running with --apply.

Why now

Discussed on private@ and there was a general consensus to align the
GitHub Releases page with the ASF policy by removing the binary
attachments and routing readers to downloads.apache.org /
archive.apache.org. A retroactive LAZY CONSENSUS thread on
dev@airflow.apache.org will follow once this lands and is run.

Test plan

uv run dev/prune_github_releases_assets.py inventory lists every
release on apache/airflow and writes a sane JSON inventory.
update-bodies and delete-assets dry-runs print the expected
mutations without changing anything.
After running with --apply (separate operation, not part of
merging this PR), the GitHub Releases page shows only release
notes with the ASF / convenience-mirror links, no binary
attachments.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

jscheffl · 2026-05-03T17:07:06Z

Is this not a "one stop thing"? Really want to merge the script of keep it on branch only for history?

potiuk · 2026-05-03T18:10:34Z

Nope. This was more for review of what it does - I do not plan to merge it - I wrote it in devlist :)

potiuk · 2026-05-03T18:11:12Z

Nope. This was more for review of what it does - I do not plan to merge it - I wrote it in devlist :)

It makes absolutely no sense to merge such scripts any more.

potiuk · 2026-05-03T20:43:12Z

Moved links to the top and replaced the downloads from SVN to our "official download page from sources".

Example of rendered release: https://github.com/potiuk/airflow/releases/tag/release-rendering-test-3.2.1

potiuk · 2026-05-03T20:44:06Z

We can even remove the pypi link ? It's already at teh top I relaized ?

jscheffl · 2026-05-03T21:10:32Z

We can even remove the pypi link ? It's already at teh top I relaized ?

I thnk so, too. Link on topis sufficient.

The scheduled K8s system-test job has been intermittently red because multiple test pods pull the unpinned `alpine:latest` (xcom sidecar) and `busybox:latest` / `ubuntu:latest` (test pods) from Docker Hub anonymously and trip its 100-pulls-per-6h limit (https://github.com/apache/airflow/actions/runs/25365187430/job/74380551079). Without a tag, kubelet defaults `imagePullPolicy` to `Always`, so even nodes that already cached the image re-pull every run. Changes ------- 1. **Production default**: `xcom_sidecar.PodDefaults.SIDECAR_CONTAINER` now uses `alpine:3.23` via a new module-level `XCOM_SIDECAR_IMAGE` constant. Tagged → `imagePullPolicy: IfNotPresent` by default → nodes with the image cached do not re-pull. 2. **System / kubernetes-tests pin**: every bare `image="ubuntu"` / `"busybox"` / `"alpine"` in `kubernetes-tests/...` and the `cncf/kubernetes` system / unit tests is now pinned (ubuntu:24.04, busybox:1.37, alpine:3.23). Test assertions in `test_pod.py` updated to match the new sidecar default. 3. **Pre-load into kind**: a new `_preload_test_images_to_kind()` helper in `breeze k8s` runs after `_upload_k8s_image()` in `_run_complete_tests`. It pulls each image on the runner with exponential-backoff retries on Docker Hub 429s, then `kind load docker-image` puts it on every node — so kubelet never has to reach out to the registry once the cluster is ready. 4. **Auto-tracker**: `scripts/ci/prek/upgrade_important_versions.py` gains `UPGRADE_ALPINE` / `UPGRADE_BUSYBOX` flags, fetchers using the existing Docker Hub `get_latest_image_version()`, regex patterns for `alpine:` / `busybox:` literals plus chart `ALPINE_VERSION` ARGs, and the relevant call-sites added to `FILES_TO_UPDATE`. The next "Upgrade important CI environment" run will keep these pins fresh automatically. Ubuntu is intentionally not auto-tracked: the tracker would prefer the highest semver, which can be an interim (non-LTS) release — system tests want LTS. Drive-by -------- `# type: ignore[no-redef]` on the standard `import tomli as tomllib` fallback in `dev/registry/extract_{metadata,versions}.py` so `mypy-dev` passes on edits to anything else under `dev/`. Identical fix lives in PR apache#66314 — whichever lands first, the other becomes a no-op rebase.

…66423) * Pin Docker Hub test images against rate-limit flakes The scheduled K8s system-test job has been intermittently red because multiple test pods pull the unpinned `alpine:latest` (xcom sidecar) and `busybox:latest` / `ubuntu:latest` (test pods) from Docker Hub anonymously and trip its 100-pulls-per-6h limit (https://github.com/apache/airflow/actions/runs/25365187430/job/74380551079). Without a tag, kubelet defaults `imagePullPolicy` to `Always`, so even nodes that already cached the image re-pull every run. Changes ------- 1. **Production default**: `xcom_sidecar.PodDefaults.SIDECAR_CONTAINER` now uses `alpine:3.23` via a new module-level `XCOM_SIDECAR_IMAGE` constant. Tagged → `imagePullPolicy: IfNotPresent` by default → nodes with the image cached do not re-pull. 2. **System / kubernetes-tests pin**: every bare `image="ubuntu"` / `"busybox"` / `"alpine"` in `kubernetes-tests/...` and the `cncf/kubernetes` system / unit tests is now pinned (ubuntu:24.04, busybox:1.37, alpine:3.23). Test assertions in `test_pod.py` updated to match the new sidecar default. 3. **Pre-load into kind**: a new `_preload_test_images_to_kind()` helper in `breeze k8s` runs after `_upload_k8s_image()` in `_run_complete_tests`. It pulls each image on the runner with exponential-backoff retries on Docker Hub 429s, then `kind load docker-image` puts it on every node — so kubelet never has to reach out to the registry once the cluster is ready. 4. **Auto-tracker**: `scripts/ci/prek/upgrade_important_versions.py` gains `UPGRADE_ALPINE` / `UPGRADE_BUSYBOX` flags, fetchers using the existing Docker Hub `get_latest_image_version()`, regex patterns for `alpine:` / `busybox:` literals plus chart `ALPINE_VERSION` ARGs, and the relevant call-sites added to `FILES_TO_UPDATE`. The next "Upgrade important CI environment" run will keep these pins fresh automatically. Ubuntu is intentionally not auto-tracked: the tracker would prefer the highest semver, which can be an interim (non-LTS) release — system tests want LTS. Drive-by -------- `# type: ignore[no-redef]` on the standard `import tomli as tomllib` fallback in `dev/registry/extract_{metadata,versions}.py` so `mypy-dev` passes on edits to anything else under `dev/`. Identical fix lives in PR #66314 — whichever lands first, the other becomes a no-op rebase. * Fix expected_pod fixtures + changelog formatting + spelling Three follow-ups to the original commit, surfaced by CI on #66423: 1. The dict-literal `image` keys in `expected_pod` fixtures inside `kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py` still pointed at the bare names (`"ubuntu"`, `"alpine"`) — only the kwarg-style `image=` references were caught by the original sed. Pinned them to match the new defaults. Without this, every pod-spec equality assertion against `self.expected_pod` failed on Python 3.10 K8s system tests. 2. The cncf.kubernetes changelog note used a level-3 `~~~` heading directly under `Changelog ---`, which (a) shifted the entire version-section hierarchy and produced ~700 cascading docs-build errors, and (b) was 1 char short of the title length triggering a `Title underline too short` warning. Replaced the heading with a bold-led paragraph — same content, no hierarchy disruption. 3. `kubelet` was missing from `docs/spelling_wordlist.txt`, so the sphinx spellcheck flagged it in the new note.

…66423) (#66687) * Pin Docker Hub test images against rate-limit flakes The scheduled K8s system-test job has been intermittently red because multiple test pods pull the unpinned `alpine:latest` (xcom sidecar) and `busybox:latest` / `ubuntu:latest` (test pods) from Docker Hub anonymously and trip its 100-pulls-per-6h limit (https://github.com/apache/airflow/actions/runs/25365187430/job/74380551079). Without a tag, kubelet defaults `imagePullPolicy` to `Always`, so even nodes that already cached the image re-pull every run. Changes ------- 1. **Production default**: `xcom_sidecar.PodDefaults.SIDECAR_CONTAINER` now uses `alpine:3.23` via a new module-level `XCOM_SIDECAR_IMAGE` constant. Tagged → `imagePullPolicy: IfNotPresent` by default → nodes with the image cached do not re-pull. 2. **System / kubernetes-tests pin**: every bare `image="ubuntu"` / `"busybox"` / `"alpine"` in `kubernetes-tests/...` and the `cncf/kubernetes` system / unit tests is now pinned (ubuntu:24.04, busybox:1.37, alpine:3.23). Test assertions in `test_pod.py` updated to match the new sidecar default. 3. **Pre-load into kind**: a new `_preload_test_images_to_kind()` helper in `breeze k8s` runs after `_upload_k8s_image()` in `_run_complete_tests`. It pulls each image on the runner with exponential-backoff retries on Docker Hub 429s, then `kind load docker-image` puts it on every node — so kubelet never has to reach out to the registry once the cluster is ready. 4. **Auto-tracker**: `scripts/ci/prek/upgrade_important_versions.py` gains `UPGRADE_ALPINE` / `UPGRADE_BUSYBOX` flags, fetchers using the existing Docker Hub `get_latest_image_version()`, regex patterns for `alpine:` / `busybox:` literals plus chart `ALPINE_VERSION` ARGs, and the relevant call-sites added to `FILES_TO_UPDATE`. The next "Upgrade important CI environment" run will keep these pins fresh automatically. Ubuntu is intentionally not auto-tracked: the tracker would prefer the highest semver, which can be an interim (non-LTS) release — system tests want LTS. Drive-by -------- `# type: ignore[no-redef]` on the standard `import tomli as tomllib` fallback in `dev/registry/extract_{metadata,versions}.py` so `mypy-dev` passes on edits to anything else under `dev/`. Identical fix lives in PR #66314 — whichever lands first, the other becomes a no-op rebase. * Fix expected_pod fixtures + changelog formatting + spelling Three follow-ups to the original commit, surfaced by CI on 1. The dict-literal `image` keys in `expected_pod` fixtures inside `kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py` still pointed at the bare names (`"ubuntu"`, `"alpine"`) — only the kwarg-style `image=` references were caught by the original sed. Pinned them to match the new defaults. Without this, every pod-spec equality assertion against `self.expected_pod` failed on Python 3.10 K8s system tests. 2. The cncf.kubernetes changelog note used a level-3 `~~~` heading directly under `Changelog ---`, which (a) shifted the entire version-section hierarchy and produced ~700 cascading docs-build errors, and (b) was 1 char short of the title length triggering a `Title underline too short` warning. Replaced the heading with a bold-led paragraph — same content, no hierarchy disruption. 3. `kubelet` was missing from `docs/spelling_wordlist.txt`, so the sphinx spellcheck flagged it in the new note. (cherry picked from commit 95bbf6a)

A maintenance script for the apache/airflow GitHub Releases page so it complies with the ASF release-distribution policy (only ASF-hosted sources are authoritative; everything else is convenience). Three modes, all dry-run by default: * `inventory` — dump every release with its assets and whether the disclaimer block is already present, write a JSON inventory file. * `update-bodies` — prepend a disclaimer block to every in-scope release body that does not yet carry one. The block names `https://downloads.apache.org/airflow/...` (current) and `https://archive.apache.org/dist/airflow/...` (archived) as the only authoritative sources, points at `https://downloads.apache.org/airflow/KEYS` for signature verification, and mentions PyPI / ArtifactHub as convenience mirrors that are byte-for-byte identical to the ASF artifacts. * `delete-assets` — delete every asset from every in-scope release (irreversible). Default scope is every release whose tag is not a 1.x core Airflow release: core 2.x / 3.x, `helm-chart/X.Y.Z`, `airflow-ctl/X.Y.Z`, the legacy `upgrade-check/X.Y.Z` releases, and release candidates / betas of any of the above. Pass `--apply` to actually mutate. Uses the local `gh` CLI's auth, so the operator must already be authenticated to an account that has permission to edit releases on `apache/airflow` before running with `--apply`.

…prepend Address review feedback on the disclaimer block: * Replace the dual `downloads.apache.org` + `archive.apache.org` URLs with the project's per-version "official downloads" pages on airflow.apache.org (e.g. `/docs/apache-airflow/X.Y.Z/installation/installing-from-sources.html`). Those pages already list the ASF source tarballs, wheels, `.asc` signatures, `.sha512` checksums, and verification instructions, and link out to `dlcdn.apache.org` themselves — so we keep the ASF-policy guarantee while presenting one airflow.apache.org URL to readers. * Switch the body update from a prepended top-of-body block to an appended footer separated by a `---` rule, so the original release notes still come first when readers open the page on GitHub. For `upgrade-check/X.Y.Z` (no airflow.apache.org docs page exists), the footer falls back to `https://archive.apache.org/dist/airflow/upgrade-check/{version}/`. The disclaimer marker used by `update-bodies` to decide whether a given release already carries the footer is updated to match the new wording (`The **only authoritative** source for`).

The disclaimer rendered into in-scope GitHub Release bodies by prune_github_releases_assets.py no longer points readers at PyPI as a convenience mirror, and now spells out that GitHub's auto-generated "Source code (zip|tar.gz)" attachments are git-tag snapshots — not ASF releases, not signed, and different from the canonical artifacts on downloads.apache.org. Drive-by: add `# type: ignore[no-redef]` to the tomli/tomllib fallback in dev/registry/extract_{metadata,versions}.py so mypy-dev passes for unrelated edits in dev/.

…66423) (#66687) * Pin Docker Hub test images against rate-limit flakes The scheduled K8s system-test job has been intermittently red because multiple test pods pull the unpinned `alpine:latest` (xcom sidecar) and `busybox:latest` / `ubuntu:latest` (test pods) from Docker Hub anonymously and trip its 100-pulls-per-6h limit (https://github.com/apache/airflow/actions/runs/25365187430/job/74380551079). Without a tag, kubelet defaults `imagePullPolicy` to `Always`, so even nodes that already cached the image re-pull every run. Changes ------- 1. **Production default**: `xcom_sidecar.PodDefaults.SIDECAR_CONTAINER` now uses `alpine:3.23` via a new module-level `XCOM_SIDECAR_IMAGE` constant. Tagged → `imagePullPolicy: IfNotPresent` by default → nodes with the image cached do not re-pull. 2. **System / kubernetes-tests pin**: every bare `image="ubuntu"` / `"busybox"` / `"alpine"` in `kubernetes-tests/...` and the `cncf/kubernetes` system / unit tests is now pinned (ubuntu:24.04, busybox:1.37, alpine:3.23). Test assertions in `test_pod.py` updated to match the new sidecar default. 3. **Pre-load into kind**: a new `_preload_test_images_to_kind()` helper in `breeze k8s` runs after `_upload_k8s_image()` in `_run_complete_tests`. It pulls each image on the runner with exponential-backoff retries on Docker Hub 429s, then `kind load docker-image` puts it on every node — so kubelet never has to reach out to the registry once the cluster is ready. 4. **Auto-tracker**: `scripts/ci/prek/upgrade_important_versions.py` gains `UPGRADE_ALPINE` / `UPGRADE_BUSYBOX` flags, fetchers using the existing Docker Hub `get_latest_image_version()`, regex patterns for `alpine:` / `busybox:` literals plus chart `ALPINE_VERSION` ARGs, and the relevant call-sites added to `FILES_TO_UPDATE`. The next "Upgrade important CI environment" run will keep these pins fresh automatically. Ubuntu is intentionally not auto-tracked: the tracker would prefer the highest semver, which can be an interim (non-LTS) release — system tests want LTS. Drive-by -------- `# type: ignore[no-redef]` on the standard `import tomli as tomllib` fallback in `dev/registry/extract_{metadata,versions}.py` so `mypy-dev` passes on edits to anything else under `dev/`. Identical fix lives in PR #66314 — whichever lands first, the other becomes a no-op rebase. * Fix expected_pod fixtures + changelog formatting + spelling Three follow-ups to the original commit, surfaced by CI on 1. The dict-literal `image` keys in `expected_pod` fixtures inside `kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py` still pointed at the bare names (`"ubuntu"`, `"alpine"`) — only the kwarg-style `image=` references were caught by the original sed. Pinned them to match the new defaults. Without this, every pod-spec equality assertion against `self.expected_pod` failed on Python 3.10 K8s system tests. 2. The cncf.kubernetes changelog note used a level-3 `~~~` heading directly under `Changelog ---`, which (a) shifted the entire version-section hierarchy and produced ~700 cascading docs-build errors, and (b) was 1 char short of the title length triggering a `Title underline too short` warning. Replaced the heading with a bold-led paragraph — same content, no hierarchy disruption. 3. `kubelet` was missing from `docs/spelling_wordlist.txt`, so the sphinx spellcheck flagged it in the new note. (cherry picked from commit 95bbf6a)

potiuk requested review from amoghrajesh, ashb, bugraoz93, choo121600, ephraimbuddy, gopidesupavan, jason810496, jedcunningham, jscheffl and vatsrahul1001 as code owners May 3, 2026 17:05

boring-cyborg Bot added area:dev-tools backport-to-v3-2-test labels May 3, 2026

potiuk changed the title ~~Add dev/prune_github_releases_assets.py~~ [DO NOT MERGE] Add dev/prune_github_releases_assets.py May 3, 2026

potiuk marked this pull request as draft May 3, 2026 18:10

potiuk mentioned this pull request May 5, 2026

Pin Docker Hub test images against K8s system-test rate-limit flakes #66423

Merged

5 tasks

potiuk added 3 commits May 17, 2026 23:00

potiuk force-pushed the add-prune-github-releases-assets-script branch from 08c3bf1 to 2424636 Compare May 17, 2026 21:02

bugraoz93 removed the backport-to-v3-2-test label Jun 10, 2026

potiuk closed this Jun 21, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DO NOT MERGE] Add dev/prune_github_releases_assets.py#66314

[DO NOT MERGE] Add dev/prune_github_releases_assets.py#66314
potiuk wants to merge 3 commits into
apache:mainfrom
potiuk:add-prune-github-releases-assets-script

potiuk commented May 3, 2026

Uh oh!

jscheffl commented May 3, 2026

Uh oh!

potiuk commented May 3, 2026

Uh oh!

potiuk commented May 3, 2026

Uh oh!

potiuk commented May 3, 2026

Uh oh!

potiuk commented May 3, 2026

Uh oh!

jscheffl commented May 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

potiuk commented May 3, 2026

Summary

Why now

Test plan

Was generative AI tooling used to co-author this PR?

Uh oh!

jscheffl commented May 3, 2026

Uh oh!

potiuk commented May 3, 2026

Uh oh!

potiuk commented May 3, 2026

Uh oh!

potiuk commented May 3, 2026

Uh oh!

potiuk commented May 3, 2026

Uh oh!

jscheffl commented May 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants