Skip to content

Pin providers in constraints to the versions published in PyPI - #71324

Merged
potiuk merged 6 commits into
mainfrom
pin-constraints-providers-to-pypi-versions
Aug 9, 2026
Merged

Pin providers in constraints to the versions published in PyPI#71324
potiuk merged 6 commits into
mainfrom
pin-constraints-providers-to-pypi-versions

Conversation

@potiuk

@potiuk potiuk commented Aug 8, 2026

Copy link
Copy Markdown
Member

Constraints cut for a release candidate have to land on the wave being voted on, whose providers exist in PyPI only as rc versions. Asking uv for a pre-release strategy (--prerelease explicit plus a >=0.0.0rc0 lower bound per provider) left it free to answer with any version satisfying those bounds, so the pins were neither the candidate nor the last release.

Now the versions are retrieved from PyPI and named, so there is nothing left to resolve — the constraints pin what is actually published:

  • every active provider is pinned with ==<version> at the newest version PyPI can actually install for the Python being resolved (fully yanked releases, releases with no files, and releases excluded by requires_python are passed over);
  • a pre-release is only ever considered when --allow-pre-releases is set, and even then it has to sort above every final release — so a provider without a candidate in the wave keeps its release, and a released constraints file can never carry an rc pin;
  • a provider PyPI does not have yet (first release still in the wave) is left unpinned, so a locally built wheel can still answer for it;
  • --prerelease explicit and the >=0.0.0rc0 bounds are gone.

Checked against live PyPI while the current wave is being voted on:

google:          final=22.2.2   with pre-releases=22.3.0rc4
cncf-kubernetes: final=10.20.0  with pre-releases=10.21.0rc3
amazon:          final=9.34.0   with pre-releases=9.34.0

A release candidate is also exempt from the provider downgrade check: it pins the wave being voted on, which sorts below the releases the constraints branch already carries, so the comparison says nothing there. The check still runs for finals, where it now catches the pinned version disappearing (yanked, or no longer offering a file for that Python).

Finally, re-running release-constraints.yml for the same candidate now deletes and re-creates that candidate's branch and tag instead of pushing onto them — otherwise the second run hits a non-fast-forward push and a tag that already exists. Finals are untouched: they still commit onto the shared constraints-X-Y.

One consequence worth calling out: with == pins on the regular run too, two providers whose newest releases need incompatible dependencies now fail generation outright instead of quietly settling on an older provider and tripping the downgrade check. The install-failure message says so and points at the additional_constraints_for_highest_resolution escape hatch.

Minimum versions added for five unbounded dependencies

The lowest-direct-resolution test suite ("Low dep tests") resolves direct dependencies to the oldest version their specifier permits. A dependency with no lower bound therefore drops all the way to its first-ever release, which for a compiled package means building a decade-old sdist against a current toolchain:

warning: The direct dependency `krb5` is unpinned. Consider setting a lower bound when using
         `--resolution lowest` or `--resolution lowest-direct` to avoid using outdated versions.
   Building krb5==0.3.0
 × Failed to build `krb5==0.3.0`

uv warns about exactly this. Each of these now carries the floor it is actually developed and tested against:

Distribution Dependency Floor
providers/samba (kerberos extra) krb5 >=0.8.0
providers/akeyless (cloud_id extra) akeyless-cloud-id >=0.3.0
providers/apache/spark (dev group) pyspark >=4.0.0
providers/common/ai (dev group) pydantic-ai-slim[mcp] >=2.0.0
dev/registry (dev group) pytest >=9.1.1

The akeyless requirement is also spelled with the normalised distribution name (akeyless-cloud-id rather than akeyless_cloud_id). The two extras are user-facing, so the provider docs tables were regenerated to match.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

Constraints cut for a release candidate had to land on the wave being voted
on, whose providers exist in PyPI only as rc versions. Asking uv for a
pre-release strategy left it free to answer with any version satisfying the
lower bounds, so the pins were neither the candidate nor the last release, and
what a candidate shipped depended on how the resolution happened to go.

Retrieving the versions from PyPI and naming them leaves nothing to resolve:
the constraints pin what is actually published. A pre-release is only ever
considered when the run allows it, and even then it has to sort above every
final release, so a provider without a candidate in the wave keeps its release
and a released constraints file can never carry an rc pin.

A candidate is exempt from the downgrade check - it sorts below the releases
the constraints branch already carries, so the comparison says nothing there.

Re-running the workflow for the same candidate now replaces that candidate's
branch and tag instead of pushing onto them, so the two always describe the run
that produced them and a wave can be re-cut.
@potiuk potiuk added this to the Airflow 3.3.1 milestone Aug 8, 2026
@potiuk potiuk added the all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs label Aug 8, 2026
@potiuk potiuk closed this Aug 8, 2026
@potiuk potiuk reopened this Aug 8, 2026
Comment thread .github/workflows/release-constraints.yml Outdated
@potiuk
potiuk requested review from Lee-W and kaxil as code owners August 9, 2026 01:07
@potiuk
potiuk requested review from shahar1 and vincbeck as code owners August 9, 2026 01:14
@potiuk potiuk removed this from the Airflow 3.3.1 milestone Aug 9, 2026
@potiuk potiuk modified the milestones: Airflow 3.3.2, Airflow 3.3.1 Aug 9, 2026
@potiuk
potiuk merged commit f9b72d8 into main Aug 9, 2026
288 checks passed
@potiuk
potiuk deleted the pin-constraints-providers-to-pypi-versions branch August 9, 2026 02:16
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker f9b72d8 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

potiuk added a commit that referenced this pull request Aug 9, 2026
… (#71340)

* Pin providers in constraints to the versions published in PyPI

Constraints cut for a release candidate had to land on the wave being voted
on, whose providers exist in PyPI only as rc versions. Asking uv for a
pre-release strategy left it free to answer with any version satisfying the
lower bounds, so the pins were neither the candidate nor the last release, and
what a candidate shipped depended on how the resolution happened to go.

Retrieving the versions from PyPI and naming them leaves nothing to resolve:
the constraints pin what is actually published. A pre-release is only ever
considered when the run allows it, and even then it has to sort above every
final release, so a provider without a candidate in the wave keeps its release
and a released constraints file can never carry an rc pin.

A candidate is exempt from the downgrade check - it sorts below the releases
the constraints branch already carries, so the comparison says nothing there.

Re-running the workflow for the same candidate now replaces that candidate's
branch and tag instead of pushing onto them, so the two always describe the run
that produced them and a wave can be re-cut.

* fixup! Pin providers in constraints to the versions published in PyPI

* fixup! Pin providers in constraints to the versions published in PyPI

* fixup! Pin providers in constraints to the versions published in PyPI

* fixup! Pin providers in constraints to the versions published in PyPI

* fixup! Pin providers in constraints to the versions published in PyPI

(cherry picked from commit f9b72d8)
potiuk added a commit to potiuk/airflow that referenced this pull request Aug 11, 2026
An unbounded requirement lets the resolver answer with any version that has
ever been published, so what our constraints pin - and what a user ends up
installing - depends on how the resolution went rather than on what the code
needs. Four such requirements were found and fixed by hand while pinning
providers in constraints (apache#71324); they sat in four different dependency
tables, which is why they went unnoticed for so long.

Distributions that are members of the uv workspace are exempt - they resolve
from the checkout, so a version range would say nothing - as are direct URL
requirements, where the URL already names the exact artifact.
potiuk added a commit that referenced this pull request Aug 11, 2026
An unbounded requirement lets the resolver answer with any version that has
ever been published, so what our constraints pin - and what a user ends up
installing - depends on how the resolution went rather than on what the code
needs. Four such requirements were found and fixed by hand while pinning
providers in constraints (#71324); they sat in four different dependency
tables, which is why they went unnoticed for so long.

Distributions that are members of the uv workspace are exempt - they resolve
from the checkout, so a version range would say nothing - as are direct URL
requirements, where the URL already names the exact artifact.
potiuk added a commit to jason810496/airflow that referenced this pull request Aug 13, 2026
…e#71324) (apache#71340)

* Pin providers in constraints to the versions published in PyPI

Constraints cut for a release candidate had to land on the wave being voted
on, whose providers exist in PyPI only as rc versions. Asking uv for a
pre-release strategy left it free to answer with any version satisfying the
lower bounds, so the pins were neither the candidate nor the last release, and
what a candidate shipped depended on how the resolution happened to go.

Retrieving the versions from PyPI and naming them leaves nothing to resolve:
the constraints pin what is actually published. A pre-release is only ever
considered when the run allows it, and even then it has to sort above every
final release, so a provider without a candidate in the wave keeps its release
and a released constraints file can never carry an rc pin.

A candidate is exempt from the downgrade check - it sorts below the releases
the constraints branch already carries, so the comparison says nothing there.

Re-running the workflow for the same candidate now replaces that candidate's
branch and tag instead of pushing onto them, so the two always describe the run
that produced them and a wave can be re-cut.

* fixup! Pin providers in constraints to the versions published in PyPI

* fixup! Pin providers in constraints to the versions published in PyPI

* fixup! Pin providers in constraints to the versions published in PyPI

* fixup! Pin providers in constraints to the versions published in PyPI

* fixup! Pin providers in constraints to the versions published in PyPI

(cherry picked from commit f9b72d8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs area:dev-tools backport-to-v3-3-test Backport to v3-3-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants