[FEATURE] Branching strategy Phase 2.B - release trigger and versioning docs

[John McCall (lowlydba)]

Summary

Phase 2.B of the branching strategy (#533): release detection on main, the changelog machinery, and the versioning docs. No publish workflows, those are Phase 3 (#509), which this unblocks.

The versioning model settled during review (thanks Victor Schappert (@vcschapp), Seth Fitzsimmons (@sethfitz)): humans own the full <major>.<minor>.<patch> per package, any bump merged to main is a release, and CI only versions the interim internal builds between releases.

What lands

Release trigger

• release-trigger.yaml + detect_version_bumps.py: on push to main, diffs every packages/*/pyproject.toml version against the previous commit. Any increase (patch included) cuts one published GitHub Release per bumped package: tag <pkg>-v<version>, title `<pkg>` <version>, notes read from that package's CHANGELOG.md. Fails loudly on version decreases and on pre-existing tags. The umbrella overture-schema release is marked Latest.
• Major-bump cascade guard: a package whose direct workspace dependency takes a major bump must take one itself, since the old dependency floor would admit a breaking version. Enforced at PR time by the version check and again by release-trigger. Direct-dep checking covers longer chains, each unbumped link fails its own check.
• create-package-release composite action with a dry-run input for testing.

Changelog machinery

• towncrier fragments per package (packages/<pkg>/changelog.d/), config centralized in the root pyproject.toml.
• require-changelog-fragment.yaml: any PR touching a package must carry a fragment or a built CHANGELOG.md. Plain git diff + grep, no action dependencies.
• Release PRs run towncrier build so the notes are reviewed as part of the version-bump diff.

Internal build versioning

• compute-version reworked: no-bump merges produce <version>.postN+main.<sha> (or +vnext.<sha>). PEP 440 post-releases order after the release, so >=X.Y.Z consumers resolve the freshest internal build while ==X.Y.Z selects the clean release; local labels are rejected by public PyPI, keeping these builds internal by construction. N is a per-version sequence resolved from CodeArtifact: the first internal build of a version is .post0, later builds increment the highest published .postN.
• Verified against uv with a local two-wheel resolution test; the one spec gotcha (> never matches post-releases) is documented.
• main and vnext builds must not share a CodeArtifact repository (local labels don't participate in ordering, so >= consumers could resolve vnext breaking builds). vnext publishing is blocked on the dedicated dev repo in ops-team#299.

Workspace dependency floors

• Intra-repo dependencies now follow uv's dual declaration pattern: explicit >= floors in project.dependencies alongside the [tool.uv.sources] workspace entries. Built wheels carry the floors directly, no build-time metadata rewriting. Floors are hand-maintained; the cascade guard above is the sanity net. This replaced an earlier just-in-time materialization script after review discussion.

Docs

• docs/versioning.md: scheme, trigger x destination table, bump rules, floor maintenance, release walkthrough, pin guidance.
• CONTRIBUTING.md: per-path release flows with gitGraphs, versioning quick reference, phase roadmap.

Bonus

• 🧹 package_versions.py replaces the dual-checkout version diff: reads pyproject.toml blobs straight from git and derives the topological order from declared dependencies instead of a hardcoded map. The reusable check drops a checkout, two uv syncs, and Python setup on the no-change path.
• 🧹 The change-type label check sheds its github-script JS for jq over the event payload.
• 🔧 The legacy publish workflow syncs with --locked so it installs exactly the committed uv.lock.

Testing

Detection, cascade enforcement, release-notes extraction, floor-carrying wheel metadata, and the resolver-ordering claims are tested locally against real repo history (simulated bump, downgrade, uncascaded-major, and multi-package cases). The workflows themselves can't run until they're on main; release-trigger supports dry-run for a low-stakes first live pass. zizmor is clean on all touched workflows.

Closes #533

[Seth Fitzsimmons (sethfitz)]

the packages inside the workspace do not refer to their monorepo-friend dependencies by explicit versions

But they can (pretty sure), so my read is that we can skip the materialization in favor of keeping things up to date manually (modulo sanity checks).

[Victor Schappert (vcschapp)]

the packages inside the workspace do not refer to their monorepo-friend dependencies by explicit versions

But they can (pretty sure), so my read is that we can skip the materialization in favor of keeping things up to date manually (modulo sanity checks).

That's also fine w me - it would simplify. John McCall (@lowlydba) Seth Fitzsimmons (@sethfitz)

[John McCall (lowlydba)]

the packages inside the workspace do not refer to their monorepo-friend dependencies by explicit versions

But they can (pretty sure), so my read is that we can skip the materialization in favor of keeping things up to date manually (modulo sanity checks).

That's also fine w me - it would simplify. John McCall (John McCall (@lowlydba)) Seth Fitzsimmons (Seth Fitzsimmons (@sethfitz))

Love it - this is exactly why I had included this callout in the latest versioning.md edit:

uv's documented alternative is static dual declaration: hand-maintained specifiers in project.dependencies alongside the workspace source. Not used here; with 13 interdependent packages...

Especially since this cuts down on invisible/magic versioning work behind the scenes that may be non-obvious to a contributor at face value (or even future us 🙃 )

[John McCall (lowlydba)]

the packages inside the workspace do not refer to their monorepo-friend dependencies by explicit versions

But they can (pretty sure), so my read is that we can skip the materialization in favor of keeping things up to date manually (modulo sanity checks).

That's also fine w me - it would simplify. John McCall (John McCall (@lowlydba)) Seth Fitzsimmons (Seth Fitzsimmons (@sethfitz))

+ Victor Schappert (@vcschapp)

Done in 7842f7c. All 12 dependent packages now declare explicit >= floors in project.dependencies alongside their [tool.uv.sources] workspace entries, uv's dual declaration pattern, and materialize_workspace_deps.py is deleted (the script's one glorious final act was writing the floors 🪦 ).

[Seth Fitzsimmons (sethfitz)]

Important thread that's not surfacing on this PR because I commented directly on a commit: 7842f7c#r194263713

It's about how versioned dependencies work within a uv workspace, and the answer complicates CI.

[John McCall (lowlydba)]

Important thread that's not surfacing on this PR because I commented directly on a commit: 7842f7c#r194263713

It's about how versioned dependencies work within a uv workspace, and the answer complicates CI.

Its showing 0 comments on that commit for me - can you re-post in the main thread here? Seth Fitzsimmons (@sethfitz)

[Seth Fitzsimmons (sethfitz)]

Development resolves against the workspace source; built wheels carry the specifier.

(docs/versioning.md:60)

Hmm. I think this fact breaks our CI assumptions, since they run against the workspace (and therefore the tip of each package version). The "lowest-direct" CI cell is the thing we depend on here, since it verifies that floor and ceiling behavior is the same, but if it's not able to test against the floor (because it can't/won't be installed), we don't have the guardrail we expect for intra-workspace dependencies.

We might be able to fix that by testing each package in isolation with the "lowest-direct" cell (so it pulls from PyPI instead of using the workspace).

This all assumes that we will only ever use >= or > comparisons, since <= and < can't be fulfilled without isolated testing within the other CI cells, which might also fail because the tips of each intra-workspace dependency haven't been published (so we can't check work in progress).

I may be wrong here. The answer hinges on whether foo = { workspace = true } forces the package to come from the workspace, or if the workspace is a viable source alongside PyPI.

(And if there's a way to list PyPI as an alternative source to the workspace, if the former.)

If the version in the workspace can't satisfy the dependency, will it fall back to PyPI for previously-published versions?

Verified, unfortunately:

workspace = true installs from the workspace regardless of the specifier (no fallback, no error, floor inert).

[John McCall (lowlydba)]

Copilot resolve the merge conflicts in this pull request

[Copilot]

Copilot resolve the merge conflicts in this pull request

Done. Merged main into the branch; the only conflict was in .github/workflows/reusable-check-python-package-versions.yaml. Kept the branch's optimized single-checkout approach (git blob diffing via package_versions.py) and adopted the dependency-bump versions from main (actions/checkout@v7.0.1, aws-actions/configure-aws-credentials@v6.2.3). Resolved in 756625d (merge) + fix in the follow-up commit.

[Victor Schappert (vcschapp)]

Development resolves against the workspace source; built wheels carry the specifier.

(docs/versioning.md:60)

Hmm. I think this fact breaks our CI assumptions, since they run against the workspace (and therefore the tip of each package version). The "lowest-direct" CI cell is the thing we depend on here, since it verifies that floor and ceiling behavior is the same, but if it's not able to test against the floor (because it can't/won't be installed), we don't have the guardrail we expect for intra-workspace dependencies.

We might be able to fix that by testing each package in isolation with the "lowest-direct" cell (so it pulls from PyPI instead of using the workspace).

This all assumes that we will only ever use >= or > comparisons, since <= and < can't be fulfilled without isolated testing within the other CI cells, which might also fail because the tips of each intra-workspace dependency haven't been published (so we can't check work in progress).

I may be wrong here. The answer hinges on whether foo = { workspace = true } forces the package to come from the workspace, or if the workspace is a viable source alongside PyPI.

(And if there's a way to list PyPI as an alternative source to the workspace, if the former.)

If the version in the workspace can't satisfy the dependency, will it fall back to PyPI for previously-published versions?

Verified, unfortunately:

workspace = true installs from the workspace regardless of the specifier (no fallback, no error, floor inert).

I want to make sure I'm understanding this. Are you saying that if package P in the WS has current version 1.5.6 but the dependent package Q says P>=1.2.3 then we are only exercising =1.5.6 in the CI but not >=1.2.3?

[Seth Fitzsimmons (sethfitz)]

I want to make sure I'm understanding this. Are you saying that if package P in the WS has current version 1.5.6 but the dependent package Q says P>=1.2.3 then we are only exercising =1.5.6 in the CI but not >=1.2.3?

Yes.

P>=1.2.3 is ignored entirely as long as there's a package in the workspace that matches by name.

[Victor Schappert (vcschapp)]

I want to make sure I'm understanding this. Are you saying that if package P in the WS has current version 1.5.6 but the dependent package Q says P>=1.2.3 then we are only exercising =1.5.6 in the CI but not >=1.2.3?

Yes.

P>=1.2.3 is ignored entirely as long as there's a package in the workspace that matches by name.

Cool.

John McCall (@lowlydba) and I just met on this topic. The tl;dr is:

• We acknowledge it's a real issue.
• The solution is a bit complex.
• We're recommending to go forward and finish this phase without solving it, and let our experience with the system in the wild tell us how important it is to add guardrails. In other words, defer for now; if needed fix later.

Seth Fitzsimmons (@sethfitz) What are your thoughts on the above?

---

The best solution I can think of to make it work correctly is to have the CI machinery:

1. Run the tests once with workspace = true to make sure that the workspace itself is in a consistent state.
2. Then, during the publish:
   • Iteratively working upward from "leaf" nodes (e.g., system) to "root" nodes (e.g., schema), publishing one package at a time...
   • ...And after each publish of a package P, run the tests of each workspace package that depends on P either directly or transitively but now using --resolution lowest-direct instead of workspace = true so P comes from the upstream, i.e., PyPI, and it takes the lowest version of P that fits the constraint.

Part 2 of the above solution is a bit complicated and comes with the downside that the publish can succeed partially and fail partially, e.g., I successfully publish package P but then the tests fail for dependent package Q so the automation has to abort without publishing Q.

This partly succeeded, partly failed, publish state is mostly fine since the packages that did succeed in being published work by definition.† But then it leaves you with the question of what do you do with the packages that we made version bump commits for but for which version bumps failed.

Here's a concrete example of that situation.

• Suppose the same commit bumps system v1.2.3 -> v1.3.0 and common v5.6.7 -> v5.7.0. Assume the reason for these bumps is that a new type or function was added to system that is then used in common, a classic minor version bump situation.
• In this situation, we needed common's dependencies to be updated to require system>=5.7.0 but let's suppose for the sake of argument that detail was missed.
• Suppose the workspace true tests pass.
• Now the iterative Part 2 kicks in:
  • Publish system v1.3.0, so far so good.
  • Now try to run the tests for common using system >= v1.2.3 and --resolution lowest-direct. This fails because the current common code requires a feature of system that's only available in v1.3.0.
• Now we've failed in a half-published state where system v1.3.0 is published, common v5.7.0 is not published, and the pyproject.toml version for common is set to v5.7.0.

How do we recover from this state? John McCall (@lowlydba) and I thought of two available manual recovery options and they're both equally valid:

• Could publish a commit setting common back to v5.6.7, fix the pyproject.toml bug, and then re-promote to v5.6.7.
• Or, fix the pyproject.toml bug, consider v5.7.0 as being burned, and publish a bump commit setting common to v5.7.1.

Anyway, and in conclusion, it's a bit complicated and I do think it makes sense to go live without that guardrail for the time being, see how pervasive these versioning bugs are, and if necessary course correct by building the guardrail later.

---

†: The partially failed state can also be avoided by creating another internal private CodeArtifact repo to use instead of PyPI, but again that's more complexity and probably not worth doing.