feat: modernize to uv + pyproject.toml + semantic-release - #461
Open
irfanuddinahmad wants to merge 10 commits into
Open
feat: modernize to uv + pyproject.toml + semantic-release#461irfanuddinahmad wants to merge 10 commits into
irfanuddinahmad wants to merge 10 commits into
Conversation
Replace setup.py with PEP 621 static metadata in pyproject.toml: name, description, license (SPDX expression), authors, classifiers, dependencies, optional-dependencies, entry points, and setuptools-scm based dynamic versioning. Remove the hardcoded __version__ from opaque_keys/__init__.py (now permanently stale under setuptools-scm) in favor of reading it from package metadata via importlib.metadata at runtime. Part of openedx/public-engineering#506.
Replace requirements/*.in + pip-compile with PEP 735 dependency groups in pyproject.toml (test-base, test, django42, quality, doc, ci, dev), managed and locked via uv: - Add [tool.uv].constraint-dependencies, machine-managed by `edx_lint write_uv_constraints` (see [tool.edx_lint].uv_constraints for repo-specific overrides). - Commit uv.lock. - Delete the requirements/ directory; update MANIFEST.in accordingly. - Update tox.ini to use tox-uv's uv-venv-lock-runner with dependency_groups instead of deps/-r requirements files. - Update the Makefile's requirements/test/upgrade targets to use `uv sync`, `uv run tox`, and `uv lock --upgrade`. - Update the CI workflow to install uv (pinned to a commit SHA) and run `uv sync --group ci` + `uv run tox`, with per-matrix-job names and read-only permissions. Part of openedx/public-engineering#506.
Add python-semantic-release configuration ([tool.semantic_release] in pyproject.toml, with major_on_zero/allow_zero_version pinned for a repo already well past 1.0) and a release.yml workflow that: - Runs the existing CI suite (via workflow_call) before releasing. - Cuts a release with python-semantic-release when master advances, restricted to contents: write only. - Publishes the built dist/ to PyPI via OIDC (id-token: write, no stored credentials) in a separate, minimally-privileged job. Delete the old token-based pypi-publish.yml (triggered on `release: published`), which is superseded by release.yml's publish_to_pypi job and would otherwise race it once semantic-release starts pushing tags/releases. Update ci.yml to be invoked via workflow_call and drop its own push-to-master trigger, since release.yml now owns that path. commitlint.yml already existed and needed no changes. The latest git tag (4.0.0) matches the latest version published to PyPI, so semantic-release will compute the next version from that baseline. Part of openedx/public-engineering#506.
openedx-webhooks
added
open-source-contribution
|
Thanks for the pull request, @irfanuddinahmad! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
15 tasks
Two things that worked under the old pip-based CI silently relied on dependencies being installed into the ambient/global Python environment rather than an isolated one, and broke once dependency-groups moved everything into uv-managed venvs: - codecov-action shells out to a bare `coverage` command to turn the .coverage data file into coverage.xml; that command isn't on PATH once it's only installed inside tox's isolated uv venv. Add an explicit `uv run --with coverage coverage xml` step before the upload so codecov-action finds an already-generated coverage.xml instead. - .readthedocs.yaml still pointed at the now-deleted requirements/doc.txt. Switch it to `method: uv` / `command: sync` / `groups: [doc]`, matching openedx/XBlock's already-working RTD config.
…) calls The pre-migration "quality" tox env never actually had Django installed (requirements/django-test.txt had its Django== pin stripped by `make upgrade`, and the "quality" env didn't match tox's "django42" factor condition that would have added it back), so pylint was silently analyzing opaque_keys/edx/django with no Django available at all. Now that the "quality" dependency group correctly includes the "django42" group (Django + pytest-django), pylint can actually import Django and attempts real inference over its Field/TestCase base classes — which is where astroid's known difficulty with Django's dynamic Field/descriptor and TestCase machinery surfaces as spurious `no-member` errors on `super().validate()`, `super().run_validators()`, `super().deconstruct()`, `super().setUp()`/`tearDown()`, and a descriptor-typed attribute access. Suppress each with a targeted `# pylint: disable=no-member` rather than disabling the check more broadly.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #461 +/- ##
==========================================
- Coverage 94.03% 92.46% -1.57%
==========================================
Files 31 11 -20
Lines 3068 1328 -1740
Branches 191 145 -46
==========================================
- Hits 2885 1228 -1657
+ Misses 157 78 -79
+ Partials 26 22 -4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…back pytest.ini's --cov-config .coveragerc was still overriding the [tool.coverage.*] settings added to pyproject.toml in the metadata commit, leaving them dead configuration. Delete .coveragerc and drop --cov-config from pytest.ini so coverage.py picks up pyproject.toml directly, which also brings the tests/migrations/settings.py omit patterns and exclude_lines (including "pragma: no cover") into effect for real. Mark the importlib.metadata PackageNotFoundError fallback in opaque_keys/__init__.py with "pragma: no cover": it's only reachable if this package is imported without being installed at all, which never happens in this repo's own test suite, so codecov's patch-coverage check correctly flagged it as an uncovered line added by this PR.
Fixing codecov/patch (previous commit) surfaced a codecov/project failure caused by the same root change: [tool.coverage.run].omit now correctly excludes opaque_keys/**/tests/* from measurement, whereas the old .coveragerc only omitted .tox/*, so test files' own (trivially self-covered) statements were previously counted toward the package's reported coverage. On this PR that drops the measured total from 94.03% (31 files, including tests) to 92.46% (11 files, production code only) -- a one-time change in what's measured, not a regression in production-code test coverage (patch coverage on this PR's actual changes is 100%). Add a project-level threshold to absorb that one-time drop (with margin) while still catching real regressions going forward, matching the pattern already applied to ccx-keys and openedx-filters in this same modernization effort.
Per the 2026-07-15 decision on openedx/public-engineering#506 ("src/ layout is in scope for this cycle"), motivated by flat-layout editable installs of packages like opaque-keys not being reliably resolvable by mypy (confirmed: openedx/xblocks-extra already does this; openedx/XBlock, despite being named as the other precedent in that decision, does not actually have a src/ directory as of this writing -- verified directly rather than assumed). - Move opaque_keys/ to src/opaque_keys/. - pyproject.toml: [tool.setuptools] package-dir = {"" = "src"}; [tool.setuptools.packages.find] where = ["src"]; drop the "settings*" include that existed only for wheel parity with the old bare find_packages() -- settings/ lives outside src/ and is no longer picked up, which is now the correct scope for a test-only Django settings module rather than something worth publishing on PyPI. [tool.coverage.run].source updated to src/opaque_keys. - MANIFEST.in, pytest.ini (testpaths), tox.ini (pycodestyle/pylint/ --ignore path arguments) updated for the new physical path. - mypy.ini switched from `files = opaque_keys` (a literal, now-stale path) to `packages = opaque_keys` (import-based resolution) -- this is the actual fix for the mypy/editable-install problem the src/ layout decision cites, not just a mechanical path rename. Verified locally: editable install resolves opaque_keys to src/opaque_keys/__init__.py; `uv run tox -e quality` (mypy + pylint + pycodestyle), `-e django42` (407 passed), `-e without-django` (394 passed), and `-e docs` all pass; `python -m build` (including with SETUPTOOLS_SCM_PRETEND_VERSION) produces a correctly-versioned wheel installing opaque_keys at the top level.
10 tasks
Verified every uses:@sha in release.yml against the GitHub API (repos/<owner>/<repo>/commits/<sha>) rather than trusting the pins as written. Two real problems, both invisible to PR CI since release.yml only runs on push to master: - python-semantic-release/python-semantic-release and actions/upload-artifact/download-artifact were pinned to SHAs that don't exist in those repos at all -- the upload-artifact and download-artifact SHAs were specifically swapped with each other. Reverted to plain version tags (@v10.6.1, @v7, @v8), matching openedx/XBlock's actual release.yml, which has already cut real production releases successfully with this exact pattern. - pypa/gh-action-pypi-publish was pinned to v1.14.1's annotated *tag object* SHA (2834a31...) rather than the underlying commit SHA (ba38be9e...) -- a tag-object SHA is a real git object but is not what a `uses:` checkout should reference. Corrected to the commit SHA, which is the one form this specific pin genuinely needs (per this repo's own docs on the @release/v1 floating-branch bug).
CHANGELOG.rst was never deleted during the uv/pyproject.toml migration,
but changelog:"false" was blindly copied from a reference template with
no ticket ever requiring it disabled. Wire it up properly:
- Add the ".. changelog-insertion-marker" as the first line of
CHANGELOG.rst so semantic-release knows where to insert new entries
above the existing history.
- Remove changelog:"false" from the python-semantic-release step in
release.yml.
- Configure [tool.semantic_release.changelog] in pyproject.toml with
mode="update" and the matching insertion_flag, plus
default_templates pointed at CHANGELOG.rst in rst format.
- Set tag_format = "{version}" to match this repo's actual tag
convention (plain "4.0.0", not "v4.0.0").
Verified with a local dry run (semantic-release changelog on a
master-named branch at HEAD): new release notes are correctly
inserted above the marker, the pre-existing history below is left
untouched, and the resulting file is valid RST.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Modernize
opaque-keystouv+pyproject.toml(PEP 621/735) +python-semantic-release, and adoptsrc/layout.Part of openedx/public-engineering#506 (tracked in openedx/public-engineering#516).
opaque_keys/tosrc/opaque_keys/(see "src/layout" below).setup.pywithpyproject.toml(PEP 621 static metadata); remove thehardcoded
__version__in favor ofimportlib.metadata+setuptools-scm.uvwith PEP 735 dependency groups (test-base,test,django42,quality,doc,ci,dev); commituv.lock.tox.inito usetox-uv'suv-venv-lock-runner.uv(pinned to a commit SHA) and runuv sync/uv run tox.python-semantic-release+release.yml, publishing to PyPI via OIDC.commitlint.ymlalready existed and needed no changes.src/layoutPer the 2026-07-15 decision on openedx/public-engineering#506 ("src/ layout is in
scope for this cycle"), motivated by flat-layout editable installs of packages like
opaque-keys not being reliably resolvable by
mypy. I verified the cited precedentdirectly rather than trusting the decision comment's own description of it:
openedx/xblocks-extradoes have a realsrc/directory (used as the reference forthe
package-dir/packages.findconfig here);openedx/XBlock, despite being namedas the other precedent, does not actually have one as of this writing.
mypy.iniswitched fromfiles = opaque_keys(a literal path, now stale) topackages = opaque_keys(import-based resolution via the editable install) — that'sthe actual fix for the problem this decision cites, not just a path rename. Verified
locally: editable install resolves
opaque_keystosrc/opaque_keys/__init__.py,mypy/pylint/pycodestyleall pass, the full test suite passes (see TestingNotes), and a built wheel correctly installs
opaque_keysat the top level.One side effect: the top-level
settingstest-settings package (previously bundledin the wheel only as an artifact of
setup.py's barefind_packages()) is no longerpicked up, since it lives outside
src/. That's now the correct scope for it.Removed
Deleted files:
setup.py,requirements/*.in,requirements/*.txt,.github/workflows/pypi-publish.yml(superseded byrelease.yml'spublish_to_pypijob — the old token-based workflow triggered onrelease: publishedand would otherwise race the new OIDC-based publish oncesemantic-release starts creating releases/tags).
Not included
adoption is out of scope for this migration cycle (its own separate epic). Existing
pylint/pycodestyle/mypy config is left untouched.
edx-platform-style constraint-sync script: not applicable — this repo neverhad the
requirements/edx-platform-constraints.txt+check_pins.pymechanismthat pattern replaces.
Known out-of-scope gaps (flagging, not silently working around)
release.yml'spublish_to_pypijob publishes via OIDC (id-token: write, no storedcredentials) instead of the old
PYPI_UPLOAD_TOKEN. This requires a PyPIproject maintainer to add a trusted publisher for
edx-opaque-keys(PyPI project→ Publishing → Add a new publisher → GitHub, repo
openedx/opaque-keys,workflow
release.yml, no environment) before the first release can actuallypublish. Not a blocker for reviewing/merging this PR, but
publish_to_pypiwillfail on the first push to
masteruntil it's done..github/workflows/upgrade-python-requirements.yml. This repo has one,calling
openedx/.github's shared reusable workflow, which hardcodesADD_PATHS="requirements"for its PR-creation step with noadd_paths-styleinput exposed via
workflow_call. Now thatrequirements/is deleted,make upgrade's real output (uv.lock/pyproject.tomlchanges) won't match thathardcoded glob — the scheduled job will keep running but silently stop producing
real dependency-upgrade PRs (empty diff, no failure, no alert). This can't be
fixed from this repo; it needs a fix in
openedx/.githubitself (parameterizeadd_paths, or auto-detectuv.lockvs.requirements/).Versioning
Dynamic, via
setuptools-scmreading the latest git tag. The latest tag (4.0.0)matches the latest version published on PyPI, so
python-semantic-releasewillcompute the next version from that baseline correctly.
Testing Notes
Verified locally (macOS, Python 3.12), both before and after the
src/layout move:uv lockresolves cleanly (86 packages);uv lock --checkconfirms it staysin sync with
pyproject.tomlafter each change.uv sync --group devsucceeds; editable install resolvesopaque_keystosrc/opaque_keys/__init__.py.uv run tox -e django42— 407 passed.uv run tox -e without-django— 394 passed.uv run tox -e docs— builds cleanly.uv run tox -e quality—mypypasses cleanly (30 source files) usingpackages = opaque_keys;pylint/pycodestylepass (targetingsrc/opaque_keysnow).
pylintinitially reportedno-memberfalse positives on Django fieldsuper()calls once the quality env correctly started installing Django (itsilently hadn't before, under the old pip-based tooling) — fixed with targeted
# pylint: disable=no-membercomments; confirmed not a real bug.python -m build, including withSETUPTOOLS_SCM_PRETEND_VERSIONset (themechanism
python-semantic-release'sbuild_commandrelies on) — both buildcorrectly-versioned wheels/sdists with the expected metadata, entry points, and
(post-
src/-move) top-levelopaque_keyspackage layout.codecov/patch(anunreachable
importlib.metadatafallback line, fixed withpragma: no cover,and fully moving coverage config off a stale
.coveragercthat was stillshadowing
pyproject.toml);codecov/project(a one-time ~1.6 point drop from94.03%→92.46% once test files were correctly excluded from coverage measurement,
not a real regression — patch coverage is 100% — absorbed with a
codecov.ymlproject threshold, matching the same pattern applied to
ccx-keys/openedx-filtersin this same effort);.readthedocs.yaml(still pointed atthe deleted
requirements/doc.txt, switched tomethod: uv/groups: [doc],matching
openedx/XBlock's working config).Code reviewer notes
django42is the only Django version group for now (this repo'stox.inienvlist has only ever tested one Django version at a time); adding a
django52-style group later just needs[tool.uv].conflictsif two Djangopins need to coexist in one lockfile.
This PR was created with Claude Code.