From c106b675947b7b244540f8d322c06596cc8d9e95 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 17 Jun 2026 13:53:08 -0700 Subject: [PATCH 1/2] ci(coverage): measure test coverage in CI, report-only (#1278 item 4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wrap the CI `test` job's suite in `coverage run` and publish a coverage table to the run Summary. Report-only — no --fail-under gate; the number is a signal for targeting backfill (#1278 item 5), not a pass/fail bar. - .coveragerc: measure the `website` app only (vendored forks + 3rd-party out of scope), branch coverage on, omit tests/ and the disabled per-env migrations/. - requirements-dev.txt: add coverage==7.14.1 (test-only; not in the prod image). CI's `test` job installs it directly so it doesn't drag in Playwright; the pin is kept in sync with this file. - workflow: `coverage run manage.py test ...` (propagates the suite's exit code, so failures still go red) followed by `coverage report` to the log and a markdown table to $GITHUB_STEP_SUMMARY. - .gitignore: ignore .coverage / htmlcov/ / coverage.xml (written into the bind-mounted repo root). - CLAUDE.md: document the coverage step + how to run it locally. Baseline locally: 59% of the website app (branch coverage), 201 tests. Co-Authored-By: Claude Opus 4.8 (1M context) --- .coveragerc | 35 +++++++++++++++++++++++++++++++++++ .github/workflows/test.yml | 25 +++++++++++++++++++++++-- .gitignore | 7 +++++++ CLAUDE.md | 1 + requirements-dev.txt | 6 ++++++ 5 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 .coveragerc diff --git a/.coveragerc b/.coveragerc new file mode 100644 index 00000000..87387416 --- /dev/null +++ b/.coveragerc @@ -0,0 +1,35 @@ +# Coverage.py configuration (#1278 item 4 — testing roadmap). +# +# We measure the `website` app only: that's the code we own and want to +# backfill. Django itself, third-party packages, and the vendored in-repo forks +# (image_cropping/, sortedm2m_filter_horizontal_widget/) are deliberately out of +# scope. Coverage is REPORT-ONLY in CI — there is no --fail-under gate yet; the +# percentage is a signal for targeting backfill (item 5), not a pass/fail bar. +# +# Run locally (inside the container, after `pip install -r requirements-dev.txt`): +# coverage run manage.py test website --settings=makeabilitylab.settings_test +# coverage report # text table with missing line numbers +# coverage html # browsable htmlcov/ report + +[run] +source = website +# Branch coverage: also tracks whether both sides of an if/else were taken, +# not just whether a line ran. More honest signal for view/utility logic. +branch = True +omit = + # The tests themselves aren't production code. + website/tests/* + # Per-environment migrations are gitignored and disabled in the test + # settings (MIGRATION_MODULES = {'website': None}), so they never execute. + website/migrations/* + +[report] +# Show the line numbers that aren't covered, so the report points at the gaps. +show_missing = True +# exclude_also ADDS to coverage's built-in defaults (which already include +# "pragma: no cover"), rather than replacing them. +exclude_also = + def __repr__ + raise NotImplementedError + if __name__ == .__main__.: + if TYPE_CHECKING: diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 1923f608..33ec0dc5 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -56,8 +56,29 @@ jobs: - name: Install Python dependencies run: pip install -r requirements.txt - - name: Run tests - run: python manage.py test website --settings=makeabilitylab.settings_test --verbosity=2 + # coverage.py is a test-only tool (lives in requirements-dev.txt) — install + # it directly here rather than pulling all of requirements-dev.txt, which + # would drag in Playwright. Keep this pin in sync with requirements-dev.txt. + - name: Install coverage + run: pip install coverage==7.14.1 + + # `coverage run` wraps the same test command and propagates its exit code, + # so a test failure still fails this step (red ✗ + email). Config is in + # .coveragerc (measures the `website` app, branch coverage on). + - name: Run tests with coverage + run: coverage run manage.py test website --settings=makeabilitylab.settings_test --verbosity=2 + + # Report-only: no --fail-under gate. Print to the log and also publish a + # table to the run's Summary so the number is visible without opening logs. + # Skipped automatically if the test step above failed. + - name: Coverage report + run: | + coverage report + { + echo "## Coverage — website app" + echo + coverage report --format=markdown + } >> "$GITHUB_STEP_SUMMARY" # Browser end-to-end tests (member-page "Load more"/"Load all", bio toggle, # section nav). Kept in a SEPARATE job from `test` so the fast unit/integration diff --git a/.gitignore b/.gitignore index 79152a9c..8abe4181 100644 --- a/.gitignore +++ b/.gitignore @@ -48,3 +48,10 @@ docker_makelab_build.log a11y-report.json repomix-output.xml codebase-analysis.md + +# coverage.py artifacts (#1278 item 4) — the data file and HTML/XML reports are +# written into the repo root (the container's CWD is the bind-mounted /code). +.coverage +.coverage.* +htmlcov/ +coverage.xml diff --git a/CLAUDE.md b/CLAUDE.md index 9eb83656..647e15cc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,6 +43,7 @@ A superuser is required to use `/admin` and add content; create one with `python - When fixing a bug reachable through a real queryset, URL, or view, add a regression test in the matching style before applying the fix (matches the tests-first workflow). - **Always use the `--settings=makeabilitylab.settings_test` shim.** It sets `MIGRATION_MODULES = {'website': None}` so the test DB is built directly from the current models, sidestepping the gitignored, per-environment `website/migrations/` history. This is the durable fix for #1267 — without it, a fresh test DB can fail at creation with `column "..." already exists` (old workaround: `docker exec makeabilitylabwebsite-db-1 psql -U admin -d postgres -c "DROP DATABASE IF EXISTS test_makeability;"`). - **CI:** `.github/workflows/test.yml` runs this same command on every push to `master` and every PR (free/unlimited for this public repo). It reports a green ✓ / red ✗ — it does not block pushes or the deploy. See the testing roadmap in #1278. + - **Coverage (#1278 item 4):** the CI `test` job wraps the suite in `coverage run` and publishes a table to the run's Summary; config is in `.coveragerc` (measures the `website` app, branch coverage on). It's **report-only** — no `--fail-under` gate; the number targets backfill. Run locally inside the container: `pip install -r requirements-dev.txt`, then `coverage run manage.py test website --settings=makeabilitylab.settings_test && coverage report` (or `coverage html` for a browsable `htmlcov/`). - Accessibility (Pa11y CI + Axe, WCAG 2.0 AA): start the site, then `docker-compose -f docker-compose-local-dev.yml --profile testing run --rm a11y`. URLs to scan are configured in `.pa11yci.json`. Run this before submitting UI changes. ## Deployment diff --git a/requirements-dev.txt b/requirements-dev.txt index a7bb3f98..e0354419 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -15,3 +15,9 @@ # lockstep (a mismatch makes `playwright install` download the wrong revision). -r requirements.txt playwright==1.60.0 + +# coverage.py — test-coverage measurement (#1278 item 4). Used locally +# (`coverage run manage.py test ...`, see .coveragerc) and pinned identically in +# the CI `test` job, which installs it directly rather than pulling all of +# requirements-dev.txt (it has no need for Playwright). Keep the two pins in sync. +coverage==7.14.1 From 2a84810cfb7ab7f1cb86e4f64d2201755cce350e Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 17 Jun 2026 13:55:33 -0700 Subject: [PATCH 2/2] docs(coverage): document coverage in CONTRIBUTING + README (#1278 item 4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - CONTRIBUTING.md: refresh the now-stale CI paragraph (coverage was listed as future roadmap; it's now implemented) and add a "Test coverage" subsection — report-only, how to run it locally, .coveragerc scope. Add the Continuous integration + Test coverage entries to the TOC. - README.md: list .coveragerc in the Documentation/config table. DEPLOYMENT.md intentionally left unchanged: its testing mention is a pre-deploy pass/fail checklist, and coverage is report-only (not a deploy gate), so adding it there would misrepresent it. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTRIBUTING.md | 18 +++++++++++++++++- README.md | 1 + 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1b9c7f67..3ff3b8e6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -22,6 +22,8 @@ This document outlines how to set up your local development environment and our - [Running the Test Suite](#running-the-test-suite) - [When to add a test](#when-to-add-a-test) - [Troubleshooting](#troubleshooting-tests) + - [Continuous integration](#continuous-integration) + - [Test coverage](#test-coverage) - [Accessibility Testing](#accessibility-testing) - [Running Accessibility Checks](#running-accessibility-checks) - [Configuring Tests](#configuring-tests) @@ -290,7 +292,21 @@ If you forget the shim and the legacy `manage.py test website` fails at test-DB ### Continuous integration -`.github/workflows/test.yml` runs the suite (with the test-settings shim, against a Postgres 16 service container) on every push to `master` and every pull request. GitHub Actions is free and unlimited for this public repo. A failing run shows a red ✗ on the commit/PR and emails the author — it **reports** status, it does not block the push or the test-server deploy. The broader testing roadmap (coverage, Pa11y-in-CI, test backfill) is tracked in [#1278](https://github.com/makeabilitylab/makeabilitylabwebsite/issues/1278). +`.github/workflows/test.yml` runs the suite (with the test-settings shim, against a Postgres 16 service container) on every push to `master` and every pull request. GitHub Actions is free and unlimited for this public repo. A failing run shows a red ✗ on the commit/PR and emails the author — it **reports** status, it does not block the push or the test-server deploy. Each run also measures coverage (see [Test coverage](#test-coverage)). The remaining testing roadmap (Pa11y-in-CI, test backfill) is tracked in [#1278](https://github.com/makeabilitylab/makeabilitylabwebsite/issues/1278). + +### Test coverage + +The CI `test` job wraps the suite in [`coverage`](https://coverage.readthedocs.io/) and posts a per-file table to the run's **Summary** tab. Coverage is **report-only** — there is no `--fail-under` gate, so a low number never fails CI; it's a signal for deciding what to backfill next (#1278 item 5). Configuration is in `.coveragerc`: it measures the `website` app with branch coverage on, and leaves the vendored forks (`image_cropping/`, `sortedm2m_filter_horizontal_widget/`) and third-party packages out of scope. + +To see coverage locally, inside the container: + +```bash +# coverage lives in requirements-dev.txt (test-only, not in the prod image) +pip install -r requirements-dev.txt +coverage run manage.py test website --settings=makeabilitylab.settings_test +coverage report # text table with the uncovered line numbers +coverage html # browsable htmlcov/index.html +``` ## Accessibility Testing diff --git a/README.md b/README.md index 91aa82eb..476bda34 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,7 @@ makeabilitylabwebsite/ | [CONTRIBUTING.md](CONTRIBUTING.md) | Local development setup, workflow, and pull request guidelines | | [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) | Production servers, logging, and data management | | [.pa11yci.json](.pa11yci.json) | Accessibility test configuration (Pa11y + Axe) | +| [.coveragerc](.coveragerc) | Test-coverage configuration (coverage.py; reported in CI) | | [Troubleshooting Wiki](https://github.com/makeabilitylab/makeabilitylabwebsite/wiki/Troubleshooting) | Common issues and solutions | ## 🤝 Contributing