Testing automation roadmap: CI, coverage, and the path beyond localhost tests

[jonfroehlich]

Summary

Tracking issue capturing a design discussion (June 2026) on maturing our testing setup. We started writing tests for the first time a few days ago (see #1267); the suite now lives in website/tests.py (~1,580 lines) with a solid two-tier design — SimpleTestCase + MagicMock for pure logic, DatabaseTestCase for the DB/view/template layer. The tests exist; nothing runs them automatically.

The core gap

We auto-deploy master to the test server on every push:

push to master → live on makeabilitylab-test.cs.washington.edu, with no checks in between.

So the regression tests we just wrote only protect us if a contributor remembers to run manage.py test in the container before pushing. With rotating student contributors and push-to-deploy, that honor-system gate is exactly what continuous integration (CI) removes.

Mechanics / cost (for the record)

• GitHub Actions is free and unlimited for public repos (this repo is public). No billing, no minute caps. A run of our suite is ~1–3 min.
• A failing test is a signal, not a gate: it shows a red ✗ on the commit and emails the author. It does not block, revert, or alter pushed code. Merge-blocking only happens if we opt into branch protection later.

Prioritized roadmap

• 1. GitHub Actions CI — run manage.py test website on PRs (and optionally pushes to master) against a Postgres 16 service container. Highest leverage. ✅ Done in Add CI + test-settings shim + split tests into a package (#1279) #1280.
• 2. Test-settings shim — makeabilitylab/settings_test.py with MIGRATION_MODULES = {'website': None} + --run-syncdb so the runner builds schema from models, bypassing the gitignored per-env migration history. Durable fix for the column already exists flakiness (Need much better testing infracture on local host #1267). Prerequisite for CI. ✅ Done in Add CI + test-settings shim + split tests into a package (#1279) #1280.
• 3. Split tests.py into a tests/ package — tests/base.py (shared DatabaseTestCase + fixtures) plus test_*.py by concern. Django auto-discovers; pure move, no config. ✅ Done in Add CI + test-settings shim + split tests into a package (#1279) #1280.
• 4. Coverage measurement — coverage.py (stay on Django's runner; no pytest). Report in CI, don't gate on a number yet — use it to target backfill. ✅ Done in ci(coverage): measure test coverage in CI, report-only (#1278 item 4) #1322 (report-only, baseline 59%).
• 5. Backfill high-risk untested code — ranked: delete_unused_files.py (runs every container start, deletes files, two latent crashes already found); a parametrized view smoke-sweep (GET every public URL → 200, catches the NoReverseMatch/AttributeError template bug class); Person.save() side effects; pure utils (timeutils, ml_utils, fileutils). ✅ Done in Test backfill for high-risk code + two latent 500 fixes (#1278 item 5) #1339 (released 2.12.2; coverage 59%→69%; the backfill found+fixed two latent 500s — Artifact.save() on empty pdf_file, get_project_dates_str() on null start_date).
• 6. Pa11y in CI — we already have the service + .pa11yci.json; CONTRIBUTING requires it on UI changes but nothing enforces it. Wire into the workflow. ✅ Done in Wire Pa11y accessibility sweep into CI (report-only) (#1278 item 6) #1340 (+ Exclude django-debug-toolbar from the Pa11y CI scan (#1278) #1342). Report-only a11y job; real baseline posted to Improve accessibility of Makeability Lab website #1240.
• 7. JS unit tests — deferred. ~2,633 lines of vanilla JS (incl. a11y-critical lightbox.js), but a Jest/node harness adds the build step we deliberately avoid. Let Pa11y cover rendered a11y for now; revisit only if we want unit-level JS coverage.

First PR

Items 1 + 2 + 3 shipped together in #1279 / PR #1280 (merged 2026-06-15) — CI now runs the suite on every push to master and every PR.

Remaining: item 7 (deferred) + optional hardening follow-ups — see the latest status comment. Roadmap items 1–6 are complete.

Related

• Need much better testing infracture on local host #1267 — localhost test foundation (the two-tier suite + fixtures; largely landed)
• Realistic test data: factory_boy fixtures + curated prod media subset #1272 — factory_boy fixtures + realistic media (composes with item 5)

[jonfroehlich]

Status (2026-06-17): item 4 (coverage measurement) is up as #1322 — CI test job now wraps the suite in coverage run and posts a per-file table to the run Summary. Report-only (no --fail-under gate), config in .coveragerc (measures the website app, branch coverage on). Baseline locally: 59% of the website app. coverage==7.14.1 lives in requirements-dev.txt (test-only, not in the prod image). Docs updated (CONTRIBUTING/README/CLAUDE).

Remaining (each its own PR):

• Item 5 — backfill high-risk untested code. Use the new coverage report to target; the issue flags delete_unused_files.py (runs every container start, deletes files) and a parametrized view smoke-sweep (GET every public URL → 200) as the top picks.
• Item 6 — Pa11y in CI. Wire the existing .pa11yci.json service into the workflow.
• Item 7 (JS unit tests). stays deferred.