Nightly pg_dump into the postgres volume, with backup health reporting (#1443)

[jonfroehlich]

Closes #1443.

A filesystem-level snapshot of a live Postgres data directory isn't guaranteed to be transaction-consistent — restoring one behaves like recovering from a hard power cut. UW CSE IT flagged this and recommended the standard mitigation: have the database write its own consistent dump on a schedule, into the same volume they snapshot, so every snapshot automatically carries a known-good restore point.

What this adds

A db-backup sidecar service (in both compose files) runs scripts/pg_backup.sh hourly. The script is a single pass: if today's dump doesn't exist it makes one, prunes anything past retention, and writes a status file.

• Dumps → /var/lib/postgresql/data/pg_backups/makeability-YYYY-MM-DD.sql.gz, mode 0600, atomic temp+rename, gzip -t verified before promotion.
• Retention 14 days, but the newest dump is never pruned regardless of age — otherwise a backup that had been failing for longer than the window would end with pruning deleting the last good dump.
• Status JSON written on failure too, with exit code and stderr tail. A failed dump that wrote nothing would be indistinguishable from one that simply hadn't run.

Django only ever reads the status, through a read-only mount:

• /version.json gains backup_ok, last_backup_at, backup_age_hours, backup_count
• Admin dashboard: superuser callout, shown only when stale or failing
• Data Health: an always-present detail panel

That reporting isn't polish. The dumps sit in a Docker volume on a host with no shell, inside a PGDATA the website container (uid 48) can't even traverse — without the status file there's no way to observe whether backups happen at all. Same reasoning as log_to_file in #1283.

Two things that are load-bearing

1. The sidecar's entrypoint is overridden. Left alone, the postgres image's own entrypoint would try to start a second server on that PGDATA.
2. The scheduling loop lives in the compose file, not in the script. docker compose up -d only recreates a container whose config changed — a loop inside the bind-mounted script would keep running stale code after every deploy, with no shell access to restart it.

Decisions

• Sidecar, not host cron. No shell on grabthar/docker-test2, so host cron would be a permanent CSE IT dependency for every schedule or retention change, with no visibility into whether it was still running.
• No per-deploy pre-migration dump (considered, rejected). It only beats the nightly dump when a destructive deploy lands on the same day as meaningful content entry — usually zero marginal rows — and costs a second dump code path plus DJANGO_ENV gating so it wouldn't fire on every push to -test. The real control for destructive one-shots is what we already do: make them idempotent and disable them once run (Re-standardize legacy talk/poster/pub filenames that were never renamed #1401, Store original uploaded filename and show it (admin-only) for talks/posters/publications #1391).
• No pg_dumpall --globals-only. The only role is admin, recreated from POSTGRES_USER on any fresh container.

Testing

An untested backup is not a backup, so the restore path is proven rather than assumed. Two Docker harnesses both genuinely destroy the volume and restore onto an empty one:

Harness	Assertions	What it proves
scripts/test_backup_restore.sh	35	Dump mechanics on a schema built to break a naive dump: unicode, embedded quotes/newlines, NULLs, bytea, a 540 KB row, views, foreign keys, sequences. Plus retention pruning, the never-delete-the-last-dump guard, failure reporting, and same-day idempotency.
scripts/test_backup_restore_django.sh	15	The real Makeability Lab schema (54 tables) built by migrate, seeded through the ORM including SortedManyToManyField through-tables and rich text. Asserts Django accepts the restored database: migrate --check and manage.py check both pass.

Both also pin the initdb-refuses-a-non-empty-directory behavior that docs/BACKUPS.md warns about, so that caveat can't silently go stale. Neither touches your real stack, database, or volumes.

Django-side unit tests (website/tests/test_backup_status.py, 28 with the /version.json additions) cover the status reader's failure modes — missing file, truncated JSON, non-object JSON, unparseable timestamps, missing keys. That code runs inside each_context, so it must degrade to "unknown" rather than take the whole admin down.

I also brought up the real compose sidecar and confirmed the full chain end to end: sidecar writes → uid 48 reads through the read-only mount → get_backup_status() reports healthy.

Full suite: 763 tests, 2 failures, both in test_logging_config and both pre-existing — the local image predates #1439's concurrent-log-handler dependency. They fail identically on untouched master with the same image, and pass on this branch once the package is installed.

Notes for review

• No screenshots yet. The UI is admin-only (a warning callout and a Data Health panel), so it's behind login and not reachable by Pa11y — .pa11yci.json has no admin URLs. The callout reuses the existing AA-verified .ml-log-warning palette rather than introducing new colors, and the panel uses <th scope="row">. Happy to add screenshots if you want them before merge.
• The backup-status volume is deliberately not external and not something CSE IT needs to back up — it holds only the status file and is regenerated every pass. The dumps live in db-data, which is the volume that gets snapshotted.
• Dumps contain Person.email, which is deliberately withheld from the public API. They're written 0600 into a Docker volume and must never be moved under a web-served path.
• docs/BACKUPS.md has the restore runbook, including the section to hand CSE IT for a production restore (which needs someone with Docker access on the host — we don't have it).

🤖 Generated with Claude Code — Opus 5 (1M context), claude-opus-5[1m]