From d8afb6563ddc5901d4f830f55a258a08f39c42c4 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Fri, 7 Aug 2026 10:33:40 -0700 Subject: [PATCH 1/3] Add nightly pg_dump into the postgres volume with health reporting (#1443) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Volume-level snapshots of a live postgres data directory are only probably restorable — they look like a power cut to postgres. CSE IT recommended the standard mitigation: have the database write its own consistent dump into the same volume, so every snapshot they take carries a known-good restore point. A db-backup sidecar in both compose files runs scripts/pg_backup.sh hourly; the script is a single pass that writes one dated `pg_dump | gzip` per UTC day to pg_backups/ inside the data volume, prunes past 14 days (never the newest dump), and writes a status file — on failure too, so a broken backup is distinguishable from one that hasn't run yet. Two details are load-bearing. The sidecar's entrypoint is overridden so the postgres image's own entrypoint can't start a second server on that PGDATA. And the scheduling loop lives in the compose file rather than in the script, because `docker compose up -d` only recreates containers whose config changed — a loop inside the bind-mounted script would keep running stale code after every deploy, with no shell access to restart it. Django only ever reads the status, via a volume mounted read-only: /version.json gains backup_ok/last_backup_at/backup_age_hours/backup_count, the admin dashboard gets a superuser callout shown only when stale or failing, and Data Health gets a detail panel. This reporting isn't polish — the dumps sit in a Docker volume on a host with no shell, inside a PGDATA the website container can't traverse, so without it there's no way to observe whether backups happen at all. Host cron was considered and rejected: no shell on grabthar/docker-test2 means a permanent CSE IT dependency for every schedule change and no visibility. A per-deploy pre-migration dump was also rejected — its marginal recovery over the nightly dump is usually zero rows, and it would cost a second dump code path. Tested, because an untested backup is not a backup. Two Docker harnesses prove dump → destroy volume → restore: scripts/test_backup_restore.sh on a synthetic schema built to break a naive dump (unicode, embedded quotes/newlines, NULLs, bytea, a 540 KB row, views, FKs, sequences), and test_backup_restore_django.sh on the real schema, asserting Django accepts the restored database via `migrate --check` and `manage.py check`. Both also pin the initdb-refuses-a- non-empty-directory gotcha that docs/BACKUPS.md warns about. Unit tests cover the status reader's failure modes — it runs inside each_context, so it must degrade to "unknown" rather than take the admin down. Restore runbook, including what to hand CSE IT for a production restore: docs/BACKUPS.md --- CLAUDE.md | 2 + docker-compose-local-dev.yml | 61 +++- docker-compose.yml | 37 ++ docs/BACKUPS.md | 196 +++++++++++ makeabilitylab/settings.py | 18 + scripts/pg_backup.sh | 192 ++++++++++ scripts/test_backup_restore.sh | 330 ++++++++++++++++++ scripts/test_backup_restore_django.sh | 223 ++++++++++++ website/admin/admin_site.py | 19 + .../admin/data_health/dashboard.html | 88 +++++ website/templates/admin/index.html | 23 ++ website/tests/test_backup_status.py | 213 +++++++++++ website/tests/test_version_endpoint.py | 56 +++ website/utils/backup_status.py | 179 ++++++++++ website/views/version.py | 15 + 15 files changed, 1651 insertions(+), 1 deletion(-) create mode 100644 docs/BACKUPS.md create mode 100755 scripts/pg_backup.sh create mode 100755 scripts/test_backup_restore.sh create mode 100755 scripts/test_backup_restore_django.sh create mode 100644 website/tests/test_backup_status.py create mode 100644 website/utils/backup_status.py diff --git a/CLAUDE.md b/CLAUDE.md index 101df8a9..b3d4e5eb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -132,6 +132,8 @@ the existing viewset/serializer pattern and keep `v1` fields additive-only - `TIME_ZONE = 'America/Los_Angeles'`. `ML_WEBSITE_VERSION` in settings is shown in the admin header and used in release tagging. - **Logging (#1283):** `debug.log` lives at `LOG_DIR/debug.log`, where `LOG_DIR` is `$ML_LOG_DIR` or `/media` (`/code/media` in the container). Keep it inside `MEDIA_ROOT` — that's the tree bind-mounted to the shared CSE filesystem, so it's what makes the log readable over SSH at all. `ML_LOG_DIR` is unset everywhere today; it exists for non-`/code` hosts. `MEDIA_ROOT` is web-served, so never log anything sensitive. If the dir isn't writable the file handler degrades to a `NullHandler` rather than crashing `django.setup()`, and since there's no console on the servers that state surfaces via `/version.json` (`log_to_file`) and a superuser-only callout on the admin dashboard. Rotation uses `concurrent-log-handler` (#1439) because Gunicorn's 3 workers share one file — the stdlib `RotatingFileHandler` races on rollover across processes. Its lock file goes in a per-uid temp dir (`/tmp/makelab-log-locks-`), never the web-served media root and never shared across users. If the package isn't importable (the bind-mounted checkout can be ahead of the image's site-packages) or no lock dir is usable, the handler degrades to the stdlib `RotatingFileHandler` instead of crashing `django.setup()`; `/version.json` reports which one is live as `log_rotation`. `django.db.backends` is pinned to INFO so per-query SQL doesn't dominate the log (or the lock). +- **Database backups (#1443):** a `db-backup` sidecar service (in *both* compose files) runs `scripts/pg_backup.sh` hourly; the script is a single pass that writes one dated `pg_dump | gzip` per UTC day to `pg_backups/` **inside the postgres data volume**, prunes past `BACKUP_RETENTION_DAYS` (14, never the newest dump), and writes `status.json` to a small volume the website container mounts read-only. Dumps go inside the data volume on purpose — that's the volume CSE IT snapshots, so every snapshot carries a consistent restore point. Two things are load-bearing: the sidecar's `entrypoint` **must** stay overridden (the postgres image's own entrypoint would start a second server on that `PGDATA`), and the scheduling loop lives in the compose file rather than in the script (`docker compose up -d` only recreates containers whose *config* changed, so a loop inside the bind-mounted script would run stale code forever after a deploy). Django only ever *reads* the status: `website/utils/backup_status.py` → `/version.json` (`backup_ok`), a superuser callout on the admin dashboard shown only when stale/failed, and a panel on Data Health. Dumps contain `Person.email` — never move one under a web-served path. Restore procedure, the `initdb`-refuses-a-non-empty-directory gotcha, and the two Docker-based restore harnesses (`scripts/test_backup_restore*.sh`): `docs/BACKUPS.md`. + ### Container startup side effects (`docker-entrypoint.sh`) Every container start runs, in order: `collectstatic` → `makemigrations` → `migrate` → `makemigrations website` → `migrate website` → `delete_unused_files` → `thumbnail_cleanup` → `generate_slugs_for_old_news_items` → `auto_close_project_roles` → `remove_year_from_forum_name` → `fix_sortedm2m_columns` → `seed_sidewalk_participants` → `warm_api_thumbnails` → `runserver 0.0.0.0:8000`. The repeated `makemigrations website` step is intentional (fixes first-run issues). If you add a one-shot data migration command under `website/management/commands/`, decide whether it belongs in this startup sequence. diff --git a/docker-compose-local-dev.yml b/docker-compose-local-dev.yml index 2e99894d..411b1243 100644 --- a/docker-compose-local-dev.yml +++ b/docker-compose-local-dev.yml @@ -72,6 +72,60 @@ services: timeout: 5s retries: 5 + # =========================================================================== + # DATABASE BACKUP SERVICE (#1443) + # =========================================================================== + # Writes a dated, gzipped pg_dump into the postgres data volume once a day. + # + # On the servers this exists because volume-level snapshots of a *live* + # postgres data directory are only probably restorable; the dump is the + # guaranteed-consistent restore point. It runs locally too so that the same + # code path is exercised in development rather than only in production. + # + # Two things here are load-bearing and should not be "simplified": + # 1. `entrypoint` is overridden. Left alone, the postgres image's own + # entrypoint would try to start a second database server on this PGDATA. + # 2. The scheduling loop lives here, not inside pg_backup.sh, because + # `docker compose up -d` only recreates containers whose config changed. + # A loop inside the bind-mounted script would keep running stale code + # after a deploy. + # + # To force a backup immediately instead of waiting for the next pass: + # docker compose -f docker-compose-local-dev.yml exec db-backup \ + # bash /backup-scripts/pg_backup.sh + # + # To verify and restore, see docs/BACKUPS.md. + db-backup: + # Same image as `db` so the pg_dump binary version matches the server. + image: 'postgres:16' + restart: always + + environment: + # libpq connection settings; must match the `db` service above. + - PGHOST=db + - PGUSER=admin + - PGPASSWORD=password + - PGDATABASE=makeability + # Delete dumps older than this, but never the most recent one. + - BACKUP_RETENTION_DAYS=14 + # How long to wait between passes, and after a failed pass. + - BACKUP_POLL_SECONDS=3600 + - BACKUP_RETRY_SECONDS=300 + + volumes: + # The database volume, so dumps land inside the thing that gets snapshotted. + - postgres-data:/var/lib/postgresql/data + # Small shared volume for the status file Django reads. + - backup-status:/var/backup-status + # The backup script, mounted as a directory so edits survive git checkouts. + - ./scripts:/backup-scripts:ro + + entrypoint: ["/bin/bash", "-c", "while true; do if bash /backup-scripts/pg_backup.sh; then sleep \"$${BACKUP_POLL_SECONDS}\"; else sleep \"$${BACKUP_RETRY_SECONDS}\"; fi; done"] + + depends_on: + db: + condition: service_healthy + # =========================================================================== # WEBSITE SERVICE (Django Application) # =========================================================================== @@ -94,6 +148,9 @@ services: # This enables "live reloading"—when you edit local files, the changes # are immediately visible inside the container without rebuilding. - .:/code + # Read-only view of the backup status file so Django can report backup + # health on the admin dashboard and /version.json (#1443). + - backup-status:/var/backup-status:ro healthcheck: # Check if Django is responding @@ -158,4 +215,6 @@ services: # Run 'docker volume ls' to see all volumes. # Run 'docker volume rm postgres-data' to delete (WARNING: destroys all data). volumes: - postgres-data: \ No newline at end of file + postgres-data: + # Disposable: holds only the backup status.json, regenerated every pass. + backup-status: \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index 17fde606..923150b9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -10,6 +10,37 @@ services: - POSTGRES_PASSWORD=password volumes: - db-data:/var/lib/postgresql/data + # Consistent daily pg_dump into the postgres data volume (#1443). CSE IT's + # ZFS snapshots capture the raw volume, which is only *probably* restorable + # for a live database; this writes a guaranteed-consistent restore point + # inside that same volume so every snapshot carries one. See docs/BACKUPS.md. + # + # Uses the same image as `db` so pg_dump's version always matches the server. + # `entrypoint` MUST stay overridden: the postgres image's own entrypoint would + # otherwise try to bring up a second server on this PGDATA. + # + # The scheduling loop lives here rather than inside pg_backup.sh on purpose — + # `docker compose up -d` only recreates a container whose config changed, so a + # loop inside the bind-mounted script would keep running the old code forever + # after a deploy. Re-invoking the script each pass makes logic changes deploy. + db-backup: + image: "${POSTGRES_IMAGE:-postgres}" + restart: always + environment: + - PGHOST=db + - PGUSER=admin + - PGPASSWORD=password + - PGDATABASE=makeability + - BACKUP_RETENTION_DAYS=${BACKUP_RETENTION_DAYS:-14} + - BACKUP_POLL_SECONDS=${BACKUP_POLL_SECONDS:-3600} + - BACKUP_RETRY_SECONDS=${BACKUP_RETRY_SECONDS:-300} + volumes: + - db-data:/var/lib/postgresql/data + - backup-status:/var/backup-status + - ./scripts:/backup-scripts:ro + entrypoint: ["/bin/bash", "-c", "while true; do if bash /backup-scripts/pg_backup.sh; then sleep \"$${BACKUP_POLL_SECONDS}\"; else sleep \"$${BACKUP_RETRY_SECONDS}\"; fi; done"] + depends_on: + - db website: environment: - DJANGO_ENV=${DJANGO_ENV:-TEST} @@ -23,6 +54,7 @@ services: - .:/code - ${MEDIA_PATH:-./media}:/code/media - ${CONFIG_PATH:-./config-dev.ini}:/code/config.ini # for loading vars into ConfigParser in Django + - backup-status:/var/backup-status:ro # read-only: Django reports backup health (#1443) depends_on: - db command: ["./docker-entrypoint.sh", "db", "python", "manage.py"] @@ -32,3 +64,8 @@ volumes: db-data: external: true name: "makeabilitylabcswashingtonedu_${POSTGRES_VOLUME:-postgres-data}" + # Holds only the backup status.json that Django reads to report backup health. + # Intentionally NOT external and NOT something CSE IT needs to back up: it is + # disposable state, regenerated on the next backup pass. The dumps themselves + # live in db-data above, which is the volume that gets snapshotted. + backup-status: diff --git a/docs/BACKUPS.md b/docs/BACKUPS.md new file mode 100644 index 00000000..fe9b96a6 --- /dev/null +++ b/docs/BACKUPS.md @@ -0,0 +1,196 @@ +# Database backups and restore + +How the Makeability Lab website's data is backed up, how to check that it's +actually happening, and how to restore it. See issue +[#1443](https://github.com/makeabilitylab/makeabilitylabwebsite/issues/1443). + +**Read the [Restoring](#restoring) section before you need it.** The one step +people get wrong under pressure is documented there: Postgres will not +initialize into a non-empty data directory, and the dumps live *inside* that +directory. + +## What is backed up, and by whom + +| Data | Where it lives | How it's protected | +| --- | --- | --- | +| Uploaded media (PDFs, images) | `/cse/web/research/makelab/www[-test]/` on the shared CSE filesystem | CSE IT's standard snapshot schedule — hourly, weekly, monthly, plus off-site to UW's lolo service. Retained 1 year. Plain files, so a snapshot is always consistent. | +| Code and schema | git | GitHub | +| **Database contents** | the `db` container's named volume (`db-data` → `/var/lib/postgresql/data`) | Two tiers, below | + +The database is the only piece that needs special handling, because a +filesystem-level snapshot of a **live** Postgres data directory is not +guaranteed to be transaction-consistent. Restoring one behaves like recovering +from a hard power cut — usually fine, occasionally not. + +So there are two tiers: + +| Tier | Cadence | Guarantee | +| --- | --- | --- | +| CSE IT's ZFS snapshot of the raw volume | hourly | *Probably* restorable. Postgres crash recovery is designed for exactly this case. | +| Our `pg_dump`, written **into** that volume | daily | Guaranteed consistent restore point. | + +Because the dump lives inside the volume that gets snapshotted, every snapshot +automatically carries a known-good dump. A snapshot from six months ago contains +that day's dump, which is why in-volume retention only needs to be 14 days. + +> **The dump cadence is the guaranteed worst-case RPO, and more snapshots don't +> improve it.** Every hourly snapshot taken between two dumps contains the *same* +> dump. Hourly snapshots give more copies of one restore point, not more restore +> points. + +## How it works + +A `db-backup` sidecar service in `docker-compose.yml` runs +[`scripts/pg_backup.sh`](../scripts/pg_backup.sh) once an hour. The script is a +single pass: if today's dump doesn't exist yet it makes one, prunes anything past +retention, and writes a status file. + +- **Dumps:** `/var/lib/postgresql/data/pg_backups/makeability-YYYY-MM-DD.sql.gz` + (UTC date, mode 0600). +- **Retention:** 14 days, but the newest dump is *never* pruned regardless of + age — otherwise a backup that had been failing for longer than the retention + window would end with pruning deleting the last good dump too. +- **Status:** `status.json` on a small shared volume that the website container + mounts read-only. + +Two things in the compose config are load-bearing and shouldn't be "simplified": + +1. `entrypoint` is overridden. Left alone, the postgres image's own entrypoint + would try to start a second database server on that `PGDATA`. +2. The scheduling loop lives in `docker-compose.yml`, not inside + `pg_backup.sh`. `docker compose up -d` only recreates a container whose + *config* changed, so a loop inside the bind-mounted script would keep running + stale code after a deploy — and there's no shell access to restart it by hand. + +## Checking that backups are actually running + +Because the dumps sit in a Docker volume on a host nobody has a shell on — and +inside a `PGDATA` the website container can't even traverse — the status file is +the only way to observe this. Three places surface it: + +- **`/version.json`** — `backup_ok`, `last_backup_at`, `backup_age_hours`, + `backup_count`. No auth needed; use this for any external check. +- **Admin dashboard** — a superuser-only warning callout, shown *only* when + backups are stale or failing. +- **Admin → Data Health** — a panel with last success, age, size, how many + dumps are retained, and the last error. + +"Stale" means the newest dump is more than 36 hours old (`BACKUP_STALE_AFTER_HOURS`). +That's 1.5× the daily cadence, so one missed run doesn't cry wolf but a second +consecutive one does. + +## Restoring + +### The gotcha, first + +The dumps live at `pg_backups/` **inside** the Postgres data directory. `initdb` +refuses to initialize into a non-empty directory, so you cannot wipe the database +and leave the backups sitting there. **Copy the dump out of the volume first.** +This is pinned by a test in `scripts/test_backup_restore.sh` so the warning can't +silently go stale. + +### Restoring locally (development, or verifying a dump) + +```bash +# 1. Get the dump out of the volume and onto your machine. +docker compose -f docker-compose-local-dev.yml cp \ + db-backup:/var/lib/postgresql/data/pg_backups/makeability-2026-08-07.sql.gz . + +# 2. Stop the stack and destroy the database volume. +docker compose -f docker-compose-local-dev.yml down +docker volume rm makeabilitylabwebsite_postgres-data + +# 3. Bring just the database back up on a fresh, empty volume. +docker compose -f docker-compose-local-dev.yml up -d db + +# 4. Restore. +gunzip -c makeability-2026-08-07.sql.gz | \ + docker compose -f docker-compose-local-dev.yml exec -T db \ + psql -v ON_ERROR_STOP=1 -U admin -d makeability + +# 5. Start the site and confirm Django agrees the database is complete. +docker compose -f docker-compose-local-dev.yml up -d +docker compose -f docker-compose-local-dev.yml exec website python manage.py migrate --check +``` + +Step 5 is the real test. `migrate --check` exits non-zero if Django thinks +migrations are pending, which is how you'd catch a restore that brought back +tables but not the `django_migrations` table. + +### Restoring production or test + +**This requires someone with Docker access on the host** — `grabthar` for +production, `docker-test2` for test. The maintainer does not have that (see the +server access model in `CLAUDE.md`), so a production restore means opening a +ticket with UW CSE IT. Send them this section. + +The steps are the same as above, against `docker-compose.yml` and the external +volume `makeabilitylabcswashingtonedu_postgres16-data`. Before destroying +anything: + +1. **Copy the chosen dump somewhere off the volume first.** If the volume itself + is the problem, ask CSE IT to recover the dump from a ZFS snapshot or from + lolo instead — the dump inside a snapshot is exactly what this whole scheme + exists to provide. +2. **Take a copy of the current broken volume before overwriting it.** A + corrupt database still contains data; a hasty restore over the top of it + destroys any chance of salvaging rows the dump predates. +3. Restore, then confirm via `/version.json` and by loading the site. + +There's no ad-hoc "back up right now" button, deliberately — see the follow-up +note in #1443. If you need a fresh dump before something risky and you can't +reach the host, the practical options are to wait for the next pass or ask CSE +IT to run one. + +## Testing the backups + +Two harnesses, both self-contained and namespaced so they never touch your real +stack, database, or volumes. **An untested backup is not a backup** — run these +after any change to `pg_backup.sh` or the compose wiring. + +```bash +# Mechanics: dump → destroy → restore, on a synthetic schema built to break a +# naive dump (unicode, embedded quotes and newlines, NULLs, binary columns, +# a 540 KB row, views, foreign keys, sequences). ~1 minute. +bash scripts/test_backup_restore.sh + +# The real thing: builds the actual Makeability Lab schema with `migrate`, seeds +# through the ORM (sortedm2m through-tables, rich text), backs up, destroys the +# volume, restores, and asserts Django accepts the result — `migrate --check` +# and `manage.py check` both pass. Needs a built website image. ~3 minutes. +bash scripts/test_backup_restore_django.sh +``` + +The Django-side unit tests (status file parsing, staleness, failure reporting) +run in the normal suite: + +```bash +python manage.py test website.tests.test_backup_status --settings=makeabilitylab.settings_test +``` + +## Security + +The dumps contain personal data — `Person.email`, which is deliberately withheld +from the public API. They are written mode 0600 into a Docker volume. + +**Never move a dump under `media/`, `static/`, or any other web-served path.** +Everything under those is publicly downloadable. The status file is safe to +surface in the admin because it carries no row data. + +## Configuration + +Set on the `db-backup` service in `docker-compose.yml`: + +| Variable | Default | Meaning | +| --- | --- | --- | +| `BACKUP_RETENTION_DAYS` | 14 | Delete dumps older than this (never the newest). | +| `BACKUP_MIN_KEEP` | 1 | Dumps always kept regardless of age. | +| `BACKUP_POLL_SECONDS` | 3600 | Time between passes. | +| `BACKUP_RETRY_SECONDS` | 300 | Time between passes after a failure. | + +Django-side, in `settings.py`: + +| Setting | Default | Meaning | +| --- | --- | --- | +| `BACKUP_STATUS_FILE` | `/var/backup-status/status.json` | Where to read status from (`ML_BACKUP_STATUS_FILE`). | +| `BACKUP_STALE_AFTER_HOURS` | 36 | Age at which the dashboard warns (`ML_BACKUP_STALE_AFTER_HOURS`). | diff --git a/makeabilitylab/settings.py b/makeabilitylab/settings.py index 7ecb35b6..98f2bed8 100644 --- a/makeabilitylab/settings.py +++ b/makeabilitylab/settings.py @@ -365,6 +365,24 @@ def _file_log_handler(log_file, level, enabled): f"multiprocess-safe (concurrent-log-handler importable: " f"{_HAS_CONCURRENT_LOG_HANDLER}). Check /version.json 'log_rotation'.") +# --------------------------------------------------------------------------- +# Database backup health (#1443) +# --------------------------------------------------------------------------- +# The db-backup sidecar writes a small JSON status file to a volume shared +# read-only with this container. Django reads it purely to *report* backup +# health; it never writes here and never touches the dumps themselves. +# +# The reporting is the whole point: the dumps live in a Docker named volume on +# a host nobody has a shell on, and PGDATA is mode 700/uid 999 while this +# container runs as uid 48 — so without this file there is no way to observe +# whether backups are running at all. Same reasoning as LOG_TO_FILE above. +BACKUP_STATUS_FILE = os.environ.get('ML_BACKUP_STATUS_FILE', '/var/backup-status/status.json') + +# How old the newest dump may get before the admin dashboard complains. 36h, +# not 24h: backups run daily, so one missed pass or a little clock skew should +# not raise an alarm — but a second consecutive miss should. +BACKUP_STALE_AFTER_HOURS = int(os.environ.get('ML_BACKUP_STALE_AFTER_HOURS', '36')) + # Application definition INSTALLED_APPS = [ 'website.apps.WebsiteConfig', diff --git a/scripts/pg_backup.sh b/scripts/pg_backup.sh new file mode 100755 index 00000000..e55a869b --- /dev/null +++ b/scripts/pg_backup.sh @@ -0,0 +1,192 @@ +#!/bin/bash +# +# One backup pass for the Makeability Lab database (#1443). +# +# Writes a dated, gzipped pg_dump into a subdirectory of the postgres data +# volume so that CSE IT's volume-level ZFS snapshots always contain a +# transaction-consistent restore point. A filesystem snapshot of a *live* +# PGDATA is only probably restorable (it looks like a power cut to postgres); +# this dump is the guaranteed tier. See docs/BACKUPS.md. +# +# This script deliberately performs a SINGLE pass and exits. The scheduling +# loop lives in docker-compose.yml instead, because `docker compose up -d` +# only recreates a container whose *config* changed: if the loop lived here, +# in this bind-mounted file, edits to the backup logic would never take effect +# on a running server (and there is no shell access to restart it by hand). +# Re-invoking the script each pass means logic changes deploy normally. +# +# Runs as root, which is needed to write the shared status volume: Docker +# creates named-volume roots as root:root, so an unprivileged uid could not +# create files there. The container's only job is to run pg_dump over the +# stack's private network. +# +# Environment: +# PGHOST/PGUSER/PGPASSWORD/PGDATABASE standard libpq connection settings +# BACKUP_DIR where dumps are written (default: inside PGDATA) +# STATUS_DIR where status.json is written for Django to read +# BACKUP_RETENTION_DAYS delete dumps older than this (default 14) +# BACKUP_MIN_KEEP never prune below this many dumps (default 1) +# +# Usage (one pass, as run by the db-backup service): +# bash /backup-scripts/pg_backup.sh + +# No `set -e`: a failed dump must still fall through to write a status file +# recording the failure, otherwise a broken backup is indistinguishable from +# one that has simply not run yet. pipefail is required so that a pg_dump +# failure is not masked by gzip's success. +set -uo pipefail + +BACKUP_DIR="${BACKUP_DIR:-/var/lib/postgresql/data/pg_backups}" +STATUS_DIR="${STATUS_DIR:-/var/backup-status}" +STATUS_FILE="$STATUS_DIR/status.json" +RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-14}" +MIN_KEEP="${BACKUP_MIN_KEEP:-1}" +DB_NAME="${PGDATABASE:-makeability}" + +# Dumps contain personal data (Person.email, which is deliberately withheld +# from the public API), so they are created mode 0600. They live in a Docker +# volume, never under a web-served path. +umask 077 + +now_iso() { date -u +%Y-%m-%dT%H:%M:%SZ; } + +# Escape a string for embedding in a JSON string literal, and collapse it to a +# single line. Only used for error text, which is untrusted-ish (stderr from +# pg_dump) and must not be able to corrupt the status file's JSON. +json_escape() { + printf '%s' "$1" | tr '\n\r\t' ' ' \ + | sed -e 's/\\/\\\\/g' -e 's/"/\\"/g' | cut -c1-500 +} + +file_mtime_iso() { date -u -d "@$(stat -c %Y "$1")" +%Y-%m-%dT%H:%M:%SZ; } + +# --------------------------------------------------------------------------- +# Write status.json atomically (temp file + rename) so Django never reads a +# half-written file. Mode 0644 because the website container reads it as a +# different uid (48/apache) than this one. +# --------------------------------------------------------------------------- +write_status() { + local ok="$1" error="$2" + local last_at='null' last_file='null' last_bytes='null' + local oldest_at='null' count=0 + local newest oldest + + newest="$(find "$BACKUP_DIR" -maxdepth 1 -type f -name '*.sql.gz' -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -n 1 | cut -d' ' -f2-)" + oldest="$(find "$BACKUP_DIR" -maxdepth 1 -type f -name '*.sql.gz' -printf '%T@ %p\n' 2>/dev/null | sort -n | head -n 1 | cut -d' ' -f2-)" + count="$(find "$BACKUP_DIR" -maxdepth 1 -type f -name '*.sql.gz' 2>/dev/null | wc -l)" + + if [ -n "$newest" ]; then + last_at="\"$(file_mtime_iso "$newest")\"" + last_file="\"$(json_escape "$(basename "$newest")")\"" + last_bytes="$(stat -c %s "$newest")" + fi + if [ -n "$oldest" ]; then + oldest_at="\"$(file_mtime_iso "$oldest")\"" + fi + + local error_json='null' + if [ -n "$error" ]; then + error_json="\"$(json_escape "$error")\"" + fi + + mkdir -p "$STATUS_DIR" + local tmp="$STATUS_FILE.tmp" + cat > "$tmp" </dev/null \ + | sort -rn | head -n "$MIN_KEEP" | cut -d' ' -f2-) + + find "$BACKUP_DIR" -maxdepth 1 -type f -name '*.sql.gz' \ + -mtime "+$RETENTION_DAYS" "${keep_args[@]}" -delete 2>/dev/null +} + +# --------------------------------------------------------------------------- +# Main pass +# --------------------------------------------------------------------------- +if ! mkdir -p "$BACKUP_DIR"; then + # Can't even reach the volume; still try to report it. + write_status false "Could not create backup directory $BACKUP_DIR" + exit 1 +fi +chmod 0700 "$BACKUP_DIR" + +# UTC date, so the rollover point does not shift with daylight saving. Note +# this means the file named for a given day rolls over at 5pm/4pm Pacific. +target="$BACKUP_DIR/${DB_NAME}-$(date -u +%F).sql.gz" + +if [ -f "$target" ]; then + # Today's dump already exists. Still prune and refresh status so a running + # container keeps its status file current between dumps. + prune_old_backups + write_status true "" + exit 0 +fi + +tmp_target="$target.partial" +err_file="$(mktemp)" + +# --no-owner/--no-privileges keep the dump restorable into a database owned by +# a differently-named role. There is no separate pg_dumpall for globals: the +# only role is the one POSTGRES_USER recreates on any fresh container. +pg_dump --no-owner --no-privileges "$DB_NAME" 2>"$err_file" | gzip -c > "$tmp_target" +dump_status=$? + +if [ "$dump_status" -ne 0 ]; then + rm -f "$tmp_target" + write_status false "pg_dump failed (exit $dump_status): $(cat "$err_file")" + rm -f "$err_file" + exit 1 +fi + +# Catch a truncated or corrupt archive before it is promoted to today's dump — +# otherwise a bad file would satisfy the "today already done" check above and +# suppress every retry until tomorrow. +if ! gzip -t "$tmp_target" 2>>"$err_file"; then + rm -f "$tmp_target" + write_status false "gzip integrity check failed: $(cat "$err_file")" + rm -f "$err_file" + exit 1 +fi + +# Promote atomically: a container killed mid-dump leaves only a .partial file, +# never something that looks like a finished backup. +mv -f "$tmp_target" "$target" +rm -f "$err_file" + +prune_old_backups +write_status true "" +echo "Wrote $(basename "$target") ($(stat -c %s "$target") bytes)" diff --git a/scripts/test_backup_restore.sh b/scripts/test_backup_restore.sh new file mode 100755 index 00000000..7ed2be0e --- /dev/null +++ b/scripts/test_backup_restore.sh @@ -0,0 +1,330 @@ +#!/bin/bash +# +# End-to-end proof that scripts/pg_backup.sh produces a dump that actually +# restores (#1443). An untested backup is not a backup. +# +# This exercises the real disaster path, not a simulation of it: +# 1. stand up a throwaway postgres and seed it with awkward data +# 2. run the REAL scripts/pg_backup.sh against it +# 3. assert the dump and status file look right +# 4. copy the dump out, then DESTROY the container and its volume +# 5. bring up postgres on a genuinely fresh volume and restore +# 6. assert the restored database is byte-identical in content +# +# It also pins the initdb-refuses-a-non-empty-directory behavior that +# docs/BACKUPS.md warns about, so that caveat can't silently go stale. +# +# Everything is namespaced under mlbackuptest-* and torn down at the end, so +# this never touches the developer's real stack, database, or volumes. +# +# Usage: +# bash scripts/test_backup_restore.sh +# +# Requires: docker, and the postgres image used by the stack. + +set -uo pipefail + +PROJECT=mlbackuptest +IMAGE="${POSTGRES_IMAGE:-postgres:16}" +NET="$PROJECT-net" +DATA_VOL="$PROJECT-data" +STATUS_VOL="$PROJECT-status" +DB_CONTAINER="$PROJECT-db" +DB_NAME=makeability +DB_USER=admin +DB_PASS=password + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Under /tmp specifically: Docker Desktop on macOS shares /tmp by default, and +# the rescued dump has to be bind-mountable back into a container. +WORK_DIR="$(mktemp -d /tmp/mlbackuptest.XXXXXX)" + +PASS=0 +FAIL=0 + +green() { printf '\033[32m%s\033[0m\n' "$1"; } +red() { printf '\033[31m%s\033[0m\n' "$1"; } +step() { printf '\n\033[1m== %s\033[0m\n' "$1"; } + +ok() { PASS=$((PASS+1)); green " PASS $1"; } +bad() { FAIL=$((FAIL+1)); red " FAIL $1"; } + +assert_eq() { + # assert_eq