Skip to content

feat: NLDS mandatory-quiz-completion export#608

Draft
nicholasjjlim wants to merge 8 commits into
mainfrom
feat/nlds-export
Draft

feat: NLDS mandatory-quiz-completion export#608
nicholasjjlim wants to merge 8 commits into
mainfrom
feat/nlds-export

Conversation

@nicholasjjlim

Copy link
Copy Markdown
Collaborator

🚀 Summary

Adds a daily batch export that ships two curated datasets from the Onward DB to the NLDS-owned S3 data lake as NDJSON with mandatory SSE-KMS, so NLDS can track mandatory quiz completion (which users have / have not completed the mandatory modules). Invoked by the scheduled ECS task via pnpm run export:nlds; runs to completion and exits 0 on full success, non-zero on any failure.

Design docs land as the first commit:

  • ADR: docs/decisions/0006-package-nlds-export-as-bundled-script-in-app-image.md
  • Spec: docs/superpowers/specs/2026-07-01-nlds-export-design.md
  • Plan: docs/superpowers/plans/2026-07-01-nlds-export.md

✏️ Changes

  • scripts/export-nlds.ts — standalone export script: env-config loader (fail-loud on any missing var), two datasets (users; mandatory_quiz_outcomes = journeys where LearningUnit.is_required = true), NDJSON serializer, SSE-KMS S3 uploader, and a sequential orchestrator with an entry guard. Read-only select-only queries, explicit columns (no SELECT *), keys scoped under NLDS_PREFIX.
  • Packaging (per ADR-0006) — bundled at Docker build time to build/export-nlds.js (generated Prisma client inlined; AWS SDK / Prisma runtime / pg kept external). Adds esbuild as a devDependency (build-stage only), two npm scripts (export:nlds, export:nlds:bundle), and one RUN pnpm export:nlds:bundle step in the Dockerfile (after pnpm build, before the prod prune). No new runtime dependencies; no new COPY.
  • Tests — 16 unit tests (config validation, key/date/NDJSON helpers, dataset select-shape + snake_case/date/null mapping, SSE-KMS put fields, orchestrator sequencing + abort, prefix scoping). pnpm check clean.

Datasets

Object Fields
${NLDS_PREFIX}users/<UTC-date>.ndjson id, name, email (PII — scoped to exactly these), created_at
${NLDS_PREFIX}mandatory_quiz_outcomes/<UTC-date>.ndjson user_id, learning_unit_id, learning_unit_title, due_date, is_completed, is_quiz_attempted, is_quiz_passed, number_of_attempts, updated_at

Full daily snapshot (overwrites the day's key; idempotent).

⚠️ Pre-merge verification still required

These couldn't run in the dev sandbox (no Docker network, no dev DB/KMS) — please run in CI/staging before merging:

  • docker build of the image, then in-container node build/export-nlds.js to confirm the three externals resolve in the pruned --prod image.
  • A live run against a dev DB + test prefix → expect two .ndjson objects + exit 0; force a missing NLDS_KMS_KEY_ID → expect non-zero.

The bundle load was smoke-tested locally (loads cleanly, exits non-zero on missing env), but only a real container + KMS-scoped PutObject confirms the operational path.

🤖 Generated with Claude Code

nicholasjjlim and others added 8 commits July 1, 2026 18:31
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oping

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant