docs: add ADR 0005 for Glow-to-NLDS warehouse connector

[nicholasjjlim]

🚀 Summary

Resolves #597 — a research spike (no implementation) producing a documented, recommended approach for connecting Glow learning data to the NLDS data warehouse for centralized analytics and reporting.

The ADR (docs/decisions/0005-glow-nlds-warehouse-connector.md, MADR 4.0, status accepted) recommends an S3 data-lake intermediary with RDS snapshot export as the producer: Glow's database exports whole-database snapshots to S3 as Parquet (no export code), landing cross-account in a scoped prefix of a bucket sited in the NLDS account; NLDS ingests natively within its own account on its own schedule.

There is no requirement to curate what is sent — Glow's learning data is not sensitive and NLDS decides how to use it — so the connector exports the whole database. Curating to specific datasets can be re-explored later; because that would only swap the producer within the same S3 pattern, it would stay within this decision rather than supersede it.

Decision rests on the established constraints: batch freshness is acceptable (no near-real-time need), Glow and NLDS are separate AWS accounts in the same AWS Organization, and no policy mandates a particular network posture.

✏️ Changes

• Add docs/decisions/0005-glow-nlds-warehouse-connector.md — evaluates five options across the rubric dimensions (cost, security, freshness/latency, implementation/operational effort, coupling):
  • S3 data-lake intermediary — chosen (cheapest, least network exposure, lowest effort; matches batch freshness). Content-agnostic: the same path carries a whole-DB dump or curated slices equally well.
  • RDS snapshot export to S3 — chosen as the producer for the intermediary: lowest effort of all (a configured export, no code), reads an automated snapshot rather than the live instance, and writes analytics-friendly Parquet. Its usual drawback — exporting the entire database — does not apply, since there is no curation requirement.
  • DB replica via AWS PrivateLink — rejected; near-real-time not needed, exposes a live DB replica. Kept as the documented upgrade path.
  • DB replica via VPC peering — rejected; broadest network exposure, CIDR coordination, no freshness gain over PrivateLink.
  • AWS DMS — rejected as a rival; reads the source DB and runs a standing replication instance. Noted as the likely loading mechanism if a CDC upgrade ever happens.
• Includes a "Bucket ownership" sub-decision (bucket in NLDS account; bucket-owner-enforced to avoid the cross-account object-ownership trap), a Mermaid data-flow diagram, and a deferred-patterns note (federated query, streaming CDC, app-level extract API).
• Add the ADR to the decisions index in docs/decisions/README.md.

Notes

• Status is accepted — signed off by the Glow engineering and NLDS Product Ops reps.
• Revised after acceptance to drop the curated-export requirement: stakeholders confirmed there is no need to filter what is sent, so the recommendation moved from a purpose-built curated export to whole-DB RDS snapshot export within the same S3 architecture.
• Out of scope per the issue: connector implementation, warehouse schema/table mapping, and historical backfill. Dataset curation is likewise deferred — re-explorable later without superseding this ADR.

🤖 Generated with Claude Code