From 1bb79b1810ebd4d3114b0cd656910deb5db88141 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 6 Aug 2026 11:35:14 -0400 Subject: [PATCH] Document public+gated HuggingFace repo posture The policyengine-uk-data-private HF repo changed from private to public + gated (manual approval) on 31 July 2026 so that external access grants function (they are no-ops on private repos). Update the data-safety rules in CLAUDE.md and CONTRIBUTING.md to protect the gate rather than the private flag, and warn against flipping visibility back, which voids approved external grants. Refs PolicyEngine/policyengine-uk#1816. Co-Authored-By: Claude Fable 5 --- .github/CONTRIBUTING.md | 2 +- CLAUDE.md | 2 +- changelog.d/hf-public-gated-docs.changed.md | 1 + 3 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 changelog.d/hf-public-gated-docs.changed.md diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index ffc952586..73a49779f 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -30,7 +30,7 @@ This repo builds the `.h5` files that feed `policyengine-uk`: The enhanced FRS dataset is licensed under strict UK Data Service terms. Violating them risks losing access, which would end PolicyEngine UK. -- **Never upload data to any public location.** The HuggingFace repo `policyengine/policyengine-uk-data-private` is private and authenticated. +- **Never expose data contents publicly.** The HuggingFace repo `policyengine/policyengine-uk-data-private` is public + **gated** (manual approval): file contents are restricted to the org and to UKDS-licensed users approved through the gate. Never disable or weaken the gate, never grant access without UKDS proof, and never upload UKDS-derived data to any ungated public location. Do not flip the repo back to private — that voids all approved external grants (see PolicyEngine/policyengine-uk#1816). - **Never modify `upload_completed_datasets.py` or `utils/data_upload.py`** to change upload destinations without explicit confirmation from the data controller (currently Nikhil Woodruff). - **Never print, log, or output individual-level records.** Aggregates (sums, means, counts, weighted totals) are fine; individual rows are not. - **If you see a private/public repo split, assume it is intentional** — ask why before changing it. diff --git a/CLAUDE.md b/CLAUDE.md index e786ba4de..636bbd433 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ The purpose of this repo is to build the .h5 files that feed as input into the p ### Rules — no exceptions -1. **NEVER upload data to any public location.** The HuggingFace repo `policyengine/policyengine-uk-data-private` is private and authenticated. The separate public repo (`policyengine/policyengine-uk-data`) is maintained through a separate process — do NOT modify the upload pipeline to push data there. +1. **NEVER expose data contents publicly.** The HuggingFace repo `policyengine/policyengine-uk-data-private` is **public + gated** (manual approval) as of 31 July 2026: file names are world-visible, but file contents are restricted to the org and to external users approved through the gate after showing UKDS FRS access. NEVER disable or weaken the gate, NEVER grant gate access without UKDS proof, and NEVER upload UKDS-derived data to any ungated public location. Do NOT flip the repo back to private to "fix" CI auth errors — private repos silently void every approved external grant (see PolicyEngine/policyengine-uk#1816); grant the CI account through the gate instead. The separate public repo (`policyengine/policyengine-uk-data-public`) is maintained through a separate process — do NOT modify the upload pipeline to push data there. 2. **NEVER modify `upload_completed_datasets.py` or `data_upload.py` to change upload destinations** without explicit confirmation from the data controller (currently Nikhil Woodruff). 3. **NEVER print, log, or output individual-level records** from the dataset. Aggregates (sums, means, counts, weighted totals) are fine; individual rows are not. 4. **If you see a private/public repo split, assume it is intentional** — ask why before changing it. diff --git a/changelog.d/hf-public-gated-docs.changed.md b/changelog.d/hf-public-gated-docs.changed.md new file mode 100644 index 000000000..8acb942a3 --- /dev/null +++ b/changelog.d/hf-public-gated-docs.changed.md @@ -0,0 +1 @@ +Documentation: the private HuggingFace repo is now public + gated (manual approval) as of 31 July 2026 — file contents remain restricted; never weaken the gate or flip the repo back to private.