From 3c44dd59f0cdd433fca05aab49e933e6a137089a Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Wed, 12 Aug 2026 13:51:55 +0100 Subject: [PATCH] Send HF_TOKEN so gated dataset downloads authenticate Every dataset-backed job on main and on PRs has failed with 401 since 31 July 2026. The cause is not the HUGGING_FACE_TOKEN secret: no token is being sent at all. policyengine-core's download_huggingface_dataset passes a token to hf_hub_download only when the repo reports private=True (tools/hugging_face.py:81-90). policyengine-uk-data-private has been public + gated (manual approval) since 31 July, so private is False, core passes token=None, the download goes out anonymously and the gate returns 401 GatedRepoError. Replacing the secret on 10 August could not have fixed this, and the out-of-band validation of that token passed because it supplied the token explicitly - the step core skips. huggingface_hub falls back to the HF_TOKEN environment variable when the explicit token is None (get_token_to_send -> get_token), so exporting the same secret under that name restores authentication. Verified green on PolicyEngine/policyengine-uk#1815, where the gated dataset downloads and the microsimulation suite passes (177 passed, 1 skipped). This is a workaround. The durable fix belongs in policyengine-core, which should pass the token whenever one is available, or test gated alongside private. Refs #1816. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/code_changes.yaml | 7 +++++++ .github/workflows/pr_code_changes.yaml | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/.github/workflows/code_changes.yaml b/.github/workflows/code_changes.yaml index 6a0008f2e..6cef7d63e 100644 --- a/.github/workflows/code_changes.yaml +++ b/.github/workflows/code_changes.yaml @@ -54,6 +54,13 @@ jobs: # Only export token if non-empty (defensive against missing secrets) if [ -n "$HF_TOKEN_VALUE" ]; then export HUGGING_FACE_TOKEN="$HF_TOKEN_VALUE" + # policyengine-core only passes a token to hf_hub_download when + # the repo reports private=True. policyengine-uk-data-private + # has been public + gated since 31 July 2026, so core sends the + # request anonymously and the gate returns 401. HF_TOKEN is + # huggingface_hub's implicit-token variable, used when the + # explicit token is None, which restores authentication. + export HF_TOKEN="$HF_TOKEN_VALUE" fi make test env: diff --git a/.github/workflows/pr_code_changes.yaml b/.github/workflows/pr_code_changes.yaml index 17c973b5e..769ff3501 100644 --- a/.github/workflows/pr_code_changes.yaml +++ b/.github/workflows/pr_code_changes.yaml @@ -73,6 +73,13 @@ jobs: # Only export token if non-empty (avoids 'Bearer ' error for Dependabot PRs) if [ -n "$HF_TOKEN_VALUE" ]; then export HUGGING_FACE_TOKEN="$HF_TOKEN_VALUE" + # policyengine-core only passes a token to hf_hub_download when + # the repo reports private=True. policyengine-uk-data-private + # has been public + gated since 31 July 2026, so core sends the + # request anonymously and the gate returns 401. HF_TOKEN is + # huggingface_hub's implicit-token variable, used when the + # explicit token is None, which restores authentication. + export HF_TOKEN="$HF_TOKEN_VALUE" fi make test env: