Send HF_TOKEN so gated dataset downloads authenticate

[vahid-ahmadi]

Fixes the CI breakage tracked in #1816. Every dataset-backed job on main and on PRs has failed with 401 since 31 July 2026.

The cause is not the secret — no token is being sent at all

policyengine-core's download_huggingface_dataset passes a token to hf_hub_download only when the repo reports private=True (tools/hugging_face.py:81-90):

fetched_model_info = model_info(repo)
is_repo_private = fetched_model_info.private   # False since 31 July
authentication_token = None
if is_repo_private:
    authentication_token = get_or_prompt_hf_token()
hf_hub_download(..., token=authentication_token)   # token=None

policyengine-uk-data-private has been public + gated (manual approval) since 31 July, confirmed against the live Hub:

private: False
gated  : manual

So private is False, core passes token=None, the download goes out anonymously, and the gate returns 401 GatedRepoError. private is simply the wrong predicate once a repo is gated rather than private.

This explains what looked contradictory in #1816: replacing the HUGGING_FACE_TOKEN secret on 10 August changed nothing because the credential was never the variable, and the out-of-band validation of that token legitimately passed because it supplied the token explicitly — precisely the step core skips.

The fix

huggingface_hub falls back to the HF_TOKEN environment variable when the explicit token is None (get_token_to_send → get_token), so exporting the same secret under that name restores authentication. Two lines per workflow, no new secret.

Verification

Applied to #1815, where the previously-401ing gated dataset now downloads and the full suite passes:

1116 passed                                                   (policy YAML)
177 passed, 1 skipped                                         (unit + microsimulation)
test_uc_deductions_aggregates.py::test_deduction_statistics_match_dwp  PASSED
Test: success   Lint: success   all 6 smoke-imports: success

Split out of #1815 so it can land on main independently — every dataset-backed PR in the repo stays red until it does.

Follow-up

This is a workaround. The durable fix belongs in policyengine-core: pass the token whenever one is available, or test gated alongside private. Any country package downloading from a gated HF repo has the same latent bug. Filed as PolicyEngine/policyengine-core#529.

No changelog.d fragment: CI-only change with no user-facing model behaviour, and adding one would trigger a package release.

🤖 Generated with Claude Code