Port UK compact source stages and 2024-25 refresh from uk-data

[MaxGhenis]

Parent: #144.

Why

Populace UK can replace the local-geography/matrix path, and the current compact H5 already carries several recent uk-data columns. But if policyengine-uk-data is archived, Populace needs to own the source-stage logic and provenance for those columns instead of relying on uk-data build history.

Scope migrated from uk-data

Open PRs:

• Impute mortgage and consumer debt from the Wealth and Assets Survey policyengine-uk-data#426: mortgage_debt and consumer_debt from WAS for net_wealth.
• Add PIP points from disability categories policyengine-uk-data#408: pip_dl_points / pip_m_points from PIP categories once model variables exist.

Recent merged PRs that need Populace-owned source-stage equivalents:

• Regenerated ledger feeds fail the period contract: year-filtered bundles drop the CBO aging pivot and prior-year SOI tables #410/Add release-gate preflight: catch statically-knowable gate failures without a solve #436: FRS 2024-25 ingestion plus household-ID sort invariant for raw compact builds.
• Extend export-mass parity exclusions for the Build M frame #433: employment_sector, sic_industry_division, and public-sector target source.
• Measure household_weight from typed frame weights in the parity universe #428/Recognize series-identity ledger qualifiers as inert in the materializer guard #429/Gate the SSI swap delta at a runaway cap, not assignment granularity #431: bus_fare_spending, bus subsidy scaling, and model-version floor.
• Let voluntary filing predict from PUF-only survivor units #425: seeded stochastic draws for property purchase, BRMA, and capital-gains amounts.
• Restore ASEC ED_VAL via a pinned sidecar and close the source-carry class #421: SPI synthetic prior weights, household_is_spi_synthetic, household_is_capital_gains_clone, and calibration source-share diagnostics.
• Checkpoint and cache identity omit the staged frame: reform-vector cache ignores seed, constraint matrix unfingerprinted, manual materializer bumps #380/SYSTEM_REQUIREMENTS.md memory model is stale: attributes the 29 GB calibration peak to a dense-row vstack the compiler no longer does #383/Should *_would_be_qualified columns be derivations rather than QRF targets? #416/Child-support expense prevalence: survey-faithful 0.47% vs retired eCPS's drifted 2-3% vs OCSE-implied ~1.5-2% #417/SNAP state take-up: fill rates use pre-mass-repair weights and the export drops the eligibility surface — calibrated caseloads don't reproduce from the artifact #419: disability, PIP category, child/QYP, and transfer-side PIP component fixes.
• Assign SNAP take-up state-conditionally: reported receipt + national rate bakes in CPS state underreporting #372/Release-native loading: certificate-verified datasets by default; immutable-first publication #385/ACS-multispine staging: SSI probe finds spine-inconsistent incidence (2.6% vs 1.9% baseline rate) and weight-composition skew (ASEC 1.5 persons/hh, ESS 0.3%) #403-Fire the release Slack alert from publish_release; warn when the webhook is unset #407/Published release exports can no longer serve as --base-h5: QBI input reconciliation requires base-only PUF columns #412-Fix SNAP state take-up crash: Frame.broadcast only targets person #414: income/SPI/fuel/consumption prerequisite updates and road-fuel volume uprating.

Acceptance criteria

• Populace has UK source-stage modules/manifests for these compact-data inputs, with no policyengine_uk_data imports.
• The compact artifact can be rebuilt for at least the current Populace UK release year and the FRS 2024-25 refresh path.
• Export-variable gates prove the compact H5 includes the same PolicyEngine-UK variables expected by policyengine.py.
• Determinism tests cover every stochastic assignment on the live build path.
• Source flags and diagnostics make synthetic SPI/capital-gains support inspectable without row-order inference.

---

2026-08-12 update — this issue is workstream E of the UK-migration master epic #665 (it absorbs the source-stage track of the now-closed #144).

Full item list to port, each as a declarative source stage + gate (children filed as picked up); order constraints: wealth before consumption (num_vehicles), income before capital gains.

• Full FRS table ingest (incumbent reads ~14 tables; microcosm reads 2): entities/IDs, grossing weights, demographics, council-tax imputation, LHA/BRMA, education, legacy-benefit proxies, disability categories
• Stochastic inputs: 11 take-up parameter files with reported-receipt anchoring, TV licence/FTB draws, seeded RNG discipline, household_is_* provenance flags as first-class metadata
• WAS wealth (11 outputs incl. num_vehicles, student-loan balance allocation) — note the owned_land release-to-release instability recorded in Scottish/Welsh council tax ~20% above devolved budget figures; WAS owned_land imputation unstable across releases policyengine-uk-data#448
• LCFS consumption (18 outputs) + energy raking to NEED targets
• ETB VAT; ETB services + NHS usage
• Capital gains incl. top tail (Impute capital gains amounts from the published joint distribution #560 in flight; UK capital gains: imputed distribution has no top tail, missing 55% of CGT #552/UK CGT stage: HMRC outturn level anchors + Advani-Summers shape, with accrued/realized/taxable name separation #364/UK: banded CGT gains targets (by size of gain) to constrain the distribution, not just count + total #467)
• Salary sacrifice; student loan plans (SLC stocks); childcare take-up fitting; regional property uprating
• From Migrate remaining policyengine-uk-data backlog before archive #144: debt, bus fares (Add UK bus-spending imputation plan and DfT calibration targets #153), employer sector/SIC, PIP/disability, child/QYP
• New stages requested against the incumbent that land here instead: UC deduction attributes (Impute UC deduction attributes (latent rate, type combination) at dataset build policyengine-uk-data#450)
• Known incumbent defects the ports must not replicate: UC caseload ~40% below administrative counts (UC caseload runs ~40% below administrative counts, capping aggregate reform estimates policyengine-uk-data#452), Scottish/Welsh council tax 20–30% above devolved budgets (Scottish/Welsh council tax ~20% above devolved budget figures; WAS owned_land imputation unstable across releases policyengine-uk-data#448)
• FRS 2024-25 refresh (incumbent is already there; calibration year 2025)

Acceptance: the microcosm-built spine replaces the certified incumbent H5 in the national build; eFRS parity fixtures beat or match the incumbent on the frozen register with signed differences.

[MaxGhenis]

Three portable lessons from policyengine-uk-v2#133 for the SPI source stage this issue tracks:

1. Region map: uk-v2 found and fixed a GORCODE bug (code 3 skipped → Yorkshire misassigned; corrected to the contiguous 1–12 map). The populace spi_income stage spec should adopt the corrected map from day one and carry a test pinning all 12 regions — the bug lived only in one of the org's two SPI extractors, which is the cost of parallel extraction.
2. Tape-year naming convention: uk-v2's year folders mix start-year and end-year naming (spi/2021 = end-year, spi/2022 = start-year). The populace stage should declare ONE convention explicitly in the spec (recommend labeling by tax year, e.g. 2022_23, which is unambiguous).
3. Multi-year tapes + uprating: uk-v2 carries 2020 and 2022 tapes with explicit uprating to 2025–27. The populace stage spec should plan for multiple vintages with declared uprating stages rather than a single hardcoded tape.

Also worth adopting repo-wide: uk-v2 runs a "Data pipeline dry run" CI job on PRs — a cheap wiring check populace's CI (tests-only) doesn't have; a build-plan dry-run job would catch stage-wiring breaks pre-merge.