UK dense/local epic: rowwise constituency-grain dataset by cloning the national staging pool (US dense-arm analog)

[MaxGhenis]

Max greenlit scoping the "bigger UK dataset" arm 2026-07-22. This epic is the UK analog of the US dense/local arm: one national rowwise artifact whose rows carry finest-available geography and whose weights are calibrated against a constituency/local-authority target surface — built by cloning the compact FRS-based pool, because the UK has no ACS-scale microdata to substitute for density. It subsumes the publication goal of #146 (rowwise H5 + long weights), sits under the #144 migration umbrella, and is the sibling of #355 (the pool should get a smaller national default at the same time as this arm gets bigger).

Everything below is designed against the machinery that already exists (uk_runtime/rowwise_geography.py, rowwise_dataset.py, geography_ladder.py, local_targets.py, local_solver.py, local_runner.py, tools/build_uk_rowwise_dataset.py); the epic is mostly wiring, fences, and targets, not new solvers.

Base artifact and pipeline order (fence-determined, not a preference)

The clone input is the national staging H5 — the output of the national orchestration seam after the 145/0 input-coverage gate and the adjudicated HMRC/SPI replay — not the raw certified compact:

	certified compact (f17306cc…)	national staging (829e843f…)
households	535,080 (200,000 zero-weight dead SPI)	535,080 (rebuilt SPI channel, positive mass)
persons	1,157,100	1,157,739 (724,960 FRS + 432,779 SPI)
benunits	618,980	618,884
household mass	28,840,551.182	28,840,551.182 (conserved; 1 mass-change record)
weight kind	(unlabeled)	importance + mass log in H5 attrs
bytes	1,315,880,118	1,532,379,785 (local_untracked)

(Compact figures verified directly against the cached certified H5; staging figures from the committed national_staging_build_record.json.)

The ordering is forced by the fences in hmrc_restoration.py: the HMRC replay byte-binds its input to the certified candidate (_assert_dataset_matches_certified_candidate checks the source-file fingerprint captured at SHA-256 verification), so cloning can never run upstream of the seam, and cloning the certified compact directly would clone 200k dead SPI households and skip the 145/0 restoration. Pipeline:

certified compact (byte-verified)
  → national seam (retained FRS leaves → HMRC/SPI replay, IMPORTANCE weights, mass log)
  → staging H5  ← the 145/0 gate passes here
  → rowwise clone + geography ladder assignment      (this epic)
  → one uniform local+national calibration solve      (this epic)
  → rowwise release, private HF only                  (this epic)

Clone count K: rows, memory, support

The pool is already a 10× clone: household_id = tier·10⁸ + base with exactly 53,508 unique base ids × 10 tiers (verified on the certified H5; #355 documents the construction). eFRS-grain source lineage is therefore recoverable as household_id mod 10⁸, and the rowwise build must carry it — per-area "distinct source households" diagnostics computed at pool grain would overstate real support 10×.

K below means the additional rowwise multiplier on top of that 10×. Clones add geographic resolution and solver degrees of freedom only — zero new information — so K is sized by per-area support, not ambition:

K	households	persons	H5 (≈, linear)	mean rows/constituency (650)	smallest constituency (≈)
1	535,080	1,157,739	~1.5 GB	823	~240
2	1,070,160	2,315,478	~3.1 GB	1,646	~480
4	2,140,320	4,630,956	~6.1 GB	3,293	~960
10	5,350,800	11,577,390	~15 GB	8,232	~2,400

The smallest-constituency column is the population-proportional expectation for a Na h-Eileanan an Iar-scale area and is approximate; the dry-run tool (increment 2) computes the exact expected support per constituency from the ladder/crosswalk populations and the pool's region mix. For calibration comparison: the US dense arm supports 436 CDs on a 337,704-record pool ≈ 775 records/CD (#355, #449) — K=1 is already at US-dense per-area scale, because the 10× pool cloning did the density work up front.

Proposal: pilot at K=1 (geography assignment only, no row growth), first published candidate at K=2–4, chosen from the dry-run support/ESS diagnostics. K>4 requires a measured justification. avoid_constituency_collisions stays on (no source household repeated in the same constituency across rowwise clones).

Geography assignment: the OA ladder is the operator

Two mechanisms exist. The crosswalk path (#142/#143) samples an OA within (country, region) ∝ population and covers all four countries. The OA ladder (#349 ratified, #354 merged) samples a 2024 Westminster constituency within the region ∝ census household counts, then an OA within the constituency ∝ population, carries sha-pinned best-fit lookups and per-layer vintages (vintage_policy: error), and ships a release-blocking gate — but is England & Wales only today (575 of 650 constituencies).

The epic declares the ladder as the assignment operator (it is the ratified US-block-ladder parallel with vintage discipline). Work items:

• extend the ladder artifact to Scotland (2022 census OAs, NRS sources) and Northern Ireland (Data Zones 2021, NISRA) — geography_sources.py already pins those source URLs for the crosswalk path; they need per-layer vintage pinning in oa_ladder_sources.py;
• until S/NI land, a full-UK rowwise build through the ladder fails loudly on the missing regions (already the ladder's contract — no silent partial coverage);
• rewire tools/build_uk_rowwise_dataset.py from the crosswalk sampler to assign_uk_geography_ladder + uk_geography_ladder_gate, keeping the crosswalk as the coverage-check path.

Local target surface: census first, then bind

local_targets.py computes today, at household grain:

• constituency: HMRC employment/self-employment amount+count (SPI frame via income_tax > 0), age bands (8 × 10y), UC households, UC households by child count (0/1/2/3+);
• local authority: the HMRC/age/UC families plus ONS equivalised net income BHC/AHC, housing costs, tenure (4 classes), private rent.

That mirrors the deprecated uk-data constituency surface. What official UK local data offers beyond it (ONS mid-year age structure by PCON24/LAD; HMRC Personal Incomes by constituency — taxpayer counts and employment/self-employment/pension totals; DWP Stat-Xplore UC/Pension Credit/Housing Benefit by constituency and LA; Census 2021/2022 tenure and composition; ONS PIPR rents by LA; ONS small-area income at MSOA) needs a census artifact, not prose: increment 1 ships a tool that enumerates every in-code metric family and every documented official family with area type, source, vintage, and status (bound_in_code / documented_unpinned / needs_adjudication), checked in and tested, so binding work is enumerable and claimable per family.

Two standing fences on targets:

• the 208 banded HMRC facts stay fenced (see below) — the constituency HMRC family is a different, unbanded surface and gets its own admissibility adjudication before it binds;
• the income_tax > 0 SPI-frame membership in the HMRC metrics is a modeled proxy frame; the census must record that honestly and the binding adjudication must accept or replace it explicitly.

The local solve: one uniform operator, declared bounds (doctrine)

Per the #492 lane and #493: no per-target calibration knobs. The solve is one operator over the whole surface — national families (149 targets in the published release, plus the natively declared CGT facts) and all area×metric rows together — with globally declared loss shape, cap, and weight-ratio bounds. A miss is local-support work (rows/clones/ladder) or target work (fix or fence the target), never a knob.

• Binding: the rowwise artifact solves one weight vector of length 535,080·K where each row supports only its assigned constituency's rows (the US Build-N CD shape). The stacked all-areas×all-households machinery (build_stacked_local_matrix, Publish a Populace UK local variant with rowwise H5 and long weights #146 long weights) remains the research harness and can express the rowwise solve as its block-diagonal special case; the released artifact is the rowwise H5 with its solved household_weight column.
• Declared bounds to adjudicate once: solve_stacked_local_weights currently defaults max_weight_ratio=100.0 while the US dense design bound is 5× (max_weight_ratio anchors differently per arm: 5x vs design (dense) but ~25x effective (sparse refit re-anchors) — declare one stretch contract #493 wants one declared stretch contract). The epic picks the UK local bound explicitly and records it in the release manifest.
• Diagnostics as contract: the Loss shape: flat cap at 1.0 makes past-cap rows free dumping grounds (17 rows pushed out in Build N); bounded-tail experiment + past-cap census diagnostic #492 past-cap census (init-past / final-past / escaped / pushed-out) ports to the local solve as a standard release artifact, alongside per-area support (min/median rows, ESS, distinct eFRS-grain sources) and the per-family fit table.

HMRC IMPORTANCE-weight fences under cloning

Read against hmrc_restoration.py, cloning survives the fences honestly if and only if the metadata rides along:

1. Replay-before-clone is structural — the replay's byte-binding to the certified candidate means the clone consumes the seam's output; the fence cannot be reordered away.
2. Mass conservation is exact under cloning: weights divide by K, every row duplicates, so total mass and every column's effective-mass share are invariant by construction. The 145/0 gate still re-runs on the rowwise output (fail-closed, never assumed), and the SPI channel's reviewed 50% mass share is re-checked post-solve as a declared bound.
3. Live gap (increment 2 fixes): rowwise_dataset.py reads and writes only the four entity tables — it drops the staging H5's populace_household_weight_kind (importance) and populace_mass_log_json attrs. Cloning the staging H5 through today's tool silently launders IMPORTANCE weights into unlabeled weights. The clone step must carry both attrs, refuse an input whose weight kind it does not recognize, and assert exact mass conservation against the input.
4. No calibration against the 208 banded facts, before or after cloning: complete FRS Total Income band assignment remains unavailable, so those facts remain reviewed exclusions with canonical fences in every rowwise release exactly as in the national seam. Nothing in this epic touches IMPORTANCE-weight replay internals, the reviewed release parameters, or the source contracts.
5. The local solve transitions weight kind explicitly (importance → locally calibrated) as a recorded step in the mass log, never implicitly.

Release contract for the rowwise artifact

The 145/0 contract covers input columns and carries over unchanged (the gate re-runs on the rowwise output). The rowwise artifact additionally promises, in its manifest:

1. identity chain: staging H5 sha → certified candidate sha → source-stage shas (donor/ODS identities by reference to the committed replay artifacts, never row-level data);
2. geography: ladder artifact sha + per-layer vintages, 650/650 (or the declared E&W 575 subset) constituencies assigned, zero missing-geography rows, ladder gate pass;
3. weights: exact mass conservation vs staging, declared weight-kind chain (importance → locally calibrated), carried mass log, declared stretch bound;
4. solve: uniform-operator config (loss shape, cap, ratio bound, epochs, seed), per-family fit, past-cap census, per-area support with eFRS-grain distinct-source counts;
5. role: dataset_role: non_default_local_area per Release contract needs dataset-role classes: national-default checks reject valid non-default local-area artifacts #398 — never the latest.json pointer (--no-latest), immutable tag, the compact (and eventually the UK small national default: L0-selected ~50k-household file from the 535k pool #355 small default) stays the national default. Until Release contract needs dataset-role classes: national-default checks reject valid non-default local-area artifacts #398's role-class validator lands, publication uses the Build-L routed path (all applicable gates recorded).

Publish mechanics (licensing)

FRS-derived microdata must never reach a public repo. The rowwise H5 and every sidecar that carries household-grain rows publish exclusively to the private HF dataset repo policyengine/populace-uk-private (the certified compact's home), via populace-publish-release --repo-id policyengine/populace-uk-private --no-latest, tagged populace-uk-2023-rowwise-<git>-<ts>. Aggregate diagnostics (manifest, fit tables, support summaries, past-cap census) are committable to the repo exactly like the HMRC replay artifacts — no row-level donor or FRS data, no local paths.

Increment ladder

1. Local-target census tool — tools/census_uk_local_targets.py + checked-in census JSON + tests. Pure observability. (claimed, this lane)
2. Rowwise fence carriage + dry-run — carry weight-kind/mass-log attrs through clone read/write, refuse unrecognized weight kinds, assert exact mass conservation in the driver manifest, and add --dry-run: the full clone plan (rows/bytes at K, exact expected per-constituency support from crosswalk populations × the pool's region mix, id-multiplier and lineage checks) with no H5 written. (claimed, this lane)
3. Ladder S/NI extension — pin NRS/NISRA sources with per-layer vintages; full-UK ladder artifact.
4. Solve doctrine hardening — single-operator config surface, declared stretch bound (max_weight_ratio anchors differently per arm: 5x vs design (dense) but ~25x effective (sparse refit re-anchors) — declare one stretch contract #493 alignment), past-cap census port (Loss shape: flat cap at 1.0 makes past-cap rows free dumping grounds (17 rows pushed out in Build N); bounded-tail experiment + past-cap census diagnostic #492), structural refusal of per-target knobs.
5. K=1 pilot build on the real staging H5 (licensed machine): geography + diagnostics, no solve; adjudicate K from the measured support.
6. First calibrated rowwise candidate (K per adjudication): bound target families only, full contract above, cross-family review, private publish.
7. Loader/consumer wiring (policyengine.py) — out of scope here; coordinate on the campaign index.

Refs: #144, #146, #355, #349, #354 (ladder), #420 (145/0 contract), #398 (dataset roles), #492/#493 (solve doctrine), #275 (one national dataset, filter by geography), #142/#143 (crosswalk + driver), UK_COVERAGE_PROGRESS.md (fence history).

[MaxGhenis]

Claiming this epic (UK-dense scoping lane, greenlit 2026-07-22). Starting with the two unambiguous increments now, one PR each:

1. Local-target census tool — tools/census_uk_local_targets.py + checked-in census artifact + tests (increment 1).
2. Rowwise fence carriage + dry-run — weight-kind/mass-log carriage through the clone read/write path, exact mass-conservation assertion, and a --dry-run clone plan with expected per-constituency support (increment 2).

Increments 3–6 (ladder S/NI, solve doctrine hardening, K=1 pilot, first calibrated candidate) stay open for claiming after design review of this issue. Coordination via comments here; campaign index cross-post on #449.

[MaxGhenis]

Increment 1 merged: #498 (local-target census). The census landed with the cross-family review hardening: fail-closed exact/prefix family matching, per-area ordered-metric drift pinning, canonical-JSON currency comparison, register isolation, and six reviewed binding fences — the review pass minted three new ones (ons_bhc_ahc_noncomparable: ons/equiv_housing_costs has no declared supplier; uc_unit_vs_household_grain: runtime benefit-unit counts vs DWP claim-household statistics; population_universe_private_households: FRS private households vs ONS usual residents). Those three are now standing inputs to the binding adjudications in increments 4/6.

Increment 2 (rowwise fence carriage + dry-run) is implemented and in final verification; PR to follow.

[MaxGhenis]

Increment 2 PR up: #501 — rowwise fence carriage (weight-kind/mass-log through the clone, closing the IMPORTANCE-laundering gap named in the epic), exact mass-conservation record with the declared 1e-9 bound, --source-lineage-modulus (the verified 10^8 pool-tier structure), and --dry-run with the sampler's exact expected per-area support. Sol cross-family review in progress; merge after adjudication + green CI.

[MaxGhenis]

Increment 2 merged: #501 (rowwise fence carriage + dry-run). Landed after a substantive cross-family review round — all five sol blockers confirmed and fixed, three verified against the pinned staging artifact. Design facts the epic should absorb (issue body remains authoritative for everything else):

• The staging H5 already carries immediate-layer lineage: source_household_id with 335,080 distinct FRS-channel parents plus household_support_channel. Pool-grain lineage is therefore a separate derived layer (pool_source_household_id, --source-lineage-modulus 100000000), and on the staging pool it resolves 33,508 FRS bases — rebuilt-SPI households sit outside the tier scheme, so pool-lineage diagnostics are per-channel. The increment-5 pilot should adjudicate per-channel lineage semantics on the real artifact.
• Collision avoidance materially reshapes small-group support: the collision-free expectation diverges hard when n_clones approaches a group's sampleable constituency count, so the dry-run's authoritative number is the realized assignment (real sampler at the build seed — identical draws to the real build). K-sizing in increment 5 should read realized_support, not the analytic block.
• The clone now refuses stale mass chains, zero-mass pools, and unknown weight kinds; writes are atomic; the mass log gains one conserving record per clone under the declared 1e-9 bound.

Increments 3–6 (ladder S/NI extension, solve doctrine hardening, K=1 pilot on the real staging H5, first calibrated rowwise candidate) remain open for claiming.

[MaxGhenis]

Continuing the lane: claiming increments 3 (ladder S/NI extension), 4 (solve doctrine hardening), and 5 (K pilot on the real staging H5). Order of attack: 5's measurement half first (dry-run K sweep on the real staging artifact with the merged #501 tooling — realized support per constituency/LA, lineage layers, weight chain — results posted here for the K adjudication), then the increment-4 doctrine PR, then increment 3.

[MaxGhenis]

Increment 5a: K-sweep pilot on the real staging artifact

Ran against the byte-verified staging H5 (sha 829e843f…, re-hashed before use) with the #501 tooling; crosswalk built fresh from the official sources. Dry-run plans at K=1/2/4 plus a full K=1 build.

Realized per-area support (real sampler at seed 42, zeros included)

K	households	persons	est. size	constituency min / median / max	LA min / median	bottom constituencies
1	535,080	1,157,739	≥1.5 GB	249 / 727 / 3,868	18 / 1,099	S14000027 (249), S14000051 (416), E14001304 (448)
2	1,070,160	2,315,478	≥3.1 GB	551 / 1,454 / 7,564	35 / 2,178	same three, ~2×
4	2,140,320	4,630,956	≥6.1 GB	1,078 / 2,908 / 14,868	66 / 4,364	same three, ~4×

650/650 constituencies and 361 LAs covered at every K; zero missing-geography rows. Collision-free vs realized minima differ by <5% at these K (groups have far more constituencies than clones), validating both blocks. The thin tail is exactly the expected protected/island class (S14000027 = Na h-Eileanan an Iar-scale, S14000051 Orkney-scale) plus the micro-LAs: Isles of Scilly 18 rows and City of London 32 at K=1 — LA-grain binding needs either K≥4 (Scilly 66) or an explicit micro-LA support adjudication.

Full K=1 build (the pilot artifact, local-untracked)

1.54 GB rowwise H5, sha 50c674f9…. The #501 fence chain held end-to-end on the real artifact: weight kind importance carried, 2 mass-log records (reviewed SPI allocation + clone record), conservation abs delta exactly 0.0, and the output loads under load_uk_national_dataset.

Lineage measurements (design fact for increment 6)

Pool lineage (mod 10⁸): 33,508 distinct pool sources; by channel: frs 33,508, spi 33,506 — the rebuilt SPI channel's ids resolve into the FRS base space because the seam samples its replacement households from the FRS base per stratum (spi_support._sample_replacement_household_ids, read this session). So pool lineage = eFRS pool-slot ancestry on both channels, which is stronger than the #501 caution suggested. Immediate layer: 335,080 distinct, as measured before. Pool copies per source: min 10 / median 16 / max 20.

K adjudication proposal

• Constituency-grain-only candidate: K=2 (min 551 rows/constituency, ~3.1 GB) — comfortably above the US dense arm's ~775 mean at the minimum, at half the memory of K=4.
• If LA families bind: K=4 plus a reviewed micro-LA decision (Scilly/City of London).
• K=1 remains the pilot/diagnostics grain.

Objections welcome here; increment 6 will treat K=2 constituency-grain as the working default absent adjudication otherwise.

[MaxGhenis]

Increment 3 scoping: ladder Scotland/NI extension — the gap register

Measured against the pinned sources this session, the full-UK ladder is closer than the epic assumed:

Already solved UK-wide (no new work):

• ITL: the pinned LAD (April 2023) → LAU1 → ITL3/2/1 lookup covers all four countries — verified from the cached file: 41 S12 Scottish councils and 11 N09 NI LGDs are present alongside E/W. The ladder's ITL join generalizes as-is.
• Scotland base layers: geography_sources.py already pins OA2022 → DZ2022 → IZ2022, OA → council/LAU, OA → PCON24, and OA population (the crosswalk path builds them today).
• NI base layers: DZ2021 hierarchy + population pinned; constituency via the reviewed postcode-based inference.

The four genuinely missing sources (the whole remaining work):

1. Scotland OA22 household counts (stage-one constituency-draw weight): Scotland's Census 2022 publishes ~24 tables at OA grain (~50 households/OA, 46,363 OAs) — the household-count table id needs pinning from the census data downloads (scotlandscensus.gov.uk; OA product register on spatialdata.gov.scot).
2. Scotland OA22 → electoral ward best-fit (NRS geography lookups).
3. NI DZ21 household counts: NISRA MS-E01 households table (Census 2021 person and household estimates for Data Zones) — publication verified; exact file + sha to pin.
4. NI DZ21 → ward/DEA: NISRA geography support lookups (DEA 2014 grain — the tables above already publish at DEA, so the lookup exists).

Mechanics: join_uk_oa_ladder_layers/assemble_uk_oa_ladder are country-agnostic on their inputs; the extension is (a) a per-country layer-bundle assembly that concatenates E&W + S + NI OA frames with per-country, per-layer vintages in the artifact metadata (vintage_policy: error extended to record e.g. ward: {ew: 2022, s: <vintage>, ni: DEA2014}), (b) the four loaders above with sha-pinned identities, and (c) the ladder gate's London-mass floor gaining S/NI analogues (Central Belt / Belfast mass floors). Until all four land, the ladder keeps refusing S/NI regions by name — no silent partial coverage.

Remains claimed by this lane; if it hands off, this register is the complete work list.