UK parity increment 4: weighted integrity gates (input-mass parity + QRF tail concentration) in the terminal battery (#609)

[juaristi22]

Closes #609. Increment 4 of the #578 parity plan: the UK terminal battery gains the two US integrity gates that were each purchased with a named incident — input-mass parity (#278) and QRF tail concentration (#462) — under the attested, batched, fail-closed report from increment 1 (#588).

What landed (deliverables 1–5 of #609)

Shared helper promoted, not forked (deliverable 1). us_input_mass_totals moved to populace.build.input_mass.input_mass_totals; the us_runtime path re-exports it, so US call sites are byte-identical. A UK analog uk_dataset_input_mass_totals handles the national person/benunit/household table layout (household weights broadcast through membership; names namespaced entity.column per the degenerate-gate convention, since national tables don't enforce globally unique column names).

uk_input_mass_parity_gate (deliverable 1). Wraps the shared gate verbatim — zero candidate mass fails at any tolerance, candidate-only columns are reported and never fail, near-zero reference columns are skipped, unused entries are reported — and adds the universal exclusion discipline the shared gate lacks: a stale exclusion (column now within tolerance) fails, via a per-column probe that reuses the shared gate's exact semantics; below-floor and absent-from-reference entries are dormant and reported. The frozen reference's filename, revision, sha256, and vintage are required by UKInputMassReference, recorded in the gate details, and bound into the evidence digest.

uk_qrf_tail_concentration_gate (deliverable 2). Surface derived from the fit_weighted_qrf_stage* operations' declared outputs in hmrc_income_source_stages.json via a new uk_hmrc_weighted_qrf_output_columns() in the source-contract module (47 columns today; a new manifest-declared QRF output is covered the day it's declared). Per the issue, no sparsity filter: every declared, present, numeric output is checked; min_nonzero_records is the sole thinness guard. Stale/dormant discipline comes from the shared gate.

Wiring (deliverable 3). Both gates follow the increment-1 optional-evidence pattern: a path that supplies no reference/thresholds omits them (never a placeholder pass), and an armed gate missing either fails closed with a named line. Policies are sealed into policy_sha256 (_terminal_gate_policy_payload gained input_mass_parity / qrf_tail_concentration slots); two new evidence_sha256 families follow the existing precedents (input-mass hashes the frozen reference identity+totals; QRF tail projects the gate's recorded observables, like release_dataset does). Report schema 2→3, attestation 4→5, membership validation extended. Evidence artifacts still land before the batched raise — a red run leaves the signed diagnosis on disk (verified). No second raise added.

Registers (deliverable 4). Committed empty registers (uk/input_mass_reviewed_exclusions.json, uk/qrf_tail_reviewed_exclusions.json) declared in the country package, loaded by default by the new --input-mass-* / --qrf-tail-* flags on tools/build_uk_national_dataset.py (complete-arming enforced: partial threshold sets are a CLI error). Mandatory non-empty reason, dormant reported, stale fails.

Mutation tests (deliverable 5). A zeroed column, a 99.9%-loss column, and a concentrated column each fail by name through the armed battery; a crash inside either evaluator appends a named failure without masking pending gates (#547); dropping an evidenced gate fails attested-membership validation and a committed membership test; stale exclusions fail in both gates; thresholds provably move policy_sha256; legacy no-evidence paths keep the exact five-gate membership.

Publication contract in lockstep. populace-data/contract.py pins moved with the producer (schema 3, attestation 5, new policy digest, evidence-stage→gate map, detail schemas, stale-exclusion and reference-identity observables). The pinned policy digest is the certified default policy, in which the increment-4 slots are unarmed — arming a published release requires a reviewed pin move alongside the committed threshold constants, which is exactly the "a threshold outside the policy hash is not attested" rule.

Deliverable 0 (measurement pass) — tooling shipped, numbers deferred

No licensed UK artifact exists on this machine, so the measurement pass itself could not run here. What shipped so it can run anywhere with gated data:

• tools/measure_uk_weighted_integrity_baselines.py — weighted per-column totals plus top-{10,100,500,1000} concentration and carrier counts for any national H5 (certified compact, staging), including a ready-to-fill reference-JSON template.
• tools/build_uk_efrs_parity_reference.py --emit-weighted-totals PATH — the pinned, sha-verified eFRS incumbent's totals in the exact schema the gate consumes. It refuses to write inside the repository.

Licensing: resolved, and the recorder enforces it

The EUL question is answered from the licence text rather than left open. End User Licence CD137 v16.00 (25 February 2026) clause 8 requires "adhere[nce] to the statistical disclosure control standards as detailed in the Research data handling and security guide for users in any outputs I produce and publish" — publishing outputs is permitted, conditioned on SDC. Clause 4's bar on sharing "any Dataset(s) derived from the Data Collection(s)" governs record-level derived data (clause 13's deposit duty reads the same way), not summary statistics. Clauses 11–12 require acknowledgement and citation in any publication.

CD171-ResearchDataHandling v17.00 §5.2.1 supplies the rules: minimum threshold of 3 (10 advised against secondary disclosure, 30 for some studies), and "any output that refers to unit records, e.g. a maximum or minimum value, must be avoided".

So population-level weighted per-column totals are publishable with citation. But the first version of the recorder emitted a per-column max_abs_value and would have reported a top-k share for a column with a handful of carriers — and tail concentration is the (n, k) dominance statistic, so at small k the share times the total approximates the mean of those few records. Fixed in fdbcbb0: no maxima or minima, thin columns suppressed below --sdc-minimum-count (default 10), --top-k narrower than that count refused, and the applied rules plus citation obligations recorded in the payload. A genuinely concentrated column above the threshold stays fully visible, so the #462 signature remains detectable.

Still the caller's to confirm: the SPI PUT (SN 9422) carries its own SPI usage agreement as a Special Condition under clause 3, with HMRC's statutory taxes-confidentiality duties behind it. Raise --sdc-minimum-count to 30 if it or the FRS terms require it.

Measurement pass: run, and it is why the gates ship unarmed

The pass ran against the pinned incumbent (enhanced_frs_2023_24.h5, sha 584ae33d… verified before reading) and the certified compact (populace_uk_2023.h5, sha f17306cc… verified). Four results, all recorded beside the policy dataclasses so no later reader reintroduces the US constants:

1. The US 0.75 top-share threshold is wrong for this surface. At the US top_k=100 / min_nonzero_records=500 settings the incumbent exceeds 0.75 on 16 of 28 checked columns. UK reported-benefit columns sit on small, high-intensity subpopulations, so concentrated weighted mass is their normal state.
2. No single global top-share threshold is both incumbent-compatible and incident-catching. The lowest no-headroom value on a broad surface is ~0.996, which would not have caught Build M live default: non_sch_d_capital_gains lands at $74.6B vs its $10.2B SOI target (+635%); Schedule-D CGD route is empty #462 itself (89% of mass in 100 of 2,295 carriers). Tightening to a discriminating ~0.54 requires min_nonzero_records=5000, dropping 17 of 31 columns.
3. The mass floor is adjudicated at 0.0. The US 1e9 would stop checking 50 of 131 columns, including person.gift_aid (£996M) and person.charitable_investment_gifts — the two release_input_coverage_manifest.json names as distributional-required. The surface mixes units (currency into the trillions, weighted person counts, flag counts), so a magnitude floor is not a coherent materiality filter; 0.0 still skips the three exact-zero reference columns because the shared gate compares <=.
4. The certified compact is not a valid candidate against this reference, confirming the issue's reading of option 2. It carries 8.1% less household mass (28,840,551 vs 31,389,678), a different surface (150 columns vs 131, 22 candidate-only), and per-column drift with a 22.3% median and 578% maximum — a differently calibrated build, not a mass-preserving derivative. It also zeroes gift_aid and charitable_investment_gifts and lacks all eleven hmrc_spi_* columns, because it is the input to the HMRC/SPI stage that creates them. Comparing the two reproduces the Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327 failure mode on the restoration surface.

Fifth result — the production release pair (added after review questions). The comparison production actually runs is the shipped release against the next shipping of the same kind — the _export_input_mass_gate pairing whose reference #327 adjudicated to a certified release. Measured on the only consecutive certified UK pair (72aeefc 2026-06-11 → dd68c73 2026-06-19, both sha-verified): zero columns dropped, zero zeroed, four added — the #278 incident class the gate exists for passes cleanly on a real shipping with no threshold judgment. Per-column drift, however, ran median 10.52% with a +2,973.71% maximum (person.adult_ema, intentional reported-benefit repopulation): any tolerance between 5% and 50% would have failed 22–89 columns of a reviewed shipping. The no-headroom release-arm boundary is therefore the pair's exact maximum, relative_tolerance = 29.737060 with minimum_reference_total = 0.0, ratcheting down (never up) as consecutive same-code shippings stabilise, re-measured per certified pair. Adjudicated decision: the constant is deliberately not committed here — it lands with the release-arm call-site PR (items 6/8), because a threshold without the call site that exercises it cannot be attested in a report; the derivation lives in the weighted_integrity docstring so that PR arms from evidence.

So both thresholds require a staged candidate, and the gates correctly remain evidence-gated and unarmed. The floor's derivation is recorded but not committed as a constant, because a policy arms both thresholds together; both land in the follow-up with the two lockstep digest moves.

Acknowledgement: figures derive from the enhanced Family Resources Survey artifact pinned at revision 655dd07e…, itself derived from the DWP Family Resources Survey distributed by the UK Data Service, and from the certified PolicyEngine UK compact. Per EUL clauses 11–12 the citation form specified in the study metadata applies to any onward publication.

Still open (user-owned)

1. Wire the release-arm call site (shipped release vs next shipping) when the items-6/8 lanes land, arming input-mass from the measured pair boundary above; the staging-seam arm still needs a staged candidate: tools/build_uk_national_dataset.py needs the licensed UKDS SPI donor put2223uk.tab (SN 9422), the HMRC ODS surface, and a raw FRS directory. That is the only remaining blocker on arming either gate.
2. Threshold-commit follow-up: constants at the measured boundary with derivations next to UK_MAX_TO_MEDIAN_WEIGHT_RATIO, plus terminal_gates.UK_TERMINAL_GATE_POLICY_SHA256 and contract._UK_TERMINAL_GATE_POLICY_SHA256 moved together.
3. Confirm study Special Conditions (SPI usage agreement; FRS terms) before posting measurement numbers anywhere public.
4. Unrelated bug worth its own issue: registry.py:103 points populace.data.download("uk", 2023) at policyengine/populace-uk-private, which 404s for a fine-grained token without explicit policyengine org scope — the documented UK download path fails confusingly for org members whose token omits it.
5. Second call site (calibrated candidate vs staging, mirroring _export_input_mass_gate): the UK dense/local epic: rowwise constituency-grain dataset by cloning the national staging pool (US dense-arm analog) #495 rowwise arm re-gates household weights only; both helpers are importable from populace.build.uk_runtime and wire in when that arm carries full entity tables (items 6/8 territory).

Verification

• uv run pytest — full workspace suite green (includes 16 new weighted-integrity unit tests and 9 new battery tests).
• uv run ruff check . — clean.
• Red-path smoke: armed gate on a zeroed column → report written with signed attestation, named failure, and reference identity before the raise.

🤖 Generated with Claude Code

[MaxGhenis]

Thorough adversarial pass (same battery every populace PR gets this week). The core of this PR is right — the shared-helper promotion keeps US call sites byte-identical, the UK household-weight broadcast checks out against hand computation at fixture scale, and the differential tests genuinely pin the UK wrappers to the shared gate semantics. The blockers are all in one family, and it's the same family we've been curing across the repo this week (see the #616 review trail): places where the gate's authority over its own inputs is incomplete, so absence or malformation slips through as success.

1. Legacy output path bypasses the fail-closed HMAC contract (pre-existing, but it now swallows your new attestations too). _AttestedUKTerminalGateReport inherits a .passed that checks only functional gates (gates.py:143); with signing credentials unset, the serialized payload correctly fails — but the legacy --input-coverage-json branch skips the raising writer, trusts the object property, and proceeds to staging output (national_build.py:443). Repro on the green four-row fixture with the key unset: report.passed == True while report.report_payload()["passed"] == False with a live RuntimeError in _signing_error.

2. A declared-but-absent QRF output produces a passing zero-column gate. Absent/nonnumeric declared outputs are recorded but omitted from the maps handed to the shared gate (weighted_integrity.py:638, :667), and publication checks only stale exclusions (contract.py:1322). Repro: passed=True, columns_checked=0, absent_columns=['declared_but_absent']. Declared-but-missing should be a named failure — this is exactly the "family silently vanishes" class that run 7 just demonstrated at production scale on the US side.

3. The receipted/expiring exclusion contract from the PR body isn't in the schema. It's column -> reason only (weighted_integrity.py:155) — no approval identity, no expiry, so an exclusion is immortal. 4. Two adjacent leaks: a present column thinner than min_nonzero_records exits before exclusion classification (gates.py:2459 — its exclusion appears in no collection: repro thin={'person.x': 4}, passed=True, reviewed={}, stale=[], dormant=[]), and the plain json.loads + str() loader accepts duplicate keys, nulls, and non-string names.

Suggested path: #616's fix rounds are landing a canonical-authority pattern (module-level plan/surface authorities, live-content digests receipted into gate manifests, no caller substitution) that solves 2–4 as a class — happy to converge on that machinery rather than have you build a parallel one; it should be on main within a day. Finding 1 is a scoped fix in the legacy branch (route it through the raising writer). Everything else here is merge-ready once those close.

[MaxGhenis]

Pushed the review remediation on your branch (María — your gate design and naming are intact; ten fix commits): the legacy --input-coverage-json branch now routes through the raising writer so missing signing credentials fail closed on every output path; declared-but-absent QRF outputs fail by name (zero-checked-columns can no longer pass); the exclusion loader strictly rejects malformed shapes; thin-column exclusions always classify; input-mass evidence is anchored to the reviewed enhanced-FRS identity at load, gate, and publication; and the differential US-parity tests plus the review repros are committed regressions. One thing deliberately left for you: the receipted/expiring exclusion schema design (approval identity, expiry semantics) — marked with TODO(PR #610 review) and noted in the changelog, since that's a policy choice about your gate rather than a bug. Full workspace 5,166 passed / 0 failed; UK battery 706; #583 guard 495 exact.

[MaxGhenis]

Findings closed within scope (receipts in the fix comment); the exclusion-schema design stays with María per the TODO. Approving to dismiss my earlier request-changes.