Retire the UK schema-3 gate path and keep household weights typed in-build (#654)

[juaristi22]

Closes #654. Phase 1 of the #630/#654 build campaign under the #665 master epic: the retire-and-clean stack that lands before the credentialed measurement runs. Three sections, one coherent motion — every seam the Frame migration superseded is retired, not flagged.

1. Typed household weights are the only in-build weight state (#654 tail 1, expanded)

uk_national_frame now consumes and strips the household_weight input column (or takes an explicit household_weights argument), and validate_uk_national_frame refuses exported weight columns in carrier tables instead of asserting column/vector agreement — the invariant is structural, not conventional. Every in-build reader moved to frame.weights_for("household"): the weighted-integrity evidence path, SPI support, CGT/HMRC calibration, HMRC restoration's distributional-mass mapper, diagnostics (now takes a Frame — the carrier invariant), sampling, and the driver's build-record total.

The H5 boundary is unchanged: engine_tables materializes the column from the typed vector at every export, and the loader still seeds typed weights from the stored column and validates on read. Table-level APIs (spi_support builders, the tables-level rowwise clone) keep their explicit column parameters — those are export-surface contracts, not carrier state.

2. Legacy schema-3 terminal gate path retired, with a differential receipt

Retirement was earned by verification, not a caller grep: a differential harness drove identical evidence through both paths — the legacy uk_terminal_gate_report trio and the schema-4 GateBatteryRun — across six scenarios (clean pass, weight-ratio breach, ESS-floor breach, degenerate column, missing evidence, malformed evidence), 66 case×gate comparisons:

Classification	Count	Items
identical	63	all clean/breach/malformed verdicts
battery richer/stricter	2	missing-evidence: uk_input_mass_parity + uk_qrf_tail_concentration (legacy silently omitted un-armed gates; the battery names them)
legacy stricter → ported (review round)	1	missing-evidence: uk_weights_audit — legacy blocked every posture, the battery blocked candidates only. Per the retirement rule (stricter-local behavior is ported, never reclassified away), the entry now declares evidence_absent_blocks in gates.json and the battery blocks its absence in every posture, status staying honestly evidence_absent. The UK vintage digests move with it (mirrored in contract.py).
contradiction	0	—

Full differential receipt (66 rows)

case	gate	legacy	battery	classification
clean_pass	uk_release_input_coverage	True	True	identical
clean_pass	uk_degenerate_release_surface	True	True	identical
clean_pass	uk_zero_weight_strata	True	True	identical
clean_pass	uk_weight_ess	True	True	identical
clean_pass	uk_weight_ratio	True	True	identical
clean_pass	uk_weights_audit	True	True	identical
clean_pass	uk_export_surface	True	True	identical
clean_pass	uk_target_surface	True	True	identical
clean_pass	uk_target_fit	True	True	identical
clean_pass	uk_input_mass_parity	True	True	identical
clean_pass	uk_qrf_tail_concentration	True	True	identical
weight_ratio_breach	uk_release_input_coverage	True	True	identical
weight_ratio_breach	uk_degenerate_release_surface	True	True	identical
weight_ratio_breach	uk_zero_weight_strata	True	True	identical
weight_ratio_breach	uk_weight_ess	True	True	identical
weight_ratio_breach	uk_weight_ratio	False	False	identical
weight_ratio_breach	uk_weights_audit	True	True	identical
weight_ratio_breach	uk_export_surface	True	True	identical
weight_ratio_breach	uk_target_surface	True	True	identical
weight_ratio_breach	uk_target_fit	True	True	identical
weight_ratio_breach	uk_input_mass_parity	True	True	identical
weight_ratio_breach	uk_qrf_tail_concentration	False	False	identical
ess_floor_breach	uk_release_input_coverage	True	True	identical
ess_floor_breach	uk_degenerate_release_surface	True	True	identical
ess_floor_breach	uk_zero_weight_strata	True	True	identical
ess_floor_breach	uk_weight_ess	False	False	identical
ess_floor_breach	uk_weight_ratio	False	False	identical
ess_floor_breach	uk_weights_audit	True	True	identical
ess_floor_breach	uk_export_surface	True	True	identical
ess_floor_breach	uk_target_surface	True	True	identical
ess_floor_breach	uk_target_fit	True	True	identical
ess_floor_breach	uk_input_mass_parity	True	True	identical
ess_floor_breach	uk_qrf_tail_concentration	False	False	identical
degenerate_column	uk_release_input_coverage	True	True	identical
degenerate_column	uk_degenerate_release_surface	False	False	identical
degenerate_column	uk_zero_weight_strata	True	True	identical
degenerate_column	uk_weight_ess	True	True	identical
degenerate_column	uk_weight_ratio	True	True	identical
degenerate_column	uk_weights_audit	True	True	identical
degenerate_column	uk_export_surface	True	True	identical
degenerate_column	uk_target_surface	True	True	identical
degenerate_column	uk_target_fit	True	True	identical
degenerate_column	uk_input_mass_parity	True	True	identical
degenerate_column	uk_qrf_tail_concentration	True	True	identical
missing_evidence	uk_release_input_coverage	True	True	identical
missing_evidence	uk_degenerate_release_surface	True	True	identical
missing_evidence	uk_zero_weight_strata	True	True	identical
missing_evidence	uk_weight_ess	True	True	identical
missing_evidence	uk_weight_ratio	True	True	identical
missing_evidence	uk_weights_audit	False	evidence_absent	legacy_stricter_ported
missing_evidence	uk_export_surface	True	True	identical
missing_evidence	uk_target_surface	True	True	identical
missing_evidence	uk_target_fit	True	True	identical
missing_evidence	uk_input_mass_parity	omitted	evidence_absent	battery_richer_stricter
missing_evidence	uk_qrf_tail_concentration	omitted	evidence_absent	battery_richer_stricter
malformed_evidence	uk_release_input_coverage	True	True	identical
malformed_evidence	uk_degenerate_release_surface	True	True	identical
malformed_evidence	uk_zero_weight_strata	True	True	identical
malformed_evidence	uk_weight_ess	True	True	identical
malformed_evidence	uk_weight_ratio	True	True	identical
malformed_evidence	uk_weights_audit	True	True	identical
malformed_evidence	uk_export_surface	True	True	identical
malformed_evidence	uk_target_surface	True	True	identical
malformed_evidence	uk_target_fit	True	True	identical
malformed_evidence	uk_input_mass_parity	False	False	identical
malformed_evidence	uk_qrf_tail_concentration	True	True	identical

contradictions=0
legacy_better=0
legacy_stricter_ported=1  (uk_weights_audit missing-evidence: legacy blocked every posture, the battery blocked candidates only; strictness ported via the entry's evidence_absent_blocks flag in the #691 review round rather than reclassified away)

With the diff clean, the retirement removes uk_terminal_gate_report, write_uk_terminal_gate_report, UKReleaseParityEvidence, the parity_evidence build hook, the schema-3 signing/attestation block, and the orphaned UK_MAX_TO_MEDIAN_WEIGHT_RATIO / UK_MIN_ESS_FRACTION constants (~930 lines from terminal_gates.py). build/uk/gates.json is now the single build-side policy source — the #630 weight-ratio re-baseline becomes a two-place edit (manifest + the deliberate data-shard mirror in contract.py, untouched here so the published June release keeps verifying). Legacy-only tests retired with their subject; battery assertions were converted to battery-only regression tests.

3. Rowwise clone entry narrowed (#654 tail 2)

Adjudication recorded per the charter: clone_uk_dataset_with_rowwise_geography has one live non-test caller (tools/build_uk_rowwise_dataset.py), so the route is narrowed to Frame | str | Path (ladder-pattern refusal), not retired. The duck-typed _dataset_tables fallback is deleted — both public entries now only ever pass Frame-or-path.

Not in this PR

• microcosm-data contract mirrors and gates.json numeric parameters — those move with the National staging rebuild fails the current terminal gates: constant source_year release column and SPI weight tail above the reviewed maximum #630 receipted values in a later PR.
• The credentialed full-scale runs (UK national staging onto the outer stage runtime, increment 3 of #612 #635 confirmation, National staging rebuild fails the current terminal gates: constant source_year release column and SPI weight tail above the reviewed maximum #630 re-baseline measurement) — operational steps that follow this landing.
• cgt_calibration.py / cgt_imputation.py are staged-for-future by named issues and were adapted, not deleted.

Verification

• Full suite (uv run --no-sync pytest -q): green.
• Lint (uv run --no-sync ruff check .): clean.
• Differential harness ran before any deletion; receipt above.
• Remaining data-dependent check (10% rung bit-identity of gate evidence pre/post the flip) folds into the first measurement run of the campaign, per the Finish the #612 tails: drop the carrier household_weight column; narrow the crosswalk clone entry #654/National staging rebuild fails the current terminal gates: constant source_year release column and SPI weight tail above the reviewed maximum #630 plan.

Refs #630, #611, #612, #618, #635, #665.

🤖 Generated with Claude Code

[vahid-ahmadi]

Net −1,821 lines with a differential receipt attached is the right way to retire a path, and "retirement was earned by verification, not a caller grep" is the sentence I'd want on every deletion PR of this size. Making the exported-weight-column absence structural — validate_uk_national_frame refusing the column rather than asserting agreement with it — is a better invariant than the one it replaces, and it's coherent with dropping the in-place household_weight refresh from the sampler that we went back and forth on in #656.

One row of the receipt is classified in a way I'd push back on.

missing_evidence / uk_weights_audit is a loosening on the default path, not a strengthening.

missing_evidence  uk_weights_audit  False  evidence_absent  battery_richer_stricter

Legacy returned False — a failure, which blocked every build. The battery returns evidence_absent, and on main that only becomes a blocking outcome when the candidate posture is on:

# gate_battery.py:499
elif outcome.status is GateStatus.EVIDENCE_ABSENT and release_candidate:

--release-candidate defaults off (#662, delta 1). So on an ordinary build with no fit-weight records, legacy blocked and the battery does not. The PR body's parenthetical — "which still blocks shippability for release-blocking entries" — is true and is not the same claim: the report is non-shippable, but the build completes where it previously raised.

That matters here more than it would elsewhere, because this gate's own declared intent in uk/gates.json is:

Every fit-produced weight column carries a completed audit record. Armed by the SPI income stage; an absent audit is not a passing audit.

Under the battery on a default build, an absent audit now yields a build that isn't blocked — which is the outcome that sentence exists to forbid.

The other two rows in that bucket are genuinely richer: uk_input_mass_parity and uk_qrf_tail_concentration went omitted → evidence_absent, and legacy didn't block on those either, so naming them is strictly better. It's only False → evidence_absent that changes a blocking verdict.

To be clear about provenance: this behaviour changed in #662, when the battery became the executor — this PR only deletes the now-dead code. But the receipt is what's being offered as the warrant for deletion, and a table reading "contradiction: 0, legacy better: 0" doesn't quite carry the one case where legacy was stricter on the default path. Two options, either fine by me:

• Reclassify that row honestly (a fourth bucket, or a footnote on the existing one) so the receipt says what it found; or
• If an absent weights audit really should block every build per the spec note, make the entry block regardless of posture — either by treating evidence_absent as blocking for that entry specifically, or by arming the gate so absence resolves to failed rather than absent.

I'd lean to the second, since the spec note is unambiguous, but it's a bigger change and reclassifying is a legitimate answer if the intent has genuinely moved.

On the rest

• The typed-weights sweep looks complete from the diff — weighted-integrity, SPI support, CGT/HMRC calibration, HMRC restoration's mass mapper, diagnostics, sampling, and the driver's build-record total all moved to frame.weights_for("household"). Keeping the table-level spi_support builders and the rowwise clone on explicit column parameters is the right line to draw: those are export-surface contracts, not carrier state.
• Worth confirming the ordering at the loader seam still works now that the column is refused: the loader seeds typed weights from the stored column, so the strip has to happen before validate_uk_national_frame sees the frame. The diff reads that way to me, but it's the one place where "refuses the column" and "reads the column" meet.
• Six scenarios × eleven gates with the full 66-row receipt inline is genuinely reviewable, which is unusual for a deletion this size. Whatever you decide on the row above, keep the receipt in the merge commit.

[juaristi22]

You're right, and taking the second option — the spec note is unambiguous and our own retirement rule said stricter-local behavior gets ported, not reclassified away. Landed in the review round:

The port. New manifest vocabulary: a gate entry may declare "evidence_absent_blocks": true, and blocking_outcomes honors it — an evidence_absent outcome on that entry blocks every posture, not only release candidates. The status stays honestly evidence_absent in the report (absence is still not conflated with a measured failure); only the enforcement is strict. uk_weights_audit declares it, with the note extended to say so and name the provenance. The flag is schema-validated closed-world (bool only, refused on an excused not_applicable entry) and serialized omit-when-false, so it rides policy_sha256 and gates_manifest_sha256 exactly when armed — the UK vintage digests move (mirrored in contract.py in the same change, per the lockstep), the US manifest's serialized form is untouched.

The receipt. Reclassified rather than left flattering: the row is now its own bucket —

missing_evidence  uk_weights_audit  False  evidence_absent  legacy_stricter_ported
...
contradictions=0
legacy_better=0
legacy_stricter_ported=1

— with a footnote recording that legacy blocked every posture where the battery blocked candidates only, and that the strictness was ported here rather than reclassified away. PR body table updated to match; the receipt stays in the merge commit. You're also right on provenance: the loosening dates to #662 when the battery became executor — this round restores the pre-#662 enforcement while keeping the schema-4 taxonomy.

Tests. Three additions: the generic mechanism (flagged entry blocks a default-posture build, unflagged doesn't, and the armed flag moves both digests), the UK regression (blocking_outcomes(release_candidate=False) on an unarmed battery is exactly {uk_weights_audit}), and the manifest pin (uk_weights_audit is the only flagged entry, so future opt-outs are a reviewed diff here).

On the loader-seam ordering: confirmed — load_uk_national_frame's only construction path goes through uk_national_frame(...), which consumes-and-strips the stored column before validate_uk_national_frame ever sees the frame; the refusal can't fire on the load path by construction. (The H5 round-trip tests cover exactly this seam.)

🤖 Generated with Claude Code